Skip to content
Snippets Groups Projects
Commit 53d5a79f authored by Tamas Gal's avatar Tamas Gal :speech_balloon:
Browse files

Merge branch 'remove-uproot3' into 'master' 

Remove online support, uproot3 and update to awkward2 and uproot5

See merge request !86
parents 9ede3782 60474112
No related branches found
Tags v0.1.2 v1.2.0
1 merge request!86Remove online support, uproot3 and update to awkward2 and uproot5
Pipeline #56069 passed with warnings
Stage: test
Stage: coverage
Stage: doc
Stage: release
Stage: deploy
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 59 additions and 1596 deletions
.gitlab-ci.yml
+ 24
− 8
View file @ 53d5a79f
...@@ -39,14 +39,6 @@ stages: ...@@ -39,14 +39,6 @@ stages:
reports: reports:
junit: "reports/junit*.xml" junit: "reports/junit*.xml"
test-py3.6:
image: docker.km3net.de/base/python:3.6
stage: test
script:
- *virtualenv_definition
- make test
<<: *junit_definition
test-py3.7: test-py3.7:
image: docker.km3net.de/base/python:3.7 image: docker.km3net.de/base/python:3.7
stage: test stage: test
...@@ -71,6 +63,30 @@ test-py3.9: ...@@ -71,6 +63,30 @@ test-py3.9:
- make test - make test
<<: *junit_definition <<: *junit_definition
test-py3.10:
image: docker.km3net.de/base/python:3.10
stage: test
script:
- *virtualenv_definition
- make test
<<: *junit_definition
test-py3.11:
image: docker.km3net.de/base/python:3.11
stage: test
script:
- *virtualenv_definition
- make test
<<: *junit_definition
test-py3.12:
image: git.km3net.de:4567/common/dockerfiles/base/python:3.12
stage: test
script:
- *virtualenv_definition
- make test
<<: *junit_definition
code-style: code-style:
image: docker.km3net.de/base/python:3.9 image: docker.km3net.de/base/python:3.9
stage: test stage: test
......
CHANGELOG.rst
+ 7
− 0
View file @ 53d5a79f
...@@ -4,6 +4,13 @@ Unreleased changes ...@@ -4,6 +4,13 @@ Unreleased changes
Version 1 Version 1
--------- ---------
1.2.0 / 2024-06-24
~~~~~~~~~~~~~~~~~~
* Removed online format support (online events, timeslices and summary slices) in favour of
the `KM3io.jl <https://git.km3net.de/common/KM3io.jl>`__` Julia Package.
* uproot 5 and awkward 2 are now required
* Python 3.7+ required
1.1.0 / 2024-03-14 1.1.0 / 2024-03-14
~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~
* A few astro helpers were added: azimuth(), zenith(), phi(), theta(), ... * A few astro helpers were added: azimuth(), zenith(), phi(), theta(), ...
......
README.rst
+ 16
− 102
View file @ 53d5a79f
...@@ -58,11 +58,21 @@ If you have a question about km3io, please proceed as follows: ...@@ -58,11 +58,21 @@ If you have a question about km3io, please proceed as follows:
Introduction Introduction
------------ ------------
Most of km3net data is stored in root files. These root files are created using the `KM3NeT Dataformat library <https://git.km3net.de/common/km3net-dataformat>`__ Most of km3net data is stored in root files. These root files are created using
A ROOT file created with the `KM3NeT Dataformat library
`Jpp <https://git.km3net.de/common/jpp>`__ is an "online" file and all other software usually produces "offline" files. <https://git.km3net.de/common/km3net-dataformat>`__ A ROOT file created with
`Jpp <https://git.km3net.de/common/jpp>`__ is an "online" file and all other
km3io is a Python package that provides a set of classes: ``OnlineReader``, ``OfflineReader`` and a special class to read gSeaGen files. All of these ROOT files can be read installing any other software like Jpp, aanet or ROOT. software usually produces "offline" files.
km3io is a Python package that provides access to offline files with its
``OfflineReader`` class and a special one to read gSeaGen files. All of these
ROOT files can be read without installing any other software like Jpp, aanet or
ROOT. km3io v1.1 and earlier also support the access to online files (events,
summaryslices and timeslices). This feature has been dropped due to a lack of
mainteinance power and inf favour of the `KM3io.jl <https://git.km3net.de/common/KM3io.jl>`__` Julia Package, which
provides high-performances access to all ROOT files and should also be
prioritised over ``km3io`` when performance matters (which does, most of the
time).
Data in km3io is returned as ``awkward.Array`` which is an advance Numpy-like container type to store Data in km3io is returned as ``awkward.Array`` which is an advance Numpy-like container type to store
contiguous data for high performance computations. contiguous data for high performance computations.
...@@ -254,100 +264,4 @@ to retrieve the energy of the very first reconstructed track for the first three ...@@ -254,100 +264,4 @@ to retrieve the energy of the very first reconstructed track for the first three
Online files reader Online files reader
------------------- -------------------
``km3io`` is able to read events, summary slices and timeslices. Timeslices are The support to read online ROOT files has been dropped in ``km3io`` v1.2.
currently only supported with split level of 2 or more, which means that reading
L0 timeslices is not working at the moment (but is in progress).
Let's have a look at some online data.
Reading online events
"""""""""""""""""""""
Now we use the ``OnlineReader`` to create our file object.
.. code-block:: python3
import km3io
f = km3io.OnlineReader(data_path("online/km3net_online.root"))
That's it, we created an object which gives access to all the events, but the
relevant data is still not loaded into the memory (lazy access)!
The structure is different compared to the ``OfflineReader``
because online files contain additional branches at the top level
(summaryslices and timeslices).
.. code-block:: python3
>>> f.events
Number of events: 3
>>> f.events.snapshot_hits[1].tot[:10]
array([27, 24, 21, 17, 22, 15, 24, 30, 19, 15], dtype=uint8)
>>> f.events.triggered_hits[1].channel_id[:10]
array([ 2, 3, 16, 22, 23, 0, 2, 3, 4, 5], dtype=uint8)
The resulting arrays are numpy arrays. The indexing convention is: the first indexing
corresponds to the event, the second to the branch and consecutive ones to the
optional dimensions of the arrays. In the last step we accessed the PMT channel IDs
of the first 10 hits of the second event.
Reading SummarySlices
"""""""""""""""""""""
The following example shows how to access summary slices. The summary slices are
returned in chunks to be more efficient with the I/O. The default chunk-size is
1000. In the example file we only have three summaryslices, so there is only a single
chunk. The first index passed to the summaryslices reader is corresponding to the
chunk and the second to the index of the summaryslice in that chunk.
.. code-block:: python3
>>> f.summaryslices
<SummarysliceReader 3 items, step_size=1000 (1 chunk)>
>>> f.summaryslices[0]
SummarysliceChunk(headers=<Array [{' cnt': 671088704, ... ] type='3 * {" cnt": uint32, " vers": uint16, " ...'>, slices=<Array [[{dom_id: 806451572, ... ch30: 48}]] type='3 * var * {"dom_id": int32, "...'>)
>>> f.summaryslices[0].headers
<Array [{' cnt': 671088704, ... ] type='3 * {" cnt": uint32, " vers": uint16, " ...'>
>>> f.summaryslices[0].slices[2]
<Array [{dom_id: 806451572, ... ch30: 48}] type='68 * {"dom_id": int32, "dq_stat...'>
>>> f.summaryslices[0].slices[2].dom_id
<Array [806451572, 806455814, ... 809544061] type='68 * int32'>
>>> f.summaryslices[0].slices[2].ch23
<Array [48, 43, 46, 54, 83, ... 51, 51, 52, 50] type='68 * uint8'>
Reading Timeslices
""""""""""""""""""
Timeslices are split into different streams since 2017 and ``km3io`` currently
supports everything except L0, i.e. L1, L2 and SN streams. The API is
work-in-progress and will be improved in future, however, all the data is
already accessible (although in ugly ways ;-)
To access the timeslice data, you need to specify which timeslice stream
to read:
.. code-block:: python3
>>> f.timeslices
Available timeslice streams: SN, L1
>>> f.timeslices.stream("L1", 0).frames
{806451572: <Table [<Row 0> <Row 1> <Row 2> ... <Row 981> <Row 982> <Row 983>] at 0x00014c167340>,
806455814: <Table [<Row 984> <Row 985> <Row 986> ... <Row 1985> <Row 1986> <Row 1987>] at 0x00014c5f4760>,
806465101: <Table [<Row 1988> <Row 1989> <Row 1990> ... <Row 2236> <Row 2237> <Row 2238>] at 0x00014c5f45e0>,
806483369: <Table [<Row 2239> <Row 2240> <Row 2241> ... <Row 2965> <Row 2966> <Row 2967>] at 0x00014c12b910>,
...
809544061: <Table [<Row 48517> <Row 48518> <Row 48519> ... <Row 49240> <Row 49241> <Row 49242>] at 0x00014ca57100>}
The frames are represented by a dictionary where the key is the ``DOM ID`` and
the value an awkward array of hits, with the usual fields to access the PMT
channel, time and ToT:
.. code-block:: python3
>>> f.timeslices.stream("L1", 0).frames[809524432].dtype
dtype([('pmt', 'u1'), ('tdc', '<u4'), ('tot', 'u1')])
>>> f.timeslices.stream("L1", 0).frames[809524432].tot
array([25, 27, 28, ..., 29, 22, 28], dtype=uint8)
examples/README.rst
+ 2
− 2
View file @ 53d5a79f
Reading online and offline data Reading offline data
=============================== ====================
Feel free to explore and extend our examples! Feel free to explore and extend our examples!
examples/plot_online_example.py deleted 100644 → 0
+ 0
− 78
View file @ 9ede3782
"""
Reading Online Data
===================
The following example shows how to access hits in a ROOT file which is coming
from the detector and written by the `JDataWriter` application.
Such a file is usually called "KM3NET_00000001_00000002.root", where the first
number is the detector ID and the second the run number.
"""
import km3io as ki
from km3net_testdata import data_path
#####################################################
# Accessing the event tree
# ------------------------
# Just pass a filename to the reader class and get access to the event tree
# with:
f = ki.OnlineReader(data_path("online/km3net_online.root"))
#####################################################
# Note that only some meta information is read into memory.
#
# Printing it will simply tell you how many events it has found. Again, nothing
# else is read yet:
print(f.events)
#####################################################
# Now let's look at the hits data:
print(f.events[0].snapshot_hits.tot)
#####################################################
# the resulting arrays are numpy arrays.
#####################################################
# Reading SummarySlices
# ---------------------
# The following example shows how to access summary slices, in particular the DOM
# IDs of the slice with the index 0.
# The current implementation of the summaryslice I/O uses a chunked reading for
# better performance, which means that when you iterate through the `.slices`,
# you'll get chunks of summaryslices in each iteration instead of a single one.
#
# In the example below, we simulate a single iteration by using the `break`
# keyword and then use the data which has been "pulled out" of the ROOT file.
for chunk in f.summaryslices:
break
#####################################################
# `chunk` now contains the first set of summaryslices so `chunk.slice[0]` refers
# to the first summaryslice in the ROOT file. To access e.g. the DOM IDs, use
# the `.dom_id` attribute
dom_ids = chunk.slices[0].dom_id
print(dom_ids)
#####################################################
# The .type attribute (or in general, <TAB> completion) is useful to find out
# more about the field structure:
print(chunk.slices.type)
#####################################################
# Similar to the summaryslice data, the headers can be accessed the same way
# To read the frame index of all summaryslices in the obtained chunk:
print(chunk.headers.frame_index)
#####################################################
# To be continued...
setup.cfg
+ 3
− 10
View file @ 53d5a79f
...@@ -19,7 +19,6 @@ classifiers = ...@@ -19,7 +19,6 @@ classifiers =
Programming Language :: Python Programming Language :: Python
Programming Language :: Python :: 3 Programming Language :: Python :: 3
Programming Language :: Python :: 3 :: Only Programming Language :: Python :: 3 :: Only
Programming Language :: Python :: 3.6
Programming Language :: Python :: 3.7 Programming Language :: Python :: 3.7
Programming Language :: Python :: 3.8 Programming Language :: Python :: 3.8
Programming Language :: Python :: 3.9 Programming Language :: Python :: 3.9
...@@ -37,12 +36,10 @@ packages = find: ...@@ -37,12 +36,10 @@ packages = find:
install_requires = install_requires =
docopt docopt
numba>=0.50 numba>=0.50
awkward>=1.9 awkward>=2
awkward0>=0.15.5 uproot>=5
uproot3>=3.11.1
uproot>=4.2.2
setuptools_scm setuptools_scm
python_requires = >=3.6 python_requires = >=3.7
include_package_data = True include_package_data = True
package_dir = package_dir =
=src =src
...@@ -73,10 +70,6 @@ dev = ...@@ -73,10 +70,6 @@ dev =
sphinxcontrib-versioning sphinxcontrib-versioning
wheel wheel
[options.entry_points]
console_scripts =
KPrintTree = km3io.utils.kprinttree:main
[options.package_data] [options.package_data]
* = *.mplstyle, *.py.typed * = *.mplstyle, *.py.typed
......
src/km3io/__init__.py
+ 0
− 6
View file @ 53d5a79f
...@@ -16,11 +16,5 @@ import os ...@@ -16,11 +16,5 @@ import os
# This needs to be done before import numpy # This needs to be done before import numpy
os.environ["KMP_WARNINGS"] = "off" os.environ["KMP_WARNINGS"] = "off"
with warnings.catch_warnings():
for warning_category in (FutureWarning, DeprecationWarning):
warnings.simplefilter("ignore", category=warning_category)
import uproot3
from .offline import OfflineReader from .offline import OfflineReader
from .online import OnlineReader
from .acoustics import RawAcousticReader from .acoustics import RawAcousticReader
src/km3io/online.py deleted 100644 → 0
+ 0
− 488
View file @ 9ede3782
import binascii
from collections import namedtuple
import os
import uproot
import uproot3
import numpy as np
import numba as nb
TIMESLICE_FRAME_BASKET_CACHE_SIZE = 523 * 1024**2 # [byte]
SUMMARYSLICE_FRAME_BASKET_CACHE_SIZE = 523 * 1024**2 # [byte]
BASKET_CACHE_SIZE = 110 * 1024**2
BASKET_CACHE = uproot3.cache.ThreadSafeArrayCache(BASKET_CACHE_SIZE)
# Parameters for PMT rate conversions, since the rates in summary slices are
# stored as a single byte to save space. The values from 0-255 can be decoded
# using the `get_rate(value)` function, which will yield the actual rate
# in Hz.
MINIMAL_RATE_HZ = 2.0e3
MAXIMAL_RATE_HZ = 2.0e6
RATE_FACTOR = np.log(MAXIMAL_RATE_HZ / MINIMAL_RATE_HZ) / 255
CHANNEL_BITS_TEMPLATE = np.zeros(31, dtype=bool)
BranchConfiguration = namedtuple(
field_names=["branch_address", "interpretation"], typename="BranchConfiguration"
)
class SummarysliceReader:
"""
A reader for summaryslices which are loaded as chunks given by step_size.
To be used as an iterator (`for chunks in SummarysliceReader(...): ...`)
"""
TREE_ADDR = "KM3NET_SUMMARYSLICE/KM3NET_SUMMARYSLICE"
_subbranches = [
BranchConfiguration(
"KM3NETDAQ::JDAQSummarysliceHeader",
uproot.interpretation.numerical.AsDtype(
[
(" cnt", "u4"),
(" vers", "u2"),
(" cnt2", "u4"),
(" vers2", "u2"),
(" cnt3", "u4"),
(" vers3", "u2"),
("detector_id", ">i4"),
("run", ">i4"),
("frame_index", ">i4"),
(" cnt4", "u4"),
(" vers4", "u2"),
("UTC_seconds", ">u4"),
("UTC_16nanosecondcycles", ">u4"),
]
),
),
BranchConfiguration(
"vector<KM3NETDAQ::JDAQSummaryFrame>",
uproot.interpretation.jagged.AsJagged(
uproot.interpretation.numerical.AsDtype(
[
("dom_id", ">i4"),
("dq_status", ">u4"),
("hrv", ">u4"),
("fifo", ">u4"),
("status3", ">u4"),
("status4", ">u4"),
]
+ [(f"ch{c}", "u1") for c in range(31)]
),
header_bytes=10,
),
),
]
def __init__(self, fobj, step_size=1000):
if isinstance(fobj, str):
self._fobj = uproot.open(fobj)
else:
self._fobj = fobj
self._step_size = step_size
self._branch = self._fobj[self.TREE_ADDR]
self.ChunksConstructor = namedtuple(
field_names=["headers", "slices"], typename="SummarysliceChunk"
)
def _chunks_generator(self):
for chunk in self._branch.iterate(
dict(self._subbranches), step_size=self._step_size
):
yield self.ChunksConstructor(
*[getattr(chunk, bc.branch_address) for bc in self._subbranches]
)
def __getitem__(self, idx):
if idx >= len(self) or idx < -len(self):
raise IndexError("Chunk index out of range")
s = self._step_size
if idx < 0:
idx = len(self) + idx
chunk = self._branch.arrays(
dict(self._subbranches), entry_start=idx * s, entry_stop=(idx + 1) * s
)
return self.ChunksConstructor(
*[getattr(chunk, bc.branch_address) for bc in self._subbranches]
)
def __iter__(self):
self._chunks = self._chunks_generator()
return self
def __next__(self):
return next(self._chunks)
def __len__(self):
return int(np.ceil(self._branch.num_entries / self._step_size))
def __repr__(self):
step_size = self._step_size
n_items = self._branch.num_entries
cls_name = self.__class__.__name__
n_chunks = len(self)
return (
f"<{cls_name} {n_items} items, step_size={step_size} "
f"({n_chunks} chunk{'' if n_chunks == 1 else 's'})>"
)
@nb.vectorize(
[nb.int32(nb.int8), nb.int32(nb.int16), nb.int32(nb.int32), nb.int32(nb.int64)]
)
def get_rate(value): # pragma: no cover
"""Return the rate in Hz from the short int value"""
if value == 0:
return 0
else:
return MINIMAL_RATE_HZ * np.exp(value * RATE_FACTOR)
@nb.guvectorize(
"void(i8, b1[:], b1[:])", "(), (n) -> (n)", target="parallel", nopython=True
)
def unpack_bits(value, bits_template, out): # pragma: no cover
"""Return a boolean array for a value's bit representation.
This function also accepts arrays as input, the output shape will be
NxM where N is the number of input values and M the length of the
``bits_template`` array, which is just a dummy array, due to the weird
signature system of numba.
Parameters
----------
value: int or np.array(int) with shape (N,)
The binary value of containing the bit information
bits_template: np.array() with shape (M,)
The template for the output array, the only important is its shape
Returns
-------
np.array(bool) either with shape (M,) or (N, M)
"""
for i in range(bits_template.shape[0]):
out[30 - i] = value & (1 << i) > 0
def get_channel_flags(value):
"""Returns the hrv/fifo flags for the PMT channels (hrv/fifo)
Parameters
----------
value : int32
The integer value to be parsed.
"""
channel_bits = np.bitwise_and(value, 0x7FFFFFFF)
flags = unpack_bits(channel_bits, CHANNEL_BITS_TEMPLATE)
return np.flip(flags, axis=-1)
def get_number_udp_packets(value):
"""Returns the number of received UDP packets (dq_status)
Parameters
----------
value : int32
The integer value to be parsed.
"""
return np.bitwise_and(value, 0x7FFF)
def get_udp_max_sequence_number(value):
"""Returns the maximum sequence number of the received UDP packets (dq_status)
Parameters
----------
value : int32
The integer value to be parsed.
"""
return np.right_shift(value, 16)
def has_udp_trailer(value):
"""Returns the UDP Trailer flag (fifo)
Parameters
----------
value : int32
The integer value to be parsed.
"""
return np.any(np.bitwise_and(value, np.left_shift(1, 31)))
class OnlineReader:
"""Reader for online ROOT files"""
def __init__(self, filename):
self._fobj = uproot3.open(filename)
self._filename = filename
self._events = None
self._timeslices = None
self._summaryslices = None
self._uuid = binascii.hexlify(self._fobj._context.uuid).decode("ascii")
@property
def uuid(self):
return self._uuid
def close(self):
self._fobj.close()
def __enter__(self):
return self
def __exit__(self, *args):
self.close()
@property
def events(self):
if self._events is None:
tree = self._fobj["KM3NET_EVENT"]
headers = tree["KM3NETDAQ::JDAQEventHeader"].array(
uproot3.interpret(tree["KM3NETDAQ::JDAQEventHeader"], cntvers=True)
)
snapshot_hits = tree["snapshotHits"].array(
uproot3.asjagged(
uproot3.astable(
uproot3.asdtype(
[
("dom_id", ">i4"),
("channel_id", "u1"),
("time", "<u4"),
("tot", "u1"),
]
)
),
skipbytes=10,
)
)
triggered_hits = tree["triggeredHits"].array(
uproot3.asjagged(
uproot3.astable(
uproot3.asdtype(
[
("dom_id", ">i4"),
("channel_id", "u1"),
("time", "<u4"),
("tot", "u1"),
(" cnt", "u4"),
(" vers", "u2"),
("trigger_mask", ">u8"),
]
)
),
skipbytes=10,
)
)
self._events = OnlineEvents(headers, snapshot_hits, triggered_hits)
return self._events
@property
def timeslices(self):
if self._timeslices is None:
self._timeslices = Timeslices(self._fobj)
return self._timeslices
@property
def summaryslices(self):
if self._summaryslices is None:
self._summaryslices = SummarysliceReader(
uproot.open(self._filename)
) # TODO: remove when using uproot4
return self._summaryslices
class Timeslices:
"""A simple wrapper for timeslices"""
def __init__(self, fobj):
self._fobj = fobj
self._timeslices = {}
self._read_streams()
def _read_streams(self):
"""Read the L0, L1, L2 and SN streams if available"""
streams = set(
s.split(b"KM3NET_TIMESLICE_")[1].split(b";")[0]
for s in self._fobj.keys()
if b"KM3NET_TIMESLICE_" in s
)
for stream in streams:
tree = self._fobj[b"KM3NET_TIMESLICE_" + stream][
b"KM3NETDAQ::JDAQTimeslice"
]
headers = tree[b"KM3NETDAQ::JDAQTimesliceHeader"][b"KM3NETDAQ::JDAQHeader"][
b"KM3NETDAQ::JDAQChronometer"
]
if len(headers) == 0:
continue
superframes = tree[b"vector<KM3NETDAQ::JDAQSuperFrame>"]
hits_dtype = np.dtype([("pmt", "u1"), ("tdc", "<u4"), ("tot", "u1")])
hits_buffer = superframes[
b"vector<KM3NETDAQ::JDAQSuperFrame>.buffer"
].lazyarray(
uproot3.asjagged(
uproot3.astable(uproot3.asdtype(hits_dtype)), skipbytes=6
),
basketcache=uproot3.cache.ThreadSafeArrayCache(
TIMESLICE_FRAME_BASKET_CACHE_SIZE
),
)
self._timeslices[stream.decode("ascii")] = (
headers,
superframes,
hits_buffer,
)
setattr(
self,
stream.decode("ascii"),
TimesliceStream(headers, superframes, hits_buffer),
)
def stream(self, stream, idx):
ts = self._timeslices[stream]
return Timeslice(*ts, idx, stream)
def __str__(self):
return "Available timeslice streams: {}".format(
", ".join(s for s in self._timeslices.keys())
)
def __repr__(self):
return str(self)
class TimesliceStream:
def __init__(self, headers, superframes, hits_buffer):
# self.headers = headers.lazyarray(
# uproot3.asjagged(uproot3.astable(
# uproot3.asdtype(
# np.dtype([('a', 'i4'), ('b', 'i4'), ('c', 'i4'),
# ('d', 'i4'), ('e', 'i4')]))),
# skipbytes=6),
# basketcache=uproot3.cache.ThreadSafeArrayCache(
# TIMESLICE_FRAME_BASKET_CACHE_SIZE))
self.headers = headers
self.superframes = superframes
self._hits_buffer = hits_buffer
# def frames(self):
# n_hits = self._superframe[
# b'vector<KM3NETDAQ::JDAQSuperFrame>.numberOfHits'].lazyarray(
# basketcache=BASKET_CACHE)[self._idx]
# module_ids = self._superframe[
# b'vector<KM3NETDAQ::JDAQSuperFrame>.id'].lazyarray(basketcache=BASKET_CACHE)[self._idx]
# idx = 0
# for module_id, n_hits in zip(module_ids, n_hits):
# self._frames[module_id] = hits_buffer[idx:idx + n_hits]
# idx += n_hits
class Timeslice:
"""A wrapper for a timeslice"""
def __init__(self, header, superframe, hits_buffer, idx, stream):
self.header = header
self._frames = {}
self._superframe = superframe
self._hits_buffer = hits_buffer
self._idx = idx
self._stream = stream
self._n_frames = None
@property
def frames(self):
if not self._frames:
self._read_frames()
return self._frames
def _read_frames(self):
"""Populate a dictionary of frames with the module ID as key"""
hits_buffer = self._hits_buffer[self._idx]
n_hits = self._superframe[
b"vector<KM3NETDAQ::JDAQSuperFrame>.numberOfHits"
].lazyarray(basketcache=BASKET_CACHE)[self._idx]
try:
module_ids = self._superframe[
b"vector<KM3NETDAQ::JDAQSuperFrame>.id"
].lazyarray(basketcache=BASKET_CACHE)[self._idx]
except KeyError:
module_ids = (
self._superframe[
b"vector<KM3NETDAQ::JDAQSuperFrame>.KM3NETDAQ::JDAQModuleIdentifier"
]
.lazyarray(
uproot3.asjagged(
uproot3.astable(uproot3.asdtype([("dom_id", ">i4")]))
),
basketcache=BASKET_CACHE,
)[self._idx]
.dom_id
)
idx = 0
for module_id, n_hits in zip(module_ids, n_hits):
self._frames[module_id] = hits_buffer[idx : idx + n_hits]
idx += n_hits
def __len__(self):
if self._n_frames is None:
self._n_frames = len(
self._superframe[b"vector<KM3NETDAQ::JDAQSuperFrame>.id"].lazyarray(
basketcache=BASKET_CACHE
)[self._idx]
)
return self._n_frames
def __str__(self):
return "{} timeslice with {} frames.".format(self._stream, len(self))
def __repr__(self):
return "<{}: {} entries>".format(self.__class__.__name__, len(self.header))
class OnlineEvents:
"""A simple wrapper for online events"""
def __init__(self, headers, snapshot_hits, triggered_hits):
self.headers = headers
self.snapshot_hits = snapshot_hits
self.triggered_hits = triggered_hits
def __getitem__(self, item):
return OnlineEvent(
self.headers[item], self.snapshot_hits[item], self.triggered_hits[item]
)
def __len__(self):
return len(self.headers)
def __str__(self):
return "Number of events: {}".format(len(self.headers))
def __repr__(self):
return str(self)
class OnlineEvent:
"""A wrapper for a online event"""
def __init__(self, header, snapshot_hits, triggered_hits):
self.header = header
self.snapshot_hits = snapshot_hits
self.triggered_hits = triggered_hits
def __str__(self):
return "Online event with {} snapshot hits and {} triggered hits".format(
len(self.snapshot_hits), len(self.triggered_hits)
)
def __repr__(self):
return str(self)
src/km3io/rootio.py
+ 1
− 1
View file @ 53d5a79f
...@@ -60,7 +60,7 @@ class EventReader: ...@@ -60,7 +60,7 @@ class EventReader:
else: else:
raise TypeError("Unsupported file descriptor.") raise TypeError("Unsupported file descriptor.")
self._step_size = step_size self._step_size = step_size
self._uuid = self._fobj._file.uuid self._uuid = self._fobj.parent.uuid
self._iterator_index = 0 self._iterator_index = 0
self._keys = keys self._keys = keys
self._event_ctor = event_ctor self._event_ctor = event_ctor
......
src/km3io/tools.py
+ 6
− 25
View file @ 53d5a79f
...@@ -3,7 +3,6 @@ from collections import namedtuple ...@@ -3,7 +3,6 @@ from collections import namedtuple
import numba as nb import numba as nb
import numpy as np import numpy as np
import awkward as ak import awkward as ak
import uproot3
import km3io.definitions import km3io.definitions
from km3io.definitions import reconstruction as krec from km3io.definitions import reconstruction as krec
...@@ -12,10 +11,6 @@ from km3io.definitions import fitparameters as kfit ...@@ -12,10 +11,6 @@ from km3io.definitions import fitparameters as kfit
from km3io.definitions import w2list_genhen as kw2gen from km3io.definitions import w2list_genhen as kw2gen
from km3io.definitions import w2list_gseagen as kw2gsg from km3io.definitions import w2list_gseagen as kw2gsg
# 110 MB based on the size of the largest basket found so far in km3net
BASKET_CACHE_SIZE = 110 * 1024**2
BASKET_CACHE = uproot3.cache.ThreadSafeArrayCache(BASKET_CACHE_SIZE)
class cached_property: class cached_property:
"""A simple cache decorator for properties.""" """A simple cache decorator for properties."""
...@@ -312,41 +307,27 @@ def mask(arr, sequence=None, startend=None, minmax=None, atleast=None): ...@@ -312,41 +307,27 @@ def mask(arr, sequence=None, startend=None, minmax=None, atleast=None):
elif atleast is not None: elif atleast is not None:
np_array = _mask_atleast(ak.Array(layout), np.array(atleast)) np_array = _mask_atleast(ak.Array(layout), np.array(atleast))
return ak.layout.NumpyArray(np_array) return ak.contents.NumpyArray(np_array)
elif isinstance( elif isinstance(layout, ak.contents.ListArray):
layout,
(
ak.layout.ListArray32,
ak.layout.ListArrayU32,
ak.layout.ListArray64,
),
):
if len(layout.stops) == 0: if len(layout.stops) == 0:
content = recurse(layout.content) content = recurse(layout.content)
else: else:
content = recurse(layout.content[: np.max(layout.stops)]) content = recurse(layout.content[: np.max(layout.stops)])
return type(layout)(layout.starts, layout.stops, content) return type(layout)(layout.starts, layout.stops, content)
elif isinstance( elif isinstance(layout, ak.contents.ListOffsetArray):
layout,
(
ak.layout.ListOffsetArray32,
ak.layout.ListOffsetArrayU32,
ak.layout.ListOffsetArray64,
),
):
content = recurse(layout.content[: layout.offsets[-1]]) content = recurse(layout.content[: layout.offsets[-1]])
return type(layout)(layout.offsets, content) return type(layout)(layout.offsets, content)
elif isinstance(layout, ak.layout.RegularArray): elif isinstance(layout, ak.contents.RegularArray):
content = recurse(layout.content) content = recurse(layout.content)
return ak.layout.RegularArray(content, layout.size) return ak.contents.RegularArray(content, layout.size)
else: else:
raise NotImplementedError(repr(arr)) raise NotImplementedError(repr(arr))
layout = ak.to_layout(arr, allow_record=True, allow_other=False) layout = ak.to_layout(arr, allow_record=True)
return ak.Array(recurse(layout)) return ak.Array(recurse(layout))
......
src/km3io/utils/kprinttree.py deleted 100644 → 0
+ 0
− 45
View file @ 9ede3782
#!/usr/bin/env python
# coding=utf-8
# Filename: kprinttree.py
# Author: Tamas Gal <tgal@km3net.de>
"""
Print the available ROOT trees.
Usage:
KPrintTree -f FILENAME
KPrintTree (-h | --help)
Options:
-f FILENAME The file to print (;
-h --help Show this screen.
"""
import warnings
with warnings.catch_warnings():
for warning_category in (FutureWarning, DeprecationWarning):
warnings.simplefilter("ignore", category=warning_category)
import uproot3
def print_tree(filename):
f = uproot3.open(filename)
for key in f.keys():
try:
print("{:<30} : {:>9} items".format(key.decode(), len(f[key])))
except (TypeError, KeyError):
print("{}".format(key.decode()))
except NotImplementedError:
print("{} (TStreamerSTL)".format(key.decode()))
def main():
from docopt import docopt
args = docopt(__doc__)
print_tree(args["-f"])
if __name__ == "__main__":
main()
tests/test_online.py deleted 100644 → 0
+ 0
− 831
View file @ 9ede3782
This diff is collapsed.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment