Skip to content
Snippets Groups Projects
Commit da1b1374 authored by Tamas Gal's avatar Tamas Gal :speech_balloon:
Browse files

Merge branch '25-rename-daq-to-online' into 'master' 

Resolve "Rename "DAQ" to "online""

Closes #25

See merge request !28
parents 13c12bbf 81d86472
No related branches found
No related tags found
1 merge request!28Resolve "Rename "DAQ" to "online""
Pipeline #10238 passed with warnings
Stage: test
Stage: coverage
Stage: doc
Stage: release
Stage: deploy
Hide whitespace changes
Inline Side-by-side
README.rst
+ 12
− 12
View file @ da1b1374
...@@ -50,11 +50,11 @@ Tutorial ...@@ -50,11 +50,11 @@ Tutorial
* `Introduction <#introduction>`__ * `Introduction <#introduction>`__
* `Overview of daq files <#overview-of-daq-files>`__ * `Overview of online files <#overview-of-online-files>`__
* `Overview of offline files <#overview-of-offline-files>`__ * `Overview of offline files <#overview-of-offline-files>`__
* `DAQ files reader <#daq-files-reader>`__ * `Online files reader <#online-files-reader>`__
* `Reading Events <#reading-events>`__ * `Reading Events <#reading-events>`__
...@@ -82,9 +82,9 @@ Introduction ...@@ -82,9 +82,9 @@ Introduction
------------ ------------
Most of km3net data is stored in root files. These root files are either created with `Jpp <https://git.km3net.de/common/jpp>`__ or `aanet <https://git.km3net.de/common/aanet>`__ software. A root file created with Most of km3net data is stored in root files. These root files are either created with `Jpp <https://git.km3net.de/common/jpp>`__ or `aanet <https://git.km3net.de/common/aanet>`__ software. A root file created with
`Jpp <https://git.km3net.de/common/jpp>`__ is often referred to as "a Jpp root file". Similarly, a root file created with `aanet <https://git.km3net.de/common/aanet>`__ is often referred to as "an aanet file". In km3io, an aanet root file will always be reffered to as an ``offline file``, while a Jpp root file will always be referred to as a ``daq file``. `Jpp <https://git.km3net.de/common/jpp>`__ is often referred to as "a Jpp root file". Similarly, a root file created with `aanet <https://git.km3net.de/common/aanet>`__ is often referred to as "an aanet file". In km3io, an aanet root file will always be reffered to as an ``offline file``, while a Jpp ROOT file will always be referred to as a ``online file``.
km3io is a Python package that provides a set of classes (``DAQReader`` and ``OfflineReader``) to read both daq root files and offline root files without any dependency to aanet, Jpp or ROOT. km3io is a Python package that provides a set of classes (``OnlineReader`` and ``OfflineReader``) to read both online ROOT files and offline ROOT files without any dependency to aanet, Jpp or ROOT.
Data in km3io is often returned as a "lazyarray", a "jagged lazyarray" or a `Numpy <https://docs.scipy.org/doc/numpy>`__ array. A lazyarray is an array-like object that reads data on demand! In a lazyarray, only the first and the last chunks of data are read in memory. A lazyarray can be used with all Numpy's universal `functions <https://docs.scipy.org/doc/numpy/reference/ufuncs.html>`__. Here is how a lazyarray looks like: Data in km3io is often returned as a "lazyarray", a "jagged lazyarray" or a `Numpy <https://docs.scipy.org/doc/numpy>`__ array. A lazyarray is an array-like object that reads data on demand! In a lazyarray, only the first and the last chunks of data are read in memory. A lazyarray can be used with all Numpy's universal `functions <https://docs.scipy.org/doc/numpy/reference/ufuncs.html>`__. Here is how a lazyarray looks like:
...@@ -101,10 +101,9 @@ A jagged array, is a 2+ dimentional array with different arrays lengths. In othe ...@@ -101,10 +101,9 @@ A jagged array, is a 2+ dimentional array with different arrays lengths. In othe
# <JaggedArray [[102 102 102 ... 11517 11518 11518] [] [101 101 102 ... 11518 11518 11518] ... [101 101 102 ... 11516 11516 11517] [] [101 101 101 ... 11517 11517 11518]] at 0x7f74b0ef8810> # <JaggedArray [[102 102 102 ... 11517 11518 11518] [] [101 101 102 ... 11518 11518 11518] ... [101 101 102 ... 11516 11516 11517] [] [101 101 101 ... 11517 11517 11518]] at 0x7f74b0ef8810>
Overview of DAQ files Overview of Online files
""""""""""""""""""""" """""""""""""""""""""
DAQ files, or also called online files, are written by the DataWriter and Online files are written by the DataWriter (part of Jpp) and contain events, timeslices and summary slices.
contain events, timeslics and summary slices.
Overview of offline files Overview of offline files
...@@ -208,11 +207,12 @@ Offline files contain data about events, hits and tracks. Based on aanet version ...@@ -208,11 +207,12 @@ Offline files contain data about events, hits and tracks. Based on aanet version
"start_run", "run_id" "start_run", "run_id"
DAQ files reader Online files reader
---------------- ----------------
``km3io`` is able to read events, summary slices and timeslices (except the L0 ``km3io`` is able to read events, summary slices and timeslices. Timeslices are
slices, which is work in progress). currently only supported with split level of 2 or more, which means that reading
L0 timeslices is currently not working (but in progress).
Let's have a look at some ORCA data (``KM3NeT_00000044_00005404.root``) Let's have a look at some ORCA data (``KM3NeT_00000044_00005404.root``)
...@@ -224,7 +224,7 @@ To get a lazy ragged array of the events: ...@@ -224,7 +224,7 @@ To get a lazy ragged array of the events:
.. code-block:: python3 .. code-block:: python3
import km3io import km3io
f = km3io.DAQReader("KM3NeT_00000044_00005404.root") f = km3io.OnlineReader("KM3NeT_00000044_00005404.root")
That's it, we created an object which gives access to all the events, but the That's it, we created an object which gives access to all the events, but the
...@@ -249,7 +249,7 @@ IDs of the slice with the index ``23``: ...@@ -249,7 +249,7 @@ IDs of the slice with the index ``23``:
.. code-block:: python3 .. code-block:: python3
>>> f.summaryslices >>> f.summaryslices
<km3io.daq.SummarySlices at 0x7effcc0e52b0> <km3io.online.SummarySlices at 0x7effcc0e52b0>
>>> f.summaryslices.slices[23].dom_id >>> f.summaryslices.slices[23].dom_id
array([806451572, 806455814, 806465101, 806483369, 806487219, 806487226, array([806451572, 806455814, 806465101, 806483369, 806487219, 806487226,
806487231, 808432835, 808435278, 808447180, 808447186, 808451904, 806487231, 808432835, 808435278, 808447180, 808447186, 808451904,
......
km3io/__init__.py
+ 1
− 1
View file @ da1b1374
...@@ -3,5 +3,5 @@ from pkg_resources import get_distribution, DistributionNotFound ...@@ -3,5 +3,5 @@ from pkg_resources import get_distribution, DistributionNotFound
version = get_distribution(__name__).version version = get_distribution(__name__).version
from .offline import OfflineReader from .offline import OfflineReader
from .daq import DAQReader from .online import OnlineReader
from .gseagen import GSGReader from .gseagen import GSGReader
km3io/daq.py → km3io/online.py
+ 17
− 17
View file @ da1b1374
...@@ -107,8 +107,8 @@ def has_udp_trailer(value): ...@@ -107,8 +107,8 @@ def has_udp_trailer(value):
return np.any(np.bitwise_and(value, np.left_shift(1, 31))) return np.any(np.bitwise_and(value, np.left_shift(1, 31)))
class DAQReader: class OnlineReader:
"""Reader for DAQ ROOT files""" """Reader for online ROOT files"""
def __init__(self, filename): def __init__(self, filename):
self._fobj = uproot.open(filename) self._fobj = uproot.open(filename)
self._events = None self._events = None
...@@ -135,13 +135,13 @@ class DAQReader: ...@@ -135,13 +135,13 @@ class DAQReader:
(" cnt", "u4"), (" vers", "u2"), (" cnt", "u4"), (" vers", "u2"),
("trigger_mask", ">u8")])), ("trigger_mask", ">u8")])),
skipbytes=10)) skipbytes=10))
self._events = DAQEvents(headers, snapshot_hits, triggered_hits) self._events = OnlineEvents(headers, snapshot_hits, triggered_hits)
return self._events return self._events
@property @property
def timeslices(self): def timeslices(self):
if self._timeslices is None: if self._timeslices is None:
self._timeslices = DAQTimeslices(self._fobj) self._timeslices = Timeslices(self._fobj)
return self._timeslices return self._timeslices
@property @property
...@@ -202,8 +202,8 @@ class SummarySlices: ...@@ -202,8 +202,8 @@ class SummarySlices:
return "Number of summaryslices: {}".format(len(self.headers)) return "Number of summaryslices: {}".format(len(self.headers))
class DAQTimeslices: class Timeslices:
"""A simple wrapper for DAQ timeslices""" """A simple wrapper for timeslices"""
def __init__(self, fobj): def __init__(self, fobj):
self._fobj = fobj self._fobj = fobj
self._timeslices = {} self._timeslices = {}
...@@ -233,11 +233,11 @@ class DAQTimeslices: ...@@ -233,11 +233,11 @@ class DAQTimeslices:
self._timeslices[stream.decode("ascii")] = (headers, superframes, self._timeslices[stream.decode("ascii")] = (headers, superframes,
hits_buffer) hits_buffer)
setattr(self, stream.decode("ascii"), setattr(self, stream.decode("ascii"),
DAQTimesliceStream(headers, superframes, hits_buffer)) TimesliceStream(headers, superframes, hits_buffer))
def stream(self, stream, idx): def stream(self, stream, idx):
ts = self._timeslices[stream] ts = self._timeslices[stream]
return DAQTimeslice(*ts, idx, stream) return Timeslice(*ts, idx, stream)
def __str__(self): def __str__(self):
return "Available timeslice streams: {}".format(', '.join( return "Available timeslice streams: {}".format(', '.join(
...@@ -247,7 +247,7 @@ class DAQTimeslices: ...@@ -247,7 +247,7 @@ class DAQTimeslices:
return str(self) return str(self)
class DAQTimesliceStream: class TimesliceStream:
def __init__(self, headers, superframes, hits_buffer): def __init__(self, headers, superframes, hits_buffer):
# self.headers = headers.lazyarray( # self.headers = headers.lazyarray(
# uproot.asjagged(uproot.astable( # uproot.asjagged(uproot.astable(
...@@ -273,8 +273,8 @@ class DAQTimesliceStream: ...@@ -273,8 +273,8 @@ class DAQTimesliceStream:
# idx += n_hits # idx += n_hits
class DAQTimeslice: class Timeslice:
"""A wrapper for a DAQ timeslice""" """A wrapper for a timeslice"""
def __init__(self, header, superframe, hits_buffer, idx, stream): def __init__(self, header, superframe, hits_buffer, idx, stream):
self.header = header self.header = header
self._frames = {} self._frames = {}
...@@ -327,15 +327,15 @@ class DAQTimeslice: ...@@ -327,15 +327,15 @@ class DAQTimeslice:
len(self.header)) len(self.header))
class DAQEvents: class OnlineEvents:
"""A simple wrapper for DAQ events""" """A simple wrapper for online events"""
def __init__(self, headers, snapshot_hits, triggered_hits): def __init__(self, headers, snapshot_hits, triggered_hits):
self.headers = headers self.headers = headers
self.snapshot_hits = snapshot_hits self.snapshot_hits = snapshot_hits
self.triggered_hits = triggered_hits self.triggered_hits = triggered_hits
def __getitem__(self, item): def __getitem__(self, item):
return DAQEvent(self.headers[item], self.snapshot_hits[item], return OnlineEvent(self.headers[item], self.snapshot_hits[item],
self.triggered_hits[item]) self.triggered_hits[item])
def __len__(self): def __len__(self):
...@@ -348,15 +348,15 @@ class DAQEvents: ...@@ -348,15 +348,15 @@ class DAQEvents:
return str(self) return str(self)
class DAQEvent: class OnlineEvent:
"""A wrapper for a DAQ event""" """A wrapper for a online event"""
def __init__(self, header, snapshot_hits, triggered_hits): def __init__(self, header, snapshot_hits, triggered_hits):
self.header = header self.header = header
self.snapshot_hits = snapshot_hits self.snapshot_hits = snapshot_hits
self.triggered_hits = triggered_hits self.triggered_hits = triggered_hits
def __str__(self): def __str__(self):
return "DAQ event with {} snapshot hits and {} triggered hits".format( return "Online event with {} snapshot hits and {} triggered hits".format(
len(self.snapshot_hits), len(self.triggered_hits)) len(self.snapshot_hits), len(self.triggered_hits))
def __repr__(self): def __repr__(self):
......
tests/test_daq.py → tests/test_online.py
+ 14
− 14
View file @ da1b1374
...@@ -2,14 +2,14 @@ import os ...@@ -2,14 +2,14 @@ import os
import re import re
import unittest import unittest
from km3io.daq import DAQReader, get_rate, has_udp_trailer, get_udp_max_sequence_number, get_channel_flags, get_number_udp_packets from km3io.online import OnlineReader, get_rate, has_udp_trailer, get_udp_max_sequence_number, get_channel_flags, get_number_udp_packets
SAMPLES_DIR = os.path.join(os.path.dirname(__file__), "samples") SAMPLES_DIR = os.path.join(os.path.dirname(__file__), "samples")
class TestDAQEvents(unittest.TestCase): class TestOnlineEvents(unittest.TestCase):
def setUp(self): def setUp(self):
self.events = DAQReader(os.path.join(SAMPLES_DIR, self.events = OnlineReader(os.path.join(SAMPLES_DIR,
"daq_v1.0.0.root")).events "daq_v1.0.0.root")).events
def test_index_lookup(self): def test_index_lookup(self):
...@@ -22,9 +22,9 @@ class TestDAQEvents(unittest.TestCase): ...@@ -22,9 +22,9 @@ class TestDAQEvents(unittest.TestCase):
assert re.match(".*events.*3", self.events.__repr__()) assert re.match(".*events.*3", self.events.__repr__())
class TestDAQEvent(unittest.TestCase): class TestOnlineEvent(unittest.TestCase):
def setUp(self): def setUp(self):
self.event = DAQReader(os.path.join(SAMPLES_DIR, self.event = OnlineReader(os.path.join(SAMPLES_DIR,
"daq_v1.0.0.root")).events[0] "daq_v1.0.0.root")).events[0]
def test_str(self): def test_str(self):
...@@ -36,9 +36,9 @@ class TestDAQEvent(unittest.TestCase): ...@@ -36,9 +36,9 @@ class TestDAQEvent(unittest.TestCase):
self.event.__repr__()) self.event.__repr__())
class TestDAQEventsSnapshotHits(unittest.TestCase): class TestOnlineEventsSnapshotHits(unittest.TestCase):
def setUp(self): def setUp(self):
self.events = DAQReader(os.path.join(SAMPLES_DIR, self.events = OnlineReader(os.path.join(SAMPLES_DIR,
"daq_v1.0.0.root")).events "daq_v1.0.0.root")).events
self.lengths = {0: 96, 1: 124, -1: 78} self.lengths = {0: 96, 1: 124, -1: 78}
self.total_item_count = 298 self.total_item_count = 298
...@@ -75,9 +75,9 @@ class TestDAQEventsSnapshotHits(unittest.TestCase): ...@@ -75,9 +75,9 @@ class TestDAQEventsSnapshotHits(unittest.TestCase):
assert all(c < 31 for c in hits.channel_id.max()) assert all(c < 31 for c in hits.channel_id.max())
class TestDAQEventsTriggeredHits(unittest.TestCase): class TestOnlineEventsTriggeredHits(unittest.TestCase):
def setUp(self): def setUp(self):
self.events = DAQReader(os.path.join(SAMPLES_DIR, self.events = OnlineReader(os.path.join(SAMPLES_DIR,
"daq_v1.0.0.root")).events "daq_v1.0.0.root")).events
self.lengths = {0: 18, 1: 53, -1: 9} self.lengths = {0: 18, 1: 53, -1: 9}
self.total_item_count = 80 self.total_item_count = 80
...@@ -116,9 +116,9 @@ class TestDAQEventsTriggeredHits(unittest.TestCase): ...@@ -116,9 +116,9 @@ class TestDAQEventsTriggeredHits(unittest.TestCase):
assert all(c < 31 for c in hits.channel_id.max()) assert all(c < 31 for c in hits.channel_id.max())
class TestDAQTimeslices(unittest.TestCase): class TestTimeslices(unittest.TestCase):
def setUp(self): def setUp(self):
self.ts = DAQReader(os.path.join(SAMPLES_DIR, self.ts = OnlineReader(os.path.join(SAMPLES_DIR,
"daq_v1.0.0.root")).timeslices "daq_v1.0.0.root")).timeslices
def test_data_lengths(self): def test_data_lengths(self):
...@@ -142,9 +142,9 @@ class TestDAQTimeslices(unittest.TestCase): ...@@ -142,9 +142,9 @@ class TestDAQTimeslices(unittest.TestCase):
assert "SN" in s assert "SN" in s
class TestDAQTimeslice(unittest.TestCase): class TestTimeslice(unittest.TestCase):
def setUp(self): def setUp(self):
self.ts = DAQReader(os.path.join(SAMPLES_DIR, self.ts = OnlineReader(os.path.join(SAMPLES_DIR,
"daq_v1.0.0.root")).timeslices "daq_v1.0.0.root")).timeslices
self.n_frames = {"L1": [69, 69, 69], "SN": [64, 66, 68]} self.n_frames = {"L1": [69, 69, 69], "SN": [64, 66, 68]}
...@@ -158,7 +158,7 @@ class TestDAQTimeslice(unittest.TestCase): ...@@ -158,7 +158,7 @@ class TestDAQTimeslice(unittest.TestCase):
class TestSummaryslices(unittest.TestCase): class TestSummaryslices(unittest.TestCase):
def setUp(self): def setUp(self):
self.ss = DAQReader(os.path.join(SAMPLES_DIR, self.ss = OnlineReader(os.path.join(SAMPLES_DIR,
"daq_v1.0.0.root")).summaryslices "daq_v1.0.0.root")).summaryslices
def test_headers(self): def test_headers(self):
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment