README.rst

The km3io Python package
========================

.. image:: https://git.km3net.de/km3py/km3io/badges/master/build.svg
    :target: https://git.km3net.de/km3py/km3io/pipelines

.. image:: https://git.km3net.de/km3py/km3io/badges/master/coverage.svg
    :target: https://km3py.pages.km3net.de/km3io/coverage

.. image:: https://api.codacy.com/project/badge/Grade/0660338483874475ba04f324de2123ec
    :target: https://www.codacy.com/manual/tamasgal/km3io?utm_source=github.com&utm_medium=referral&utm_content=KM3NeT/km3io&utm_campaign=Badge_Grade

.. image:: https://examples.pages.km3net.de/km3badges/docs-latest-brightgreen.svg
    :target: https://km3py.pages.km3net.de/km3io

This software provides a set of Python classes to read KM3NeT ROOT files
without having ROOT, Jpp or aanet installed. It only depends on Python 3.5+ and the amazing `uproot <https://github.com/scikit-hep/uproot>`__ package and gives you access to the data via numpy arrays.

It's very easy to use and according to the `uproot <https://github.com/scikit-hep/uproot>`__ benchmarks, it is able to outperform the ROOT I/O performance. 

**Note:** Beware that this package is in the development phase, so the API will change until version ``1.0.0`` is released!

Installation
============

Install km3io using pip::

    pip install km3io 

To get the latest (stable) development release::

    pip install git+https://git.km3net.de/km3py/km3io.git

**Reminder:** km3io is **not** dependent on aanet, ROOT or Jpp! 

Questions
=========

If you have a question about km3io, please proceed as follows:

- Read the documentation below.
- Explore the `examples <https://km3py.pages.km3net.de/km3io/examples.html>`__ in the documentation.
- Haven't you found an answer to your question in the documentation, post a git issue with your question showing us an example of what you have tried first, and what you would like to do.
- Have you noticed a bug, please post it in a git issue, we appreciate your contribution.

Tutorial
========

**Table of contents:**

* `Introduction <#introduction>`__

  * `Overview of daq files <#overview-of-daq-files>`__

  * `Overview of offline files <#overview-of-offline-files>`__

* `DAQ files reader <#daq-files-reader>`__

* `Offline files reader <#offline-file-reader>`__

  * `reading events data <#reading-events-data>`__

  * `reading hits data <#reading-hits-data>`__

  * `reading tracks data <#reading-tracks-data>`__

  * `reading mc hits data <#reading-mc-hits-data>`__

  * `reading mc tracks data <#reading-mc-tracks-data>`__


Introduction
------------

Most of km3net data is stored in root files. These root files are either created with `Jpp <https://git.km3net.de/common/jpp>`__ or `aanet <https://git.km3net.de/common/aanet>`__ software. A root file created with 
`Jpp <https://git.km3net.de/common/jpp>`__ is often referred to as "a Jpp root file". Similarly, a root file created with `aanet <https://git.km3net.de/common/aanet>`__ is often referred to as "an aanet file". In km3io, an aanet root file will always be reffered to as an ``offline file``, while a Jpp root file will always be referred to as a ``daq file``.

km3io is a Python package that provides a set of classes (``DAQReader`` and ``OfflineReader``) to read both daq root files and offline root files without any dependency to aanet, Jpp or ROOT. 

Data in km3io is often returned as a "lazyarray", a "jagged lazyarray" or a `Numpy <https://docs.scipy.org/doc/numpy>`__ array. A lazyarray is an array-like object that reads data on demand! In a lazyarray, only the first and the last chunks of data are read in memory. A lazyarray can be used with all Numpy's universal `functions <https://docs.scipy.org/doc/numpy/reference/ufuncs.html>`__. Here is how a lazyarray looks like:

.. code-block:: python3

    # <ChunkedArray [5971 5971 5971 ... 5971 5971 5971] at 0x7fb2341ad810>


A jagged array, is a 2+ dimentional array with different arrays lengths. In other words, a jagged array is an array of arrays of different sizes. So a jagged lazyarray is simply a jagged array of lazyarrays with different sizes. Here is how a jagged lazyarray looks like:


.. code-block:: python3

    # <JaggedArray [[102 102 102 ... 11517 11518 11518] [] [101 101 102 ... 11518 11518 11518] ... [101 101 102 ... 11516 11516 11517] [] [101 101 101 ... 11517 11517 11518]] at 0x7f74b0ef8810>


Overview of daq files
"""""""""""""""""""""
# info needed here

Overview of offline files
"""""""""""""""""""""""""

# info needed here

DAQ files reader
----------------

# an update is needed here?

Currently only events (the ``KM3NET_EVENT`` tree) are supported but timeslices and summaryslices will be implemented very soon.

Let's have a look at some ORCA data (``KM3NeT_00000044_00005404.root``)

To get a lazy ragged array of the events:

.. code-block:: python3

  import km3io as ki
  events = ki.DAQReader("KM3NeT_00000044_00005404.root").events


That's it! Now let's have a look at the hits data:

.. code-block:: python3

  >>> events
  Number of events: 17023
  >>> events[23].snapshot_hits.tot
  array([28, 22, 17, 29,  5, 27, 24, 26, 21, 28, 26, 21, 26, 24, 17, 28, 23,29, 27, 24, 23, 26, 29, 25, 18, 28, 24, 28, 26, 20, 25, 31, 28, 23, 26, 21, 30, 33, 27, 16, 23, 24, 19, 24, 27, 22, 23, 21, 25, 16, 28, 22, 22, 29, 24, 29, 24, 24, 25, 25, 21, 31, 26, 28, 30, 42, 28], dtype=uint8)


Offline files reader
--------------------

Let's have a look at some muons data from ORCA 4 lines simulations - run id 5971 (``datav6.0test.jchain.aanet.00005971.root``). 

**Note:** this file was cropped to 10 events only, so don't be surprised in this tutorial if you see few events in the file.

First, let's read our file:

.. code-block:: python3

  >>> import km3io as ki
  >>> file = 'datav6.0test.jchain.aanet.00005971.root'
  >>> r = ki.OfflineReader(file)
  <km3io.aanet.OfflineReader at 0x7f24cc2bd550>

and that's it! Note that `file` can be either an str of your file path, or a path-like object. 

To explore all the available branches in our offline file: 

.. code-block:: python3

  >>> r.keys
  Events keys are:
        id
        det_id
        mc_id
        run_id
        mc_run_id
        frame_index
        trigger_mask
        trigger_counter
        overlays
        hits
        trks
        w
        w2list
        w3list
        mc_t
        mc_hits
        mc_trks
        comment
        index
        flags
        t.fSec
        t.fNanoSec
  Hits keys are:
        hits.id
        hits.dom_id
        hits.channel_id
        hits.tdc
        hits.tot
        hits.trig
        hits.pmt_id
        hits.t
        hits.a
        hits.pos.x
        hits.pos.y
        hits.pos.z
        hits.dir.x
        hits.dir.y
        hits.dir.z
        hits.pure_t
        hits.pure_a
        hits.type
        hits.origin
        hits.pattern_flags
  Tracks keys are:
        trks.fUniqueID
        trks.fBits
        trks.id
        trks.pos.x
        trks.pos.y
        trks.pos.z
        trks.dir.x
        trks.dir.y
        trks.dir.z
        trks.t
        trks.E
        trks.len
        trks.lik
        trks.type
        trks.rec_type
        trks.rec_stages
        trks.status
        trks.mother_id
        trks.fitinf
        trks.hit_ids
        trks.error_matrix
        trks.comment
  Mc hits keys are:
        mc_hits.id
        mc_hits.dom_id
        mc_hits.channel_id
        mc_hits.tdc
        mc_hits.tot
        mc_hits.trig
        mc_hits.pmt_id
        mc_hits.t
        mc_hits.a
        mc_hits.pos.x
        mc_hits.pos.y
        mc_hits.pos.z
        mc_hits.dir.x
        mc_hits.dir.y
        mc_hits.dir.z
        mc_hits.pure_t
        mc_hits.pure_a
        mc_hits.type
        mc_hits.origin
        mc_hits.pattern_flags
  Mc tracks keys are:
        mc_trks.fUniqueID
        mc_trks.fBits
        mc_trks.id
        mc_trks.pos.x
        mc_trks.pos.y
        mc_trks.pos.z
        mc_trks.dir.x
        mc_trks.dir.y
        mc_trks.dir.z
        mc_trks.t
        mc_trks.E
        mc_trks.len
        mc_trks.lik
        mc_trks.type
        mc_trks.rec_type
        mc_trks.rec_stages
        mc_trks.status
        mc_trks.mother_id
        mc_trks.fitinf
        mc_trks.hit_ids
        mc_trks.error_matrix
        mc_trks.comment

In an offline file, there are 5 main trees with data: 

* events tree
* hits tree
* tracks tree
* mc hits tree
* mc tracks tree

with km3io, these trees can be accessed with a simple tab completion: 

.. image:: https://git.km3net.de/km3py/km3io/raw/master/examples/pictures/reader.png

In the following, we will explore each tree using km3io package. 

reading events data
"""""""""""""""""""

to read data in events tree with km3io: 

.. code-block:: python3

  >>> r.events
  <OfflineEvents: 10 parsed events>

to get the total number of events in the events tree:

.. code-block:: python3

  >>> len(r.events)
  10

the branches stored in the events tree in an offline file can be easily accessed with a tab completion as seen below:

.. image:: https://git.km3net.de/km3py/km3io/raw/master/examples/pictures/events.png

to get data from the events tree, chose any branch of interest with the tab completion, the following is a non exaustive set of examples. 

to get event ids:

.. code-block:: python3

    >>> r.events.id
    <ChunkedArray [1 2 3 ... 8 9 10] at 0x7f249eeb6f10>

to get detector ids:

.. code-block:: python3

    >>> r.events.det_id
    <ChunkedArray [44 44 44 ... 44 44 44] at 0x7f249eeba050>

to get frame_index:

.. code-block:: python3

    >>> r.events.frame_index
    <ChunkedArray [182 183 202 ... 185 185 204] at 0x7f249eeba410>

to get snapshot hits:

.. code-block:: python3

    >>> r.events.hits
    <ChunkedArray [176 125 318 ... 84 255 105] at 0x7f249eebaa10>

to illustrate the strength of this data structure, we will play around with `r.events.hits` using Numpy universal `functions <https://docs.scipy.org/doc/numpy/reference/ufuncs.html>`__. 

.. code-block:: python3

    >>> import numpy as np
    >>> np.log(r.events.hits)
    <ChunkedArray [5.170483995038151 4.8283137373023015 5.762051382780177 ... 4.430816798843313 5.541263545158426 4.653960350157523] at 0x7f249b8ebb90>

to get all data from one specific event (for example event 0):

.. code-block:: python3

    >>> r.events[0]
    offline event:
          id                  :               1
          det_id              :              44
          mc_id               :               0
          run_id              :            5971
          mc_run_id           :               0
          frame_index         :             182
          trigger_mask        :              22
          trigger_counter     :               0
          overlays            :              60
          hits                :             176
          trks                :              56
          w                   :              []
          w2list              :              []
          w3list              :              []
          mc_t                :             0.0
          mc_hits             :               0
          mc_trks             :               0
          comment             :             b''
          index               :               0
          flags               :               0
          t_fSec              :      1567036818
          t_fNanoSec          :       200000000

to get a specific value from event 0, for example the number of overlays:

.. code-block:: python3

    >>> r.events[0].overlays
    60

or the number of hits: 

.. code-block:: python3

    >>> r.events[0].hits
    176


reading hits data
"""""""""""""""""

to read data in hits tree with km3io:

.. code-block:: python3

    >>> r.hits
    <OfflineHits: 10 parsed elements>

this shows that in our offline file, there are 10 events, with each event is associated a hits trees. 

to have access to all data in a specific branche from the hits tree, you can use the tab completion:

.. image:: https://git.km3net.de/km3py/km3io/raw/master/examples/pictures/hits.png

to get ALL the dom ids in all hits trees in our offline file:

.. code-block:: python3

    >>> r.hits.dom_id
    <ChunkedArray [[806451572 806451572 806451572 ... 809544061 809544061 809544061] [806451572 806451572 806451572 ... 809524432 809526097 809544061] [806451572 806451572 806451572 ... 809544061 809544061 809544061] ... [806451572 806455814 806465101 ... 809526097 809544058 809544061] [806455814 806455814 806455814 ... 809544061 809544061 809544061] [806455814 806455814 806455814 ... 809544058 809544058 809544061]] at 0x7f249eebac50>

to get ALL the time over threshold (tot) in all hits trees in our offline file:

.. code-block:: python3

    >>> r.hits.tot
    <ChunkedArray [[24 30 22 ... 38 26 23] [29 26 22 ... 26 28 24] [27 19 13 ... 27 24 16] ... [22 22 9 ... 27 32 27] [30 32 17 ... 30 24 29] [27 41 36 ... 29 24 28]] at 0x7f249eec9050>


if you are interested in a specific event (let's say event 0), you can access the corresponding hits tree by doing the following:

.. code-block:: python3

    >>> r[0].hits
    <OfflineHits: 176 parsed elements>

notice that now there are 176 parsed elements (as opposed to 10 elements parsed when r.hits is called). This means that in event 0 there are 176 hits! To get the dom ids from this event:

.. code-block:: python3

    >>> r[0].hits.dom_id
    array([806451572, 806451572, 806451572, 806451572, 806455814, 806455814,
       806455814, 806483369, 806483369, 806483369, 806483369, 806483369,
       806483369, 806483369, 806483369, 806483369, 806483369, 806487219,
       806487226, 806487231, 806487231, 808432835, 808435278, 808435278,
       808435278, 808435278, 808435278, 808447180, 808447180, 808447180,
       808447180, 808447180, 808447180, 808447180, 808447180, 808447186,
       808451904, 808451904, 808472265, 808472265, 808472265, 808472265,
       808472265, 808472265, 808472265, 808472265, 808488895, 808488990,
       808488990, 808488990, 808488990, 808488990, 808489014, 808489014,
       808489117, 808489117, 808489117, 808489117, 808493910, 808946818,
       808949744, 808951460, 808951460, 808951460, 808951460, 808951460,
       808956908, 808956908, 808959411, 808959411, 808959411, 808961448,
       808961448, 808961504, 808961504, 808961655, 808961655, 808961655,
       808964815, 808964815, 808964852, 808964908, 808969857, 808969857,
       808969857, 808969857, 808969857, 808972593, 808972698, 808972698,
       808972698, 808974758, 808974758, 808974758, 808974758, 808974758,
       808974758, 808974758, 808974758, 808974758, 808974758, 808974758,
       808974773, 808974773, 808974773, 808974773, 808974773, 808974972,
       808974972, 808976377, 808976377, 808976377, 808979567, 808979567,
       808979567, 808979721, 808979721, 808979721, 808979721, 808979721,
       808979721, 808979721, 808979729, 808979729, 808979729, 808981510,
       808981510, 808981510, 808981510, 808981672, 808981672, 808981672,
       808981672, 808981672, 808981672, 808981672, 808981672, 808981672,
       808981672, 808981672, 808981672, 808981672, 808981672, 808981672,
       808981672, 808981672, 808981812, 808981812, 808981812, 808981864,
       808981864, 808982005, 808982005, 808982005, 808982018, 808982018,
       808982018, 808982041, 808982041, 808982077, 808982077, 808982547,
       808982547, 808982547, 808997793, 809006037, 809524432, 809526097,
       809526097, 809544061, 809544061, 809544061, 809544061, 809544061,
       809544061, 809544061], dtype=int32

to get all data of a specific hit (let's say hit 0) from event 0:

.. code-block:: python3

    >>>r[0].hits[0]
    offline hit:
          id                  :               0
          dom_id              :       806451572
          channel_id          :               8
          tdc                 :               0
          tot                 :              24
          trig                :               1
          pmt_id              :               0
          t                   :      70104010.0
          a                   :             0.0
          pos_x               :             0.0
          pos_y               :             0.0
          pos_z               :             0.0
          dir_x               :             0.0
          dir_y               :             0.0
          dir_z               :             0.0
          pure_t              :             0.0
          pure_a              :             0.0
          type                :               0
          origin              :               0
          pattern_flags       :               0

to get a specific value from hit 0 in event 0, let's say for example the dom id:

.. code-block:: python3

    >>>r[0].hits[0].dom_id
    806451572

reading tracks data
"""""""""""""""""""

to read data in tracks tree with km3io:

.. code-block:: python3

    >>> r.tracks
    <OfflineTracks: 10 parsed elements>

this shows that in our offline file, there are 10 parsed elements (events), each event is associated with tracks data. 

to have access to all data in a specific branche from the tracks tree, you can use the tab completion:

.. image:: https://git.km3net.de/km3py/km3io/raw/master/examples/pictures/tracks.png

to get ALL the cos(zenith angle) in all tracks tree in our offline file:

.. code-block:: python3

    >>> r.tracks.dir_z
    <ChunkedArray [[-0.872885221293917 -0.872885221293917 -0.872885221293917 ... -0.6631226836266504 -0.5680647731737454 -0.5680647731737454] [-0.8351996698137462 -0.8351996698137462 -0.8351996698137462 ... -0.7485107718446855 -0.8229838871876581 -0.239315690284641] [-0.989148723802379 -0.989148723802379 -0.989148723802379 ... -0.9350162572437829 -0.88545604390297 -0.88545604390297] ... [-0.5704611045902105 -0.5704611045902105 -0.5704611045902105 ... -0.9350162572437829 -0.4647231989130516 -0.4647231989130516] [-0.9779941383490359 -0.9779941383490359 -0.9779941383490359 ... -0.88545604390297 -0.88545604390297 -0.8229838871876581] [-0.7396916780974963 -0.7396916780974963 -0.7396916780974963 ... -0.6631226836266504 -0.7485107718446855 -0.7485107718446855]] at 0x7f249eed2090>

to get ALL the tracks likelihood in our offline file:

.. code-block:: python3

    >>> r.tracks.lik
    <ChunkedArray [[294.6407542676734 294.6407542676734 294.6407542676734 ... 67.81221253265059 67.7756405143316 67.77250505700384] [96.75133289411137 96.75133289411137 96.75133289411137 ... 39.21916536442286 39.184645826013806 38.870325146341884] [560.2775306614813 560.2775306614813 560.2775306614813 ... 118.88577278801066 118.72271313687405 117.80785995187605] ... [71.03251451148226 71.03251451148226 71.03251451148226 ... 16.714140573909347 16.444395245214945 16.34639241716669] [326.440133294878 326.440133294878 326.440133294878 ... 87.79818671079849 87.75488082571873 87.74839444768625] [159.77779654216795 159.77779654216795 159.77779654216795 ... 33.8669134999348 33.821631538334984 33.77240735670646]] at 0x7f249eed2590>


if you are interested in a specific event (let's say event 0), you can access the corresponding tracks tree by doing the following:

.. code-block:: python3

    >>> r[0].tracks
    <OfflineTracks: 56 parsed elements>

notice that now there are 56 parsed elements (as opposed to 10 elements parsed when r.tracks is called). This means that in event 0 there is data about 56 possible tracks! To get the tracks likelihood from this event:

.. code-block:: python3

    >>> r[0].tracks.lik
    array([294.64075427, 294.64075427, 294.64075427, 291.64653113,
       291.27392663, 290.69031512, 289.19290546, 289.08449217,
       289.03373947, 288.19030836, 282.92343367, 282.71527118,
       282.10762402, 280.20553861, 275.93183966, 273.01809111,
       257.46433694, 220.94357656, 194.99426403, 190.47809685,
        79.95235686,  78.94389763,  78.90791169,  77.96122466,
        77.9579604 ,  76.90769883,  75.97546175,  74.91530508,
        74.9059469 ,  72.94007716,  72.90467038,  72.8629316 ,
        72.81280833,  72.80229533,  72.78899435,  71.82404165,
        71.80085542,  71.71028058,  70.91130096,  70.89150223,
        70.85845637,  70.79081796,  70.76929743,  69.80667603,
        69.64058976,  68.93085058,  68.84304037,  68.83154232,
        68.79944298,  68.79019375,  68.78581291,  68.72340328,
        67.86628937,  67.81221253,  67.77564051,  67.77250506])

to get all data of a specific track (let's say track 0) from event 0:

.. code-block:: python3

    >>>r[0].tracks[0]
    offline track:
          fUniqueID                      :                           0
          fBits                          :                    33554432
          id                             :                           1
          pos_x                          :            445.835395997812
          pos_y                          :           615.1089636184813
          pos_z                          :           125.1448339836911
          dir_x                          :          0.0368711082700674
          dir_y                          :        -0.48653048395923415
          dir_z                          :          -0.872885221293917
          t                              :           70311446.46401498
          E                              :           99.10458562488608
          len                            :                         0.0
          lik                            :           294.6407542676734
          type                           :                           0
          rec_type                       :                        4000
          rec_stages                     :                [1, 3, 5, 4]
          status                         :                           0
          mother_id                      :                          -1
          hit_ids                        :                          []
          error_matrix                   :                          []
          comment                        :                           0
          JGANDALF_BETA0_RAD             :        0.004957442219414389
          JGANDALF_BETA1_RAD             :        0.003417848024252858
          JGANDALF_CHI2                  :          -294.6407542676734
          JGANDALF_NUMBER_OF_HITS        :                       142.0
          JENERGY_ENERGY                 :           99.10458562488608
          JENERGY_CHI2                   :     1.7976931348623157e+308
          JGANDALF_LAMBDA                :      4.2409761837248484e-12
          JGANDALF_NUMBER_OF_ITERATIONS  :                        10.0
          JSTART_NPE_MIP                 :           24.88469697331908
          JSTART_NPE_MIP_TOTAL           :           55.88169412579765
          JSTART_LENGTH_METRES           :           98.89582506402911
          JVETO_NPE                      :                         0.0
          JVETO_NUMBER_OF_HITS           :                         0.0
          JENERGY_MUON_RANGE_METRES      :           344.9767431592819
          JENERGY_NOISE_LIKELIHOOD       :         -333.87773581129136
          JENERGY_NDF                    :                      1471.0
          JENERGY_NUMBER_OF_HITS         :                       101.0

to get a specific value from track 0 in event 0, let's say for example the liklihood:

.. code-block:: python3

    >>>r[0].tracks[0].lik
    294.6407542676734


reading mc hits data
""""""""""""""""""""

to read mc hits data:

.. code-block:: python3

    >>>r.mc_hits
    <OfflineHits: 10 parsed elements>

that's it! All branches in mc hits tree can be accessed in the exact same way described in the section `reading hits data <#reading-hits-data>`__ . All data is easily accesible and if you are stuck, hit tab key to see all the available branches:

.. image:: https://git.km3net.de/km3py/km3io/raw/master/examples/pictures/mc_hits.png

reading mc tracks data
""""""""""""""""""""""

to read mc tracks data:

.. code-block:: python3

    >>>r.mc_tracks
    <OfflineTracks: 10 parsed elements>

that's it! All branches in mc tracks tree can be accessed in the exact same way described in the section `reading tracks data <#reading-tracks-data>`__ . All data is easily accesible and if you are stuck, hit tab key to see all the available branches:

.. image:: https://git.km3net.de/km3py/km3io/raw/master/examples/pictures/mc_tracks.png