uproot4 integration

added api label

created merge request !39 to address this issue

mentioned in merge request !39

Oh dear... @zaly everything has been changed in uproot4

Oh :-O !!! I ll take a look!

I have some promising news. The 58-uproot4-integration of km3io now implements a half-way-done uproot4 usage. Luckily the h5extract fully works with that version and the conversion happens in ~30minutes for a huge file with 143k events:

143381 cycles drained in 31.390205min (CPU 31.185114min). Memory peak: 3819.62 MB
Statistics are based on the last 100000 cycles.

Forgot to pink @dguderian

changed the description

marked the checklist item implement branch parsing and attribute mapping with uproot4 as completed

cool! thank you!!

Ah, sorry for the spam, I also wanted to ping @vpestel

So as you can see, a full conversion (reading ROOT and writing HDF5) of all the supported information (at this moment, basically everything) takes half an hour.

Without writing HDF5 (just parsing and creating all the Tables in memory) takes 8 minutes for the whole file.

Just looping over the ROOT file in the Pipeline takes 45 seconds

Here is the full list of the timeit statistics for the whole processing and converting. Most of the time is spent in the RecoTableTabulator and the HDF5Sink. The 4GB memory peak is a bit annoying, but it seems it's coming from uproot4 itself, I am investigating.

143381 cycles drained in 31.992660min (CPU 31.796684min). Memory peak: 3906.29 MB
Statistics are based on the last 100000 cycles.
  wall  mean: 0.013123s  medi: 0.012734s  min: 0.005838s  max: 0.992226s  std: 0.007402s
  CPU   mean: 0.013042s  medi: 0.012731s  min: 0.005838s  max: 0.308669s  std: 0.004712s
OfflinePump - process: 42.279s (CPU 42.385s) - finish: 0.027s (CPU 0.027s)
  wall  mean: 0.000423s  medi: 0.000324s  min: 0.000254s  max: 0.296459s  std: 0.004437s
  CPU   mean: 0.000424s  medi: 0.000325s  min: 0.000255s  max: 0.296453s  std: 0.004436s
StatusBar - process: 0.043s (CPU 0.044s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000030s  medi: 0.000028s  min: 0.000024s  max: 0.000085s  std: 0.000005s
  CPU   mean: 0.000030s  medi: 0.000029s  min: 0.000025s  max: 0.000075s  std: 0.000005s
MemoryObserver - process: 0.007s (CPU 0.007s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000025s  medi: 0.000022s  min: 0.000017s  max: 0.000048s  std: 0.000007s
  CPU   mean: 0.000025s  medi: 0.000023s  min: 0.000018s  max: 0.000051s  std: 0.000007s
OfflineHeaderTabulator - process: 14.612s (CPU 14.697s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000146s  medi: 0.000138s  min: 0.000122s  max: 0.003456s  std: 0.000023s
  CPU   mean: 0.000147s  medi: 0.000139s  min: 0.000122s  max: 0.000483s  std: 0.000020s
EventInfoTabulator - process: 16.754s (CPU 16.838s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000168s  medi: 0.000157s  min: 0.000142s  max: 0.002303s  std: 0.000026s
  CPU   mean: 0.000168s  medi: 0.000158s  min: 0.000143s  max: 0.000456s  std: 0.000025s
Offline - process: 70.890s (CPU 70.990s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000709s  medi: 0.000695s  min: 0.000602s  max: 0.003691s  std: 0.000089s
  CPU   mean: 0.000710s  medi: 0.000696s  min: 0.000603s  max: 0.001715s  std: 0.000088s
MC - process: 0.410s (CPU 0.426s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000004s  medi: 0.000004s  min: 0.000003s  max: 0.000072s  std: 0.000001s
  CPU   mean: 0.000004s  medi: 0.000004s  min: 0.000003s  max: 0.000036s  std: 0.000001s
MCTracksTabulator - process: 0.276s (CPU 0.293s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000003s  medi: 0.000003s  min: 0.000002s  max: 0.000027s  std: 0.000001s
  CPU   mean: 0.000003s  medi: 0.000003s  min: 0.000002s  max: 0.000027s  std: 0.000001s
RecoTracksTabulator - process: 226.237s (CPU 226.348s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.002262s  medi: 0.002199s  min: 0.000002s  max: 0.005552s  std: 0.000268s
  CPU   mean: 0.002263s  medi: 0.002200s  min: 0.000002s  max: 0.005556s  std: 0.000268s
HDF5Sink - process: 935.457s (CPU 927.551s) - finish: 34.201s (CPU 32.092s)
  wall  mean: 0.009355s  medi: 0.009120s  min: 0.004539s  max: 0.988659s  std: 0.005868s
  CPU   mean: 0.009276s  medi: 0.009119s  min: 0.004541s  max: 0.107984s  std: 0.001360s

Wow very impressive, thank you Tom! Too bad I just used 80 days of cpu time to convert stuff Really cool anyways of course

Yo man, insane performance gain^^ Thanks!

created merge request !47 (merged) to address this issue

mentioned in merge request !47 (merged)

Alright, finished. Let's merge to master, tag version 0.18 and wait for your complaints about broken scripts

I tried to stick to the existing API and avoided to break it, but there are a few tiny changes here and there. Especially the usr stuff is now different but it now has a massive performance improevement.

Instead of

f.events.usr.RecoQuality

you now use the free function km3io.tools.usr(obj, attribute). It sounds like a step back but it's actually a much better way and I have full control over the optimisations behind the scenes. So this is how it looks like now:

>>> km3io.tools.usr(f.events, "RecoQuality")
<Array [85.5, 68.7, 50.2] type='3 * float64'>
# or simply (since `f` is now the same as `f.events`)
>>> km3io.tools.usr(f, "RecoQuality")
<Array [85.5, 68.7, 50.2] type='3 * float64'>

marked the checklist item best track methods to work with the new uproot4 structures as completed

marked the checklist item fancy indexing on events/hits/tracks as completed

marked the checklist item all the other tools from @zaly in tools.py as completed

I mean version 0.19

closed with merge request !47 (merged)

mentioned in commit 37b95aa2

uproot4 integration

Designs

Child items ...

Activity