h5extractf in uproot v4.1.3+

Oh ok, have you created an issue in uproot4? Jim is very fast with fixing such things...

Thanks for the tip, let's see. I basically just wanted to put this on the record

Yes I understand, putting constraints on minor versions will unfortunately introduce other problems downstream, so let's see if this can be sorted out quickly

mentioned in commit ml/OrcaSong@add2b435

Yes, but since I don't know if/when I get around to opening an issue for uproot, I'll fix the version in orcasong for now (especially considering the bootcamp is tomorrow )

I propose to keep this issue here open for the time being.

yes of course!

added awaiting feedback performance labels

Any news on this @sreck?

I have not gotten around to opening an issue in uproot. I just checked the speed with the most recent version of upoot (v4.2.1), and the problem still persists: h5extractf takes 14s on a file with uproot v4.1.2, and 149s with v4.2.1...

Damn

Have you noticed a reduction in speed anywhere else in km3io going from v4.1.2 to v4.1.3 and beyond?

I have not tried yet... but we should open an issue

I am trying now by myself but a progress bar would be really helpful

I see this for like 3 minutes now:

I guess so. But Im also not sure if a progress bar could even be added here, since it might be a single line doing a read operation that takes this long. But to accelerate it, you could use this smaller mc file:

iget mc/atm_muon/KM3NeT_00000049/v6.1/reco/mcv6.1.mupage_10G.sirene.jterbr00007717.jorcarec.aanet.818.root

and use the --without-full-reco flag.

With uproot==4.1.9 and awkward==1.7.0:

░ tamasgal@silentbox-(2):Data/KM3NeT/v5.2 km3pipe
░ 11:44:40 > h5extractf datav6_ORCA_test.jchain.aanet.00009257.root
/Users/tamasgal/Dev/km3pipe/venv/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Processing header
Processing event_info
Processing tracks
/Users/tamasgal/Dev/km3pipe/venv/lib/python3.9/site-packages/awkward/_connect/_numba/__init__.py:30: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if not checked_version and distutils.version.LooseVersion(
/Users/tamasgal/Dev/km3pipe/venv/lib/python3.9/site-packages/awkward/_connect/_numba/__init__.py:32: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  ) < distutils.version.LooseVersion("0.50"):
Processing hits
Processing mc_hits
Processing mc_tracks
Completed in 734.0 s
Provenance information has been written to 'datav6_ORCA_test.jchain.aanet.00009257.root.h5.prov.json'

With uproot-4.2.1 and awkward==1.8.0:

░ tamasgal@silentbox-(2):Data/KM3NeT/v5.2 km3pipe took 6s
░ 11:59:26 > h5extractf datav6_ORCA_test.jchain.aanet.00009257.root
/Users/tamasgal/Dev/km3pipe/venv/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
2022-03-14 11:59:58 WARNING ++ Provenance: Provenance output file (datav6_ORCA_test.jchain.aanet.00009257.root.h5.prov.json) exists and will be overwritten upon exit.
Processing header
Processing event_info
Processing tracks
Processing hits
Processing mc_hits
Processing mc_tracks
Completed in 730.8 s
Provenance information has been written to 'datav6_ORCA_test.jchain.aanet.00009257.root.h5.prov.json'

With uproot-4.1.2 and awkward==1.8.0:

░ tamasgal@silentbox-(2):Data/KM3NeT/v5.2 km3pipe
░ 12:20:20 > h5extractf datav6_ORCA_test.jchain.aanet.00009257.root
/Users/tamasgal/Dev/km3pipe/venv/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
2022-03-14 12:20:26 WARNING ++ Provenance: Provenance output file (datav6_ORCA_test.jchain.aanet.00009257.root.h5.prov.json) exists and will be overwritten upon exit.
Processing header
Processing event_info
Processing tracks
Processing hits
Processing mc_hits
Processing mc_tracks
Completed in 59.5 s
Provenance information has been written to 'datav6_ORCA_test.jchain.aanet.00009257.root.h5.prov.json'

And yes, the breakpoint is uproot-4.1.3:

░ tamasgal@silentbox-(2):Data/KM3NeT/v5.2 km3pipe
░ 12:46:26 > h5extractf datav6_ORCA_test.jchain.aanet.00009257.root
/Users/tamasgal/Dev/km3pipe/venv/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
2022-03-14 12:46:31 WARNING ++ Provenance: Provenance output file (datav6_ORCA_test.jchain.aanet.00009257.root.h5.prov.json) exists and will be overwritten upon exit.
Processing header
Processing event_info
Processing tracks
Processing hits
Processing mc_hits
Processing mc_tracks
Completed in 715.7 s
Provenance information has been written to 'datav6_ORCA_test.jchain.aanet.00009257.root.h5.prov.json'

OK, I created an issue: https://github.com/scikit-hep/uproot4/issues/572

I think I boiled it down quite nicely, so let's see...

Nice work, thanks a lot for this!

There is a working fix now as an open PR: https://github.com/scikit-hep/uproot4/pull/573 if it gets merged, we are good

h5extractf in uproot v4.1.3+

Child items ...

Activity