issue with hdf5pump or h5extractor
Hi @tgal,
I have converted all ARCA v5.1 MC files with the new h5extractor
, and also some new ARCA v6 MC production files (nueCC and nutau in particular).
Everything seemed to be fine until we tried to convert the h5 files into ML format using OrcaSong. Here, both @gvermarien and I noticed that for some files (note: not ALL files but some, depending on the flavour) we get this error:
2021-04-12 21:45:39 ERROR ++ km3pipe.io.hdf5: Could not determine HDF5 format version: '/sps/km3net/repo/mc/atm_neutrino/KM3NeT_-00000001_20171212/v5.1_res/h5/mcv5.1_res.genhen_numuCC.km3_AAv1.jte.jchain.aashower.146.h5'.You may encounter unexpected errors! Good luck...
2021-04-12 21:46:00 ERROR ++ Blob: No key named 'McHits' found in Blob.
Available keys: EventInfo, GroupInfo, McTracks, RawHeader, BestAashower, BestJmuon, RecStages, Tracks, Header
Traceback (most recent call last):
File "/sps/km3net/users/adomi/GNNs//OrcaSong/extract.py", line 20, in <module>
fg.run(inputfile,outputfile)
File "/sps/km3net/users/adomi/GNNs/OrcaSong/orcasong/core.py", line 143, in run
summary = pipe.drain()
File "/pbs/home/a/adomi/mypython/lib/python3.7/site-packages/thepipe/core.py", line 451, in drain
results = self._drain(cycles)
File "/pbs/home/a/adomi/mypython/lib/python3.7/site-packages/thepipe/core.py", line 383, in _drain
new_blob = module(blob_to_send)
File "/pbs/home/a/adomi/mypython/lib/python3.7/site-packages/thepipe/core.py", line 187, in __call__
return self.process(*args, **kwargs)
File "/sps/km3net/users/adomi/km3pipe/km3modules/mc.py", line 115, in process
mc_hits = blob[self.mc_hits_key]
File "/pbs/home/a/adomi/mypython/lib/python3.7/site-packages/thepipe/core.py", line 51, in __getitem__
val = OrderedDict.__getitem__(self, key)
KeyError: 'McHits'
Closing remaining open files:/sps/km3net/repo/mc/atm_neutrino/KM3NeT_-00000001_20171212/v5.1_res/h5/mcv5.1_res.genhen_numuCC.km3_AAv1.jte.jchain.aashower.146.h5...done/pbs/home/a/adomi/gnn/NuMuCC/ML_mcv5.1_res.genhen_numuCC.km3_AAv1.jte.jchain.aashower.146.h5...done
Please note the initial:
ERROR ++ km3pipe.io.hdf5: Could not determine HDF5 format version: '/sps/km3net/repo/mc/atm_neutrino/KM3NeT_-00000001_20171212/v5.1_res/h5/mcv5.1_res.genhen_numuCC.km3_AAv1.jte.jchain.aashower.146.h5'.You may encounter unexpected errors! Good luck...
And it seems to be a problem with 'McHits'.
However, if I look at this file: /sps/km3net/repo/mc/atm_neutrino/KM3NeT_-00000001_20171212/v5.1_res/h5/mcv5.1_res.genhen_numuCC.km3_AAv1.jte.jchain.aashower.146.h5
I get:
ptdump /sps/km3net/repo/mc/atm_neutrino/KM3NeT_-00000001_20171212/v5.1_res/h5/mcv5.1_res.genhen_numuCC.km3_AAv1.jte.jchain.aashower.146.h5
/ (RootGroup) 'KM3NeT'
/event_info (Table(2000,), fletcher32, shuffle, zlib(5)) 'EventInfo'
/group_info (Table(2000,), fletcher32, shuffle, zlib(5)) 'Group Info'
/mc_tracks (Table(25292,), fletcher32, shuffle, zlib(5)) 'McTracks'
/raw_header (Table(36,), fletcher32, shuffle, zlib(5)) 'RawHeader'
/hits (Group) 'Hits'
/hits/channel_id (EArray(8518364,), fletcher32, shuffle, zlib(5)) 'Channel_id'
/hits/dom_id (EArray(8518364,), fletcher32, shuffle, zlib(5)) 'Dom_id'
/hits/group_id (EArray(8518364,), fletcher32, shuffle, zlib(5)) 'Group_id'
/hits/time (EArray(8518364,), fletcher32, shuffle, zlib(5)) 'Time'
/hits/tot (EArray(8518364,), fletcher32, shuffle, zlib(5)) 'Tot'
/hits/triggered (EArray(8518364,), fletcher32, shuffle, zlib(5)) 'Triggered'
/mc_hits (Group) 'McHits'
/mc_hits/a (EArray(327237,), fletcher32, shuffle, zlib(5)) 'A'
/mc_hits/group_id (EArray(327237,), fletcher32, shuffle, zlib(5)) 'Group_id'
/mc_hits/origin (EArray(327237,), fletcher32, shuffle, zlib(5)) 'Origin'
/mc_hits/pmt_id (EArray(327237,), fletcher32, shuffle, zlib(5)) 'Pmt_id'
/mc_hits/time (EArray(327237,), fletcher32, shuffle, zlib(5)) 'Time'
/reco (Group) ''
/reco/best_aashower (Table(2000,), fletcher32, shuffle, zlib(5)) 'Reco best_aashower'
/reco/best_jmuon (Table(1910,), fletcher32, shuffle, zlib(5)) 'Reco best_jmuon'
/reco/rec_stages (Table(259298,), fletcher32, shuffle, zlib(5)) 'Reconstruction Stages'
/reco/tracks (Table(114692,), fletcher32, shuffle, zlib(5)) 'Reco tracks'
And here you see there are the /mc_hits (Group) 'McHits'
. So I don't understand where the error comes from.
The point is: if this would have happened for all the files, I would have thought it was a problem with OrcaSong. In fact I talked also with @dguderian. However, since instead for some other files the OrcaSong conversion works, and since the error clearly states ERROR ++ km3pipe.io.hdf5: Could not determine HDF5 format version:
it may be something related to the HDF5pump
(not sure of course).
Do you maybe have an idea on what is the actual problem?