error when using new extractor

changed the description

I have tried with a different file:

/sps/km3net/repo/mc/atm_neutrino/KM3NeT_-00000001_20171212/v5.1/reco/mcv5.1.genhen_numuCC.sirene.jte.jchain.aashower.1.root

with the command:

h5extract -o test_ARCA_numuCC.h5 --offline-header --event-info --offline-hits --mc-hits --mc-tracks --mc-tracks-usr-data --reco-tracks /sps/km3net/repo/mc/atm_neutrino/KM3NeT_-00000001_20171212/v5.1/reco/mcv5.1.genhen_numuCC.sirene.jte.jchain.aashower.1.root --aashower-legacy --best-tracks

and then with:

python extract.py /sps/km3net/users/adomi/km3pipe/test_ARCA_numuCC.h5 /pbs/throng/km3net/detectors/KM3NeT_-00000001_20171212.detx newTestARCA.h5

where extract.py has:

fg = FileGraph(max_n_hits=5000,extractor=get_neutrino_mc_info_extr(inputfile),det_file=detectorfile,keep_event_info = True)

but I still get this error message for many events:

Pipeline and module initialisation took 11.450s (CPU 11.341s).
2021-04-07 10:58:57 ++ orcasong.modules.TimePreproc.TimePreproc: Centering time of Hits with first triggered hit
2021-04-07 10:58:57 ++ orcasong.modules.TimePreproc.TimePreproc: Centering time of McHits with first triggered hit
2021-04-07 10:58:57 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:58:59 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:58:59 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:00 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:00 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:01 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:01 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:01 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:02 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:02 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:04 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:04 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:06 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:07 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:10 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:11 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:17 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:19 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:20 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:23 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:24 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:25 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:26 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:29 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:30 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:33 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:36 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:37 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:37 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:38 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:39 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:49 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:54 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:55 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 10:59:58 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:00 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:03 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:06 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:06 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:08 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:09 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:09 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:11 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:14 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:16 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:16 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:18 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:25 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:27 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
--------------------------[ Blob    1000 ]---------------------------
---------------------[ Memory peak: 588.555 MB ]---------------------
2021-04-07 11:00:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:36 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:37 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:39 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:43 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:45 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:47 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:48 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:54 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:55 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:00:58 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:03 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:10 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:11 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:15 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:18 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:18 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:19 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:21 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:22 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:22 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:25 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:26 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:31 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:32 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:33 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:42 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:47 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:52 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:01:59 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:02:04 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
--------------------------[ Blob    2000 ]---------------------------
---------------------[ Memory peak: 934.477 MB ]---------------------
2021-04-07 11:02:15 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:02:22 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:02:24 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:02:35 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:02:43 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:02:51 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:03:03 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:03:06 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:03:20 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:03:27 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:03:31 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:03:33 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
--------------------------[ Blob    3000 ]---------------------------
---------------------[ Memory peak: 975.613 MB ]---------------------
2021-04-07 11:04:18 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:04:21 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:04:46 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:04:51 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
2021-04-07 11:04:56 ERROR ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: Differing dtypes after appending missing columns to the table! Skipping...
================================[ . ]================================
2021-04-07 11:05:51 ++ km3pipe.io.hdf5.HDF5Sink.HDF5Sink: HDF5 file written to: newTestARCA.h5
============================================================
3942 cycles drained in 7.141016min (CPU 7.126748min). Memory peak: 993.00 MB

if I look at the final file I see:

ptdump -v newTestARCA.h5
/ (RootGroup) 'KM3NeT'
/event_info (Table(3942,), fletcher32, shuffle, zlib(1)) 'EventInfo'
  description := {
  "det_id": Int64Col(shape=(), dflt=0, pos=0),
  "event_id": Int64Col(shape=(), dflt=0, pos=1),
  "frame_index": Int64Col(shape=(), dflt=0, pos=2),
  "mc_run_id": Int64Col(shape=(), dflt=0, pos=3),
  "mc_time": Float64Col(shape=(), dflt=0.0, pos=4),
  "nanoseconds": Int64Col(shape=(), dflt=0, pos=5),
  "overlays": Int64Col(shape=(), dflt=0, pos=6),
  "run_id": Int64Col(shape=(), dflt=0, pos=7),
  "timestamp": Int64Col(shape=(), dflt=0, pos=8),
  "trigger_counter": Int64Col(shape=(), dflt=0, pos=9),
  "trigger_mask": Int64Col(shape=(), dflt=0, pos=10),
  "weight_w1": Float64Col(shape=(), dflt=0.0, pos=11),
  "weight_w2": Float64Col(shape=(), dflt=0.0, pos=12),
  "weight_w3": Float64Col(shape=(), dflt=0.0, pos=13),
  "weight_w4": Float64Col(shape=(), dflt=0.0, pos=14),
  "group_id": Int64Col(shape=(), dflt=0, pos=15),
  "n_hits_intime": Int64Col(shape=(), dflt=0, pos=16)}
  byteorder := 'little'
  chunkshape := (32,)
  autoindex := True
  colindexes := {
    "event_id": Index(6, medium, shuffle, zlib(1)).is_csi=False,
    "group_id": Index(6, medium, shuffle, zlib(1)).is_csi=False}
/group_info (Table(3942,), fletcher32, shuffle, zlib(1)) 'Group Info'
  description := {
  "blob_length": Int64Col(shape=(), dflt=0, pos=0),
  "group_id": Int64Col(shape=(), dflt=0, pos=1)}
  byteorder := 'little'
  chunkshape := (32,)
  autoindex := True
  colindexes := {
    "group_id": Index(6, medium, shuffle, zlib(1)).is_csi=False}
/x (EArray(3942, 5000, 17), fletcher32, shuffle, zlib(1)) 'nodes'
  atom := Float32Atom(shape=(), dflt=0.0)
  maindim := 0
  flavor := 'numpy'
  byteorder := 'little'
  chunkshape := (32, 5000, 17)
/x_indices (Table(3942,), fletcher32, shuffle, zlib(1)) 'Indices'
  description := {
  "index": Int64Col(shape=(), dflt=0, pos=0),
  "n_items": Int64Col(shape=(), dflt=0, pos=1)}
  byteorder := 'little'
  chunkshape := (32,)
/y (Table(3839,), fletcher32, shuffle, zlib(1)) 'event_info'
  description := {
  "event_id": Int64Col(shape=(), dflt=0, pos=0),
  "particle_type": Float64Col(shape=(), dflt=0.0, pos=1),
  "energy": Float64Col(shape=(), dflt=0.0, pos=2),
  "is_cc": Float64Col(shape=(), dflt=0.0, pos=3),
  "bjorkeny": Float64Col(shape=(), dflt=0.0, pos=4),
  "dir_x": Float64Col(shape=(), dflt=0.0, pos=5),
  "dir_y": Float64Col(shape=(), dflt=0.0, pos=6),
  "dir_z": Float64Col(shape=(), dflt=0.0, pos=7),
  "time_interaction": Float64Col(shape=(), dflt=0.0, pos=8),
  "run_id": Float64Col(shape=(), dflt=0.0, pos=9),
  "vertex_pos_x": Float64Col(shape=(), dflt=0.0, pos=10),
  "vertex_pos_y": Float64Col(shape=(), dflt=0.0, pos=11),
  "vertex_pos_z": Float64Col(shape=(), dflt=0.0, pos=12),
  "n_hits": Float64Col(shape=(), dflt=0.0, pos=13),
  "weight_w1": Float64Col(shape=(), dflt=0.0, pos=14),
  "weight_w2": Float64Col(shape=(), dflt=0.0, pos=15),
  "weight_w3": Float64Col(shape=(), dflt=0.0, pos=16),
  "n_gen": Float64Col(shape=(), dflt=0.0, pos=17),
  "jmuon_E": Float64Col(shape=(), dflt=0.0, pos=18),
  "jmuon_JCOPY_Z_M": Float64Col(shape=(), dflt=0.0, pos=19),
  "jmuon_JENERGY_CHI2": Float64Col(shape=(), dflt=0.0, pos=20),
  "jmuon_JENERGY_ENERGY": Float64Col(shape=(), dflt=0.0, pos=21),
  "jmuon_JENERGY_MUON_RANGE_METRES": Float64Col(shape=(), dflt=0.0, pos=22),
  "jmuon_JENERGY_NDF": Float64Col(shape=(), dflt=0.0, pos=23),
  "jmuon_JENERGY_NOISE_LIKELIHOOD": Float64Col(shape=(), dflt=0.0, pos=24),
  "jmuon_JENERGY_NUMBER_OF_HITS": Float64Col(shape=(), dflt=0.0, pos=25),
  "jmuon_JGANDALF_BETA0_RAD": Float64Col(shape=(), dflt=0.0, pos=26),
  "jmuon_JGANDALF_BETA1_RAD": Float64Col(shape=(), dflt=0.0, pos=27),
  "jmuon_JGANDALF_CHI2": Float64Col(shape=(), dflt=0.0, pos=28),
  "jmuon_JGANDALF_LAMBDA": Float64Col(shape=(), dflt=0.0, pos=29),
  "jmuon_JGANDALF_NUMBER_OF_HITS": Float64Col(shape=(), dflt=0.0, pos=30),
  "jmuon_JGANDALF_NUMBER_OF_ITERATIONS": Float64Col(shape=(), dflt=0.0, pos=31),
  "jmuon_JSHOWERFIT_ENERGY": Float64Col(shape=(), dflt=0.0, pos=32),
  "jmuon_JSTART_LENGTH_METRES": Float64Col(shape=(), dflt=0.0, pos=33),
  "jmuon_JSTART_NPE_MIP": Float64Col(shape=(), dflt=0.0, pos=34),
  "jmuon_JSTART_NPE_MIP_TOTAL": Float64Col(shape=(), dflt=0.0, pos=35),
  "jmuon_JVETO_NPE": Float64Col(shape=(), dflt=0.0, pos=36),
  "jmuon_JVETO_NUMBER_OF_HITS": Float64Col(shape=(), dflt=0.0, pos=37),
  "jmuon_dir_x": Float64Col(shape=(), dflt=0.0, pos=38),
  "jmuon_dir_y": Float64Col(shape=(), dflt=0.0, pos=39),
  "jmuon_dir_z": Float64Col(shape=(), dflt=0.0, pos=40),
  "jmuon_length": Float64Col(shape=(), dflt=0.0, pos=41),
  "jmuon_likelihood": Float64Col(shape=(), dflt=0.0, pos=42),
  "jmuon_pos_x": Float64Col(shape=(), dflt=0.0, pos=43),
  "jmuon_pos_y": Float64Col(shape=(), dflt=0.0, pos=44),
  "jmuon_pos_z": Float64Col(shape=(), dflt=0.0, pos=45),
  "jmuon_rec_type": Float64Col(shape=(), dflt=0.0, pos=46),
  "jmuon_t": Float64Col(shape=(), dflt=0.0, pos=47),
  "jmuon_group_id": Float64Col(shape=(), dflt=0.0, pos=48),
  "aashower_E": Float64Col(shape=(), dflt=0.0, pos=49),
  "aashower_JCOPY_Z_M": Float64Col(shape=(), dflt=0.0, pos=50),
  "aashower_JENERGY_CHI2": Float64Col(shape=(), dflt=0.0, pos=51),
  "aashower_JENERGY_ENERGY": Float64Col(shape=(), dflt=0.0, pos=52),
  "aashower_JENERGY_MUON_RANGE_METRES": Float64Col(shape=(), dflt=0.0, pos=53),
  "aashower_JENERGY_NDF": Float64Col(shape=(), dflt=0.0, pos=54),
  "aashower_JENERGY_NOISE_LIKELIHOOD": Float64Col(shape=(), dflt=0.0, pos=55),
  "aashower_JENERGY_NUMBER_OF_HITS": Float64Col(shape=(), dflt=0.0, pos=56),
  "aashower_JGANDALF_BETA0_RAD": Float64Col(shape=(), dflt=0.0, pos=57),
  "aashower_JGANDALF_BETA1_RAD": Float64Col(shape=(), dflt=0.0, pos=58),
  "aashower_JGANDALF_CHI2": Float64Col(shape=(), dflt=0.0, pos=59),
  "aashower_JGANDALF_LAMBDA": Float64Col(shape=(), dflt=0.0, pos=60),
  "aashower_JGANDALF_NUMBER_OF_HITS": Float64Col(shape=(), dflt=0.0, pos=61),
  "aashower_JGANDALF_NUMBER_OF_ITERATIONS": Float64Col(shape=(), dflt=0.0, pos=62),
  "aashower_JSHOWERFIT_ENERGY": Float64Col(shape=(), dflt=0.0, pos=63),
  "aashower_JSTART_LENGTH_METRES": Float64Col(shape=(), dflt=0.0, pos=64),
  "aashower_JSTART_NPE_MIP": Float64Col(shape=(), dflt=0.0, pos=65),
  "aashower_JSTART_NPE_MIP_TOTAL": Float64Col(shape=(), dflt=0.0, pos=66),
  "aashower_JVETO_NPE": Float64Col(shape=(), dflt=0.0, pos=67),
  "aashower_JVETO_NUMBER_OF_HITS": Float64Col(shape=(), dflt=0.0, pos=68),
  "aashower_dir_x": Float64Col(shape=(), dflt=0.0, pos=69),
  "aashower_dir_y": Float64Col(shape=(), dflt=0.0, pos=70),
  "aashower_dir_z": Float64Col(shape=(), dflt=0.0, pos=71),
  "aashower_length": Float64Col(shape=(), dflt=0.0, pos=72),
  "aashower_likelihood": Float64Col(shape=(), dflt=0.0, pos=73),
  "aashower_pos_x": Float64Col(shape=(), dflt=0.0, pos=74),
  "aashower_pos_y": Float64Col(shape=(), dflt=0.0, pos=75),
  "aashower_pos_z": Float64Col(shape=(), dflt=0.0, pos=76),
  "aashower_rec_type": Float64Col(shape=(), dflt=0.0, pos=77),
  "aashower_t": Float64Col(shape=(), dflt=0.0, pos=78),
  "aashower_group_id": Float64Col(shape=(), dflt=0.0, pos=79),
  "group_id": Int64Col(shape=(), dflt=0, pos=80)}
  byteorder := 'little'
  chunkshape := (32,)
  autoindex := True
  colindexes := {
    "event_id": Index(6, medium, shuffle, zlib(1)).is_csi=False,
    "group_id": Index(6, medium, shuffle, zlib(1)).is_csi=False}

does it make sense or there is something missing? I don't see the "best track"..

The error is thrown when there is no jmuon (JGandalf) reco for an event but still one for aashower. Then, the returned dict tracks from the extractor has not the same length as the others and thus cannot be concatenated correctly. at the moment these events are skipped. I think is acceptable, as first, these are only a few events and second, potentially difficult events to reconstruct anyway. So no big loss leaving them out of the training/analysis.

ok thank you @dguderian ! do you think the file is ok? if yes I will proceed by converting again ALL ARCA files.

A solution would be to pad with e.g. nan. I feel like keeping all events might be useful somewhere down the line, no?

Wait, also they are not completely skipped; the x is written but the y is not. This is gonna be a problem later on.

In that case, padding is probably better. After all, it is only about the std reco info, which is just for later comparisons. But the mc truth info we need in the y

Meh, but how you pad if one reco is not there for one event; then you would have to specify which recos to expect, right?

Orcasong should probably throw an error in that case, since different lengths in x and y render the file useless later on...

Meh, but how you pad if one reco is not there for one event; then you would have to > specify with recos to expect, right?

Is this not what you are doing already anyways?

Yeah, an error like a warning and then discard that event.

No, fortunately, the extractions just looks for the present recos and writes all the best tracks it find into the y. This way no user interaction is needed. But I think we have to give this up. We can add a list specifying the recos. If it is none, dont consider any std reco outputs.

Yeah, an error like a warning and then discard that event.

The problem is that this check happens in the HDF5Sink .... so unless we want to duplicate the logic this would be an issue for tom i think

Question is, if the sink should work like this. I mean not allow different sized objects to be put into the same file. In principle, if the user knows what he is doing this should be possible. Only in this example we see that it is often not intended. Let me call upon @tgal ^^

Different sized objects are perfectly fine, that's why we have _indices (for split Tables and NDArrays) and group_id for simple (unsplit) Tables.

In this case, the /y entry is a table and it should have some missing entries in group_id, indicating that for some groups (during the iteration) there was no data for this entry.

The HDF5Pump automatically fills the blob with the corresponding entries from the HDF5 file, based on group_id and _indices. If you want to read that by hand, you have to take into account that intrinsics.

Note that the same is true for NDArrays which has "holes/missing entries". That's why you should always load *_indices into memory and use that to check which group is which. It has two columns: index and n_items.

Right, so in principle this can still be used in orcanet. However, I think for simplicity orcasong should discard events where x and y are not both present.

And regarding the padding, is it ok to add this request for std reco info as a new parameter?

Ah, wait, we can catch the info about which reco is present from the file itself. So, I am gonna implement a padding with nans so that all events survive.

To fix the y set thing: We could cache the column names of y the first time the pipeline is run, and then just throw if the column names dont match.

I think its better to throw and not skip, since this only happens if somethings not right and the user should probably fix this properly.

But there might be the unlikely case that the first event is the odd one out. The info about what recos are in there is already there in the beginning of the extractor for a file. so no problem propagating it to the std info function and force it to write out something for each of these recos.

Or do you have any concerns?

For the problem with the extractor? No not at all. I was talking about the fact that orcasong produced datasets of different length which should never happen. I guess this issue is a 2 in 1

Ok, sry, I got confused^^

created merge request !19 (merged) to address this issue

mentioned in merge request !19 (merged)

Ok, the padding is ready. If there is generally a reco type done on a file, all events will now have the corresponding info objects about it. If there is no actual info for an event, nan's are added for all members.

Alba's file now gives:

/ (RootGroup) 'KM3NeT'
/group_info (Table(1602,), fletcher32, shuffle, zlib(1)) 'Group Info'
/x (EArray(1602, 2000, 8), fletcher32, shuffle, zlib(1)) 'nodes'
/x_indices (Table(1602,), fletcher32, shuffle, zlib(1)) 'Indices'
/y (Table(1602,), fletcher32, shuffle, zlib(1)) 'event_info'

mentioned in commit 04e56280

closed with merge request !19 (merged)

Thank you @dguderian! I just tried to re-convert the file (after updating everything), but I still get the error messages. Is this still expected even after your modification?

ok it seems it was not updated, now I don t get errors anymore. Thank you!

error when using new extractor

Child items 0

Activity