FileBinner crashes while trying to access particular info in the h5 file

Summary

When creating images off mc neutrino data using the FileBinner, it crashes; see below. This might be related to recent changes in the data formats?

Environment

• OrcaSong version: 2.1.1
• Python version: 3.7.5
• OS: Lyon

What is the current bug behavior?

Crash with error message below.

What is the expected correct behavior?

No crash any more :))

Steps to reproduce

 #!/usr/bin/python -i
from orcasong.core import FileBinner
import numpy as np
import sys
import os

input_file = "/sps/km3net/users/guderian/NN_stuff/raw_data/ORCA_4/anumu/mcv5.40.genhen_anumuCC_10GeV.km3_AAv1.jterbr00005790.236.h5

#bin edges for the ORCA4 detector
bin_edges_list = [
    ["pos_x", np.linspace(-13.5-4.1, 19.5+4.1 , 4+1)], 		#4 strings from -13.5 to 19.5m, mean distance 8.25m, add 4.1m
    ["pos_y", np.linspace(-11-4.25, 6+4.25, 2+1)],			#2 strings from -11 to 6, mean distance 8.5m, add 4.25m
    ["pos_z", np.linspace(29-4.5, 192+4.5, 18+1)],				#18 DOMs from 29 to 192m , mean distance 9.05m, add 4.5m
    ]

bin_edges_list.append(["channel_id", np.linspace(0-0.5, 30+0.5, 31 + 1)])			#31 channels

fb = FileBinner(bin_edges_list,mc_info_extr="neutrino")

output_file = "$TMPDIR/test.h5"

fb.run(input_file,output_file)

Relevant logs and/or screenshots

Working on file:mcv5.40.genhen_anumuCC_10GeV.km3_AAv1.jterbr00005790.236.h5
Generating ('pos_x', 'pos_y', 'pos_z', 'time') images with shape (4, 2, 18, 600)
++ km3pipe.io.hdf5.HDF5Pump.HDF5Pump: Opening mcv5.40.genhen_anumuCC_10GeV.km3_AAv1.jterbr00005790.236.h5
Traceback (most recent call last):
  File "/pbs/home/g/guderian/scripts/ml/pre_processing/ORCA4/execute_orca_song_2.py", line 70, in <module>
    fb.run(input_file,output_file)
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/orcasong/core.py", line 160, in run
    pipe = self.build_pipe(infile, outfile)
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/orcasong/core.py", line 217, in build_pipe
    pipe.attach(kp.io.hdf5.HDF5Pump, filename=infile)
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/thepipe/core.py", line 273, in attach
    module = fac(name=name, **kwargs)
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/thepipe/core.py", line 90, in __init__
    self.configure()
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/km3pipe/io/hdf5.py", line 618, in configure
    self._load_next_file()
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/km3pipe/io/hdf5.py", line 626, in _load_next_file
    self._read_group_info()
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/km3pipe/io/hdf5.py", line 685, in _read_group_info
    h5file.get_node('/raw_header')
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/km3pipe/io/hdf5.py", line 167, in from_pytable
    return cls(data)
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/km3pipe/io/hdf5.py", line 71, in __init__
    self._set_attributes()
  File "/pbs/home/g/guderian/scripts/deep_learning_source/ml_venv/lib/python3.7/site-packages/km3pipe/io/hdf5.py", line 80, in _set_attributes
    parameter, [field_names[i] for i in sorted_indices]
  File "/pbs/throng/km3net/software/python/3.7.5/lib/python3.7/collections/__init__.py", line 361, in namedtuple
    raise ValueError('Type names and field names must be valid '
ValueError: Type names and field names must be valid identifiers: ''
Closing remaining open files:mcv5.40.genhen_anumuCC_10GeV.km3_AAv1.jterbr00005790.236.h5...done

added bug label

changed the description

Ich habs mal weiter runtergebrochen:

pump = kp.io.HDF5Pump(filename=input_file)

wirft deinen Fehler. Das file ist wohl nicht lesbar mit km3pipe, hat also nicht mit orcasong zu tun. Wäre dann der tom zuständig :-)

closed

I am not sure if I understand. A file is created with FileBinner and then the HDF5Pump is showing that error when read back in?

Can you show me a link to the file?

If I understand correctly, it has nothing to do with OrcaSong, as the file is a raw file @tgal:

input_file = "/sps/km3net/users/guderian/NN_stuff/raw_data/ORCA_4/anumu/mcv5.40.genhen_anumuCC_10GeV.km3_AAv1.jterbr00005790.236.h5
pump = kp.io.HDF5Pump(filename=input_file)

Throws the error.

OK, it seems that the /raw_header table contains some characters which are invalid in Python attributes

In physics ('', 'program', 'version', 'date') there is an entry which is empty...

@dguderian can you show me the original file?

a6 a10 f4 f4 program version date GENHEN 7.2-220514 191211 2021 physics

Thanks for investigating! The original root file is in the same directory: /sps/km3net/users/guderian/NN_stuff/raw_data/ORCA_4/anumu/mcv5.40.genhen_anumuCC_10GeV.km3_AAv1.jterbr00005790.236.root

moved to km3py/km3pipe#204 (closed)