hdf merging problem with ORCA4 data
When using Zineb's files with ORCA4 data in the following way:
df_list = []
with pd.HDFStore(hdf_name) as hdf:
for key in hdf.keys():
if 'headers' in key : continue
df = pd.read_hdf(hdf, key)
df.reset_index(drop=True)
df_list.append(df)
result = pd.concat(df_list, axis=1)
result.to_hdf('data_merged.h5', key='summary', mode='w')
The output which I am getting in Lyon is this:
Killed
I am aware that the arrays which are created with the above code take a lot of memory, I have checked the code line by line and the problem appears on the hdf writing level. I need this data output to appear as a single data frame, it's true that I don't have to necessarily rewrite it as hdf, but is it possible to overcome the problem? Maybe my method is not optimal and indeed it sucks a lot of memory.