Add functionality for prediction datasets, that are too large for one GPU job
Currently, the parser_orcapred checks if there is a pred file that already exists in the /prediction folder, and if not, it makes a prediction based on the list.toml file.
However, if the prediction dataset is too large and the prediction doesnt finish in a GPU job (>24h), there is no way for OrcaNet to resume the prediction in the next job. I.e., it will recognize an (unfinished) prediction file such that it will skip the prediction step.
One way to fix this would be as follows:
- Create a prediction file after the prediction is finished for a single val file. The filename should contain the filenumber of the val file.
- If a prediction file already exists for a previous val file, append the data and update the val filenumber to the latest one
Additionally, in the beginning of the prediction, it has to be checked if a previous prediction has been done and if the prediction is fully (!) done (compare filenumber argument with the number of the last file in the val list?).