Training summary shows bad behaviour
I ran the GNN muon/neutrino classification training with FULL-ARCA-v6 MC files but I am noticing 2 things:
- compared to the training with FULL-ARCA-v5 MC files, where the number of events was similar, the training time is way longer: now it takes about 30 hours for 1 epoch, before 7 hours.
Surely the model and config file has changed a bit due to changes in orcanet/master, so I copy here what I am using now:
MODEL:
[model]
type = "DisjointEdgeConvBlock"
next_neighbors = 16
shortcut = true
blocks = [
{units=[64, 64, 64], batchnorm_for_nodes=true},
{units=[128, 128, 128]},
{units=[256, 256, 256], pooling=true},
{type="OutputCateg", transition=false, output_name="bg_output", categories=3}
]
# ----------------------------------------------------------------------
[compile]
optimizer = "adam"
[compile.losses]
bg_output = {function="categorical_crossentropy", metrics=['acc']}
CONFIG:
[config]
learning_rate = [0.025,0.02]
train_logger_display=200 #number of batches after which the training performance is displayed
train_logger_flush=-1
verbose_train = 1 #0, 1 or 2
batchsize = 32
#n_events=300000
shuffle_train=true
validate_interval=1 #number of files after which a validation is performed (and plotted)
sample_modifier="GraphEdgeConv"
label_modifier = {name="ClassificationLabels", column="particle_type", classes={class1 = [12,-12,14,-14,16,-16],cl\
ass2 = [13, -13, 0]},model_output="bg_output"}
- The summary plots are very bad:
I don't really get why it behaves this way (in the past it was ok).
In the past I used learning_rate = "triple_decay_cf"
, can this make any difference? (in speed and result). If no, what can cause this behaviour?