Is the val_set kwarg within train_ensemble used to affect the fitting process in any way?

[sgbaird]

(e.g. learning rate scheduler, internal optimizer, hyperparameter optimization) See also anthony-wang/CrabNet#15

Moved into its own issue from #25 (comment)

[CompRhys]

Tldr; not in the examples scripts but it was used for proper held out tuning when developing the model.

On this I initially was using a validation set to tune roost but after I moved from small datasets where we were performing equivalently to Magpie+RF to OQMD where we saw a huge improvement over ElemNet I fixed the architecture and optimization settings to focus on other aspects such as uncertainty and ensembles and stopped holding out the extra val data.

It would be good to be more explicit about hyperparameter choices.

We used to keep the best model checkpoint according to the val loss but stopped that due to the risk of people using that functionality to “cheat” if passing the test set as the val set.

[CompRhys]

happy to close?