Ludwig: hyperopts overtraining

[ifokeev]

Is it possible to tune hyperopts based on test or validation loss? I'm getting overtraining due to good params for train but it's moving crazy on test

train stats:

test stats:

[michaelbzhu]

What does your configuration look like?
Under the hyperopt section you can specify the split that you want to use
Also, I think hyperopt and train should normally use validation metrics by default

config = {
    'hyperopt': {
        'split': 'validation',   # this should be set to 'validation' by default, but you can also select 'training' or 'test'
        'goal': 'minimize',
        'output_feature': 'label',
        'metric': 'loss',
        'parameters': {
...

Hope this helps!

[w4nderlust]

@michaelbzhu 's answer is correct. let me know if that soulds your issue @ifokeev .
You cal also set what validation metric to use for determining early stopping within each single run of the hyperopt (for early stopping purposes).