Ludwig: Prediction of numerical features

[loretoparisi]

My yaml file looks like

training:
    epochs: 10
    learning_rate: 0.001
    batch_size: 128
    early_stop: 10

input_features:
    -
        name: lyrics
        type: text
        encoder: parallel_cnn
        level: word

output_features:
    -
        name: f1
        type: numerical
    -
        name: f2
        type: numerical

so I have two float features f1 and f2 (and so MSE will be used by default as loss) to be predicted over a input text, for which I'm using a parallel_cnn at word level.

After few epochs I'm getting a 0 accuracy.

╒════════════╤═══════════╤════════════╕
│ combined   │      loss │   accuracy │
╞════════════╪═══════════╪════════════╡
│ train      │ 1121.6427 │     0.0000 │
├────────────┼───────────┼────────────┤
│ vali       │ 1140.5157 │     0.0000 │
├────────────┼───────────┼────────────┤
│ test       │ 1136.8768 │     0.0000 │
╘════════════╧═══════════╧════════════╛

I get the same result when using for the input a different encoder like

input_features:
    -
        name: lyrics
        type: text
        encoder: rnn
        cell: lstm
        bidirectional: true

is the yaml output_features definition correct for these float values?

[w4nderlust]

Yeah, there's a bit of confusion of the meaning of that accuracy. i should either clarify or refactor.
The rationale is the following: some types of output feature have an accuracy metric. Combined combines them, meaning if i have two category features, each of them have their own accuracy, and combined accuracy tells me what is the proportion of datapoints where I do correct predictions for BOTH features AT THE SAME TIME. If you think at correct predictions as a list of booleans as long as your datapoints fo each feature, combined calculates the accuracy on the logical and of those two lists.
It's useful because if you have multiple output features, you may want to validate on this measure rather than on each independent one.

Some other features, like numerical ones, do not have a notion of accuracy, they have ther mse, mae and r2. When validating on multiple output features that do not have a notion of accuracy, the combined loss value on the validation set is used instead. Basically that accuracy is never computed nor it is used for decisions and doesn't represent how well the model is performing. In your case how well are the models performing is representented by loss, which is the sum of the mse of both (or the sum of whatever loss you specified) weighted by the weight that you specify for each (by default 1).

An enhancement could be for Ludwig to avoid showing the accuracy column at all if not all the features support it, to avoid the confusion that it can generate, like in your case.

[loretoparisi]

@w4nderlust thank you for the details, it perfectly makes sense. Also I have just realized that my output numerical features should be pre-processed and normalized between [0,1]. I was looking t the User Guide for the Numerical Output Features and Decoders, should I add a custom decoder for that? or is a built-in one?
Thank you.

[w4nderlust]

@loretoparisi let me tray to understand your usecase batter. So those numbers that you want to output are between [0,1] but they are not probabilities of a binary classifier, is that correct?

Depending on that, one solution could be to add a preprocessing parameter like normalize_01 that performs this normalization at the data level (so it would work for both numerical inputs and numerical features. There could also be a normalize_zscore and a normalize_minmax normalization strategy, so probably it would be better to have a normalize parameter that by default is None but then you can pass a string with the name of the normalization strategy (01, maxmin, zscore) and it will adopt that strategy reading it from a normalization strategy registry.

This will work at the data level, but there wouldn't be anything in the model to constraint it to produce a value in [0,1]. For that purpose one can think about writing a decoder that clips values before outputting them, or some other strategy (for instance applying a sigmoid). Adding a decoder should be pretty easy, the only difficulty is that sequence features for instance already have a machinery with a registry of decoders that are selected by their name, while numerical features don't have that because so far there has only been one decoder. Adding it would be simple, and probably I should do it for all the features anyway.

What do you think?

[loretoparisi]

@w4nderlust yes those numbers are integers in a scale between [0,100] that measures two different song high level features. This approach in the past was used by SVM classifiers in the Gaia framework of Essentia. I think having a normalization strategy for both features and numerical input would be the best option from the command line.

[w4nderlust]

Got it, will mark this as enhancement. Let me know if you are planning to contribute on this and we can chat more in detail!

[w4nderlust]

Clipping and normalization were added to numerical features, so i consider this to be solved.

[loretoparisi]

Thank you!