Gluon-ts: Item metrics dataframe incorrectly formatted

[AaronSpieler]

Description

The data-frame containing the per time step metrics has a column item_id and filled with NaNs.

To Reproduce

For example in the extended tutorial:

evaluator = Evaluator(quantiles=[0.1, 0.5, 0.9])
agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(test_ds))
item_metrics.head()

Environment

• Operating system: MacOs
• Python version: 3.6.9
• GluonTS version: 0.3.3

[lostella]

@AaronSpieler I believe that depends on the dataset you use: if the entries in the dataset have an “item_id” field, then that will be used to tag the forecasts and appropriately fill in the associated field in the metrics dataframe.

What dataset did that occur with? If it’s one of the provided datasets, then the field may be missing or have a different name, which should be fixed.

[AaronSpieler]

Its the m4_hourly dataset (as used in the quick tutorial).

[jaheba]

@AaronSpieler can you follow up on this?

[AaronSpieler]

So all the provided datasets I tested contain a categorical feature called "feat_static_cat", which is equal to the corresponding index of the series.

Is this supposed to be renamed to "item_id" @lostella ?

[matsmaiwald]

@jaheba This is probably related to the fact that, unlike in section 1.3 of the Extended Forecasting Tutorial (here), in version 0.3.3 FieldName does not have the ITEM_ID attribute anymore.

To Reproduce

from gluonts.transform import FieldName
[f"FieldName.{k} = '{v}'" for k, v in FieldName.__dict__.items() if not k.startswith('_')]

Environment

• Operating system: CentOS
• Python version: 3.6.1
• GluonTS version: 0.3.3

[AaronSpieler]

Mhhh, I see. Thanks @matsmaiwald
So FieldName.ITEM_ID was removed.
However, that does not explain why the values that were stored in "item_id" were moved to "feat_static_cat". As far as I can tell in the case of the provided datasets they really only represent the index.
Also, all provided datasets seem to be missing any covariates? Was this always the case? @jaheba @lostella

[lostella]

@AaronSpieler the “item_id” serves a different purpose than feature fields: it is used to identify which time series from a dataset a forecast refers to. So for example, if you create multiple inference scenarios from a time series (like cutting it at two different points in time) you are still able to relate both predictions to the same time series, and for example aggregate evaluation metrics accordingly. Categorical features instead are usually fed into the networks, and setting them to an incremental, 0-based counter (essentially an ID) is one way to go in case no other relevant groupings are there.

Note however, that a model trained by using such feature will be able to make predictions on time series with the same categorical features values, so the ID case puts some restrictions (no out of sample predictions allowed).

Also yes, I believe no covariates was always the case for the built in datasets.

[lostella]

Note that FieldName.ITEM_ID was not removed, but rather introduced: the documentation website happens to be ahead of 0.3.3, and reflects master instead (and we should get that fixed).

[AaronSpieler]

Ok, then it seems ITEM_ID is just not properly set for the datasets. I will look into it again and see to it that it gets fixed.

[tm1611]

Hi @AaronSpieler , do you have any updates on this question yet? In particular on how to include the item_id to associate a time series with its corresponding error measures (instead of getting NaNs for all ids)?

Update:

Simply including an item_id field in the dataset seems to be enough (did not expect it to be that easy):

I used the original series id, i.e. W1 for first weekly time series, W2 for second weekly series, and so forth...

[AaronSpieler]

Fixed by pull request: https://github.com/awslabs/gluon-ts/pull/546

[lostella]

I think this should be fixed by adjusting the scripts that generate the affected datasets, so that they write the “item_id” field

[lostella]

The problem should be gone now if you use the master branch: loading the datasets once with “regenerate=True” should do the trick. This change will be included in the next release.