Gluon-ts: lag sequences for models are unconstrained

[kashif]

Description

For the models that use the get_lags_for_frequency() helper, this helper returns an array with potentially very large lag values, e.g. for freq='D' it is:

[1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 20, 21, 22, 27, 28, 29, 30, 
 31, 56, 84, 363, 364, 365, 727, 728, 729, 1091, 1092, 1093]

meaning the past_length in the InstanceSplitter is dominated by the largest value here and the context_length is ignored as huge windows of size > 1093 are sampled...

I believe we need to upper bound the helper when we call it by the provided context_length, e.g. something like:

        self.lags_seq = (
            lags_seq
            if lags_seq is not None
            else get_lags_for_frequency(freq_str=freq, lag_ub=self.context_length)
        )

for more reasonable sized past_length windows

[StatMixedML]

I agree that very long-range lags are potentially less useful. However, keep in mind that restricting get_lags_for_frequency by an upper bound self.context_length restricts the historical availability of the response. As far as I understand, adding lags increases the context_length with additional observations which makes, e.g., DeepAR more "attentive" to a longer history.

[mbohlkeschneider]

Currently, models are supposed internally to slice the past time windows into whatever length they need (to context_length, for example).

Doesn't it basically solve the problem by providing custom lags (which you can do by passing lags_seq)?

[kashif]

@mbohlkeschneider true... but the issue is, I think, since the InstanceSplitter's past_length is now very large, so more or less the same window will be sampled from the time series meaning the forward(...) to the model is always the same giant window... But yes the custom lags will also solve this... its just by default the behaviour seems odd for DeepAR for example...

[StatMixedML]

Or we leave it as is and add an attention layer that selects the appropriate lags ...

[mbohlkeschneider]

Or we leave it as is and add an attention layer that selects the appropriate lags ...

We would be forever grateful for this PR :D

[StatMixedML]

Ok let' see what I can do. I do believe that adding attention to DeepAR would improve accuracy.

Also, we might think of adding attention also to the features added by the transform module. This might be useful if there is a lot of features, say in feat_static_real or feat_static_cat that might not be informative. However, I am not sure though to what extent the authors of DeepAR would welcome such adjustments since we`d cross the line of transformers.

[mbohlkeschneider]

@mbohlkeschneider true... but the issue is, I think, since the InstanceSplitter's past_length is now very large more or less the same window will be sampled from the time series meaning the forward(...) to the model is always the same giant window... But yes the custom lags will also solve this... its just by default the behaviour seems odd for DeepAR for example...

I think the reason for this is largely empirical. These lags did the most consistent improvement over many datasets for DeepAR. So I think choosing the default behavior that correlates with best mean performance makes some sense.

I agree that this is somewhat confusing and not ideal. I have to think about a better way of handling this.

[kashif]

@mbohlkeschneider I see, well my hypothesis is that these lags themself are not too useful but the reason you see good results is that with these lags options, you are actually training on the full or rather large windows of the time series and it is this that is causing the improved results... but again I have to test this hypothesis in a controlled manner...

anyways yes do have a look for example in DeepAR with a D freq dataset

[mbohlkeschneider]

What do you mean with "you are actually training on the full or rather large windows of the time series"? At least DeepAR will only train on the context_length + prediction_length.

[kashif]

hmm right perhaps that fixes my confusion! I'll have a look thanks!

[mbohlkeschneider]

Yeah, I can see to be somewhat confusing... It is also somewhat not clear in the code (and seems to cause confusion a lot!).

But here is the relevant line: https://github.com/awslabs/gluon-ts/blob/master/src/gluonts/model/deepar/_network.py#L442

[kashif]

@mbohlkeschneider indeed so now imagine a time series say daily with say 800 time points... the instance splitter will sample a window of size >1000 meaning the same window will be sampled? (<- is this assumption true?)

now from this same window we only train on self.history_lenght (>1000) - self.context_length end of it...

so the question is more about the instance splitter aspect... does the same big window always get sampled? thank you once again!

[mbohlkeschneider]

Hi @kashif, no it will not return the same window all over again. But it will left pad the data if you allow DeepAR to pick incomplete samples (which is the default). In your specific this would mean (as you noticed) that the large lags are not very useful because they are always drawn from padded dummy values (0 per default).

You can confirm that it is sampling different windows with this sample code:

import numpy as np
import pandas as pd
import gluonts
from gluonts import time_feature, transform
from gluonts.dataset.field_names import FieldName
train_length = 100
pred_length = 5
t = transform.InstanceSplitter(
        target_field=FieldName.TARGET,
        is_pad_field=FieldName.IS_PAD,
        start_field=FieldName.START,
        forecast_start_field=FieldName.FORECAST_START,
        train_sampler=transform.ExpectedNumInstanceSampler(1.0),
        past_length=train_length,
        future_length=pred_length,
        pick_incomplete=True,
)
data = [{'start': pd.Timestamp('2020-01-01', freq='D'), 'target':np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20])}]
for d in t(data, is_train=True):
    print(d)

[kashif]

awesome thanks! I also went through the code for the InstanceSplitter and understand it better now.

BTW in the splitters the padding can you kindly change it to:

                 pad_block = (
                        np.full(
                            shape=d[ts_field].shape[:-1] + (pad_length,),
                            fill_value=self.dummy_value,
                            dtype=d[ts_field].dtype,
                        ))

which preserves the type of the pad_block and also in the CanonicalInstanceSplitter? Its not a big deal but more "correct" I suppose.