Gluon-ts: Provide working examples for Temporal Fusion Transformer

[StatMixedML]

Description

Referencing to this issue https://github.com/awslabs/gluon-ts/issues/1013#issuecomment-687575025, I am trying to use the Temporal Fusion Transformer in GluonTS.

@dcmaddix & @Gandor26: can you please provide a detailed example on how to use the TFT, also with examples on how to use

• time_features:List[gluonts.time_feature._base.TimeFeature]
• static_cardinalities:Dict[str, int]
• dynamic_cardinalities:Dict[str, int]
• static_feature_dims:Dict[str, int]
• dynamic_feature_dims:Dict[str, int]
• past_dynamic_features:List[str]

and add it to https://github.com/awslabs/gluon-ts/tree/master/evaluations.

https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html is a really exhaustive tutorial on how to use TFT. Would be great if you can also provide it for the GluonTS implementation.

Thanks

[Gandor26]

Sure. I'll give some high-level instructions here, and maybe later we'll provide a more detailed tutorial at gluon-ts.
P.S. There are some bugs when features are not provided. A new PR is created to fix that.

We'll start by some imports

import os
import json
from copy import deepcopy
from pathlib import Path

import numpy as np
from matplotlib import pyplot as plt
import mxnet as mx
from mxnet import gluon
from mxnet import nd, sym
from mxnet.gluon import nn, HybridBlock

from gluonts.dataset.repository.datasets import get_dataset
from gluonts.dataset.common import (
    FileDataset, ListDataset, TrainDatasets,
    CategoricalFeatureInfo, BasicFeatureInfo,
)
from gluonts.dataset.loader import TrainDataLoader, InferenceDataLoader
from gluonts.model.tft import *
from gluonts.mx.trainer import Trainer
from gluonts.evaluation import Evaluator
from gluonts.evaluation.backtest import make_evaluation_predictions

1. Load data and provide features
   First we need to transform time series data into GluonTs FileDataset / ListDataset format, in which each entry is a dictionary consisting of targets, start_time and features of different types. Here we'll use the repository electricity dataset with some additional artificial features as an example.

data = get_dataset('electricity')
metadata = data.metadata
train_set = data.train
test_set = data.test

# artifical static categorical features
metadata.feat_static_cat = [
    CategoricalFeatureInfo(name='static_cat_1', cardinality='100'),
    CategoricalFeatureInfo(name='static_cat_2', cardinality='200'),
]
# artifical static real features
metadata.feat_static_real = [
    BasicFeatureInfo(name='static_real_1'),
    BasicFeatureInfo(name='static_real_2'),
]
# artifical dynamic categorical features
metadata.feat_dynamic_cat = [
    CategoricalFeatureInfo(name='dynamic_cat_1', cardinality='50'),
    CategoricalFeatureInfo(name='dynamic_cat_2', cardinality='80'),
]
# artifical dynamic real features
metadata.feat_dynamic_real = [
    BasicFeatureInfo(name='dynamic_real_1'),
    BasicFeatureInfo(name='dynamic_real_2'),
]
# indicate unknown dynamic features
feat_past_dynamic = ['dynamic_cat_1', 'dynamic_real_2']
# indicate dimension of dynamic real features (other than the temporal dimension)
# it'd be better specified in FeatureInfo in the future
static_real_1_dim = 2 
static_real_2_dim = 3
dynamic_real_1_dim = 4
dynamic_real_2_dim = 5

# generate random features for each data entry
train = []
for entry in train_set:
    length = len(entry['target'])
    train.append({
        'target': entry['target'],
        'start': entry['start'],
        'static_cat_1': np.random.randint(100, size=(1,)),
        'static_cat_2': np.random.randint(200, size=(1,)),
        'static_real_1': np.random.randn(static_real_1_dim),
        'static_real_2': np.random.randn(static_real_2_dim),
        'dynamic_cat_1': np.random.randint(50, size=(length,)),
        'dynamic_cat_2': np.random.randint(80, size=(length,)),
        'dynamic_real_1': np.random.randn(dynamic_real_1_dim, length),
        'dynamic_real_2': np.random.randn(dynamic_real_2_dim, length),
    })
train = ListDataset(train, metadata.freq)

test = []
for entry in test_set:
    length = len(entry['target'])
    test.append({
        'target': entry['target'],
        'start': entry['start'],
        'static_cat_1': np.random.randint(100, size=(1,)),
        'static_cat_2': np.random.randint(200, size=(1,)),
        'static_real_1': np.random.randn(static_real_1_dim),
        'static_real_2': np.random.randn(static_real_2_dim),
        'dynamic_cat_1': np.random.randint(50, size=(length,)),
        'dynamic_cat_2': np.random.randint(80, size=(length,)),
        'dynamic_real_1': np.random.randn(dynamic_real_1_dim, length),
        'dynamic_real_2': np.random.randn(dynamic_real_2_dim, length),
    })
test = ListDataset(test, metadata.freq)

dataset = TrainDatasets(metadata=metadata, train=train, test=test)

1. Create model and specify certain hyperparameters. The cardinalities of categorical features and dimension of real features are included in the metadata, otherwise you may have to manually set them, such as static_cardinalities={ 'static_cat_1': 100, 'static_cat_2': 200 }

trainer = Trainer(
    ctx=mx.gpu(0),
    epochs = 10,
    batch_size = 32,
    num_batches_per_epoch=1000,
    learning_rate = 2e-3,
    patience = 5,
    minimum_learning_rate = 1e-5,
    clip_gradient = 10.0,
    weight_decay = 1e-4,
    hybridize = False, 
)
estimator = TemporalFusionTransformerEstimator(
    freq = dataset.metadata.freq,
    context_length=168,
    prediction_length = dataset.metadata.prediction_length,
    trainer=trainer,
    hidden_dim=32,
    variable_dim=8, # this is the dimension of variable encodings; in the original paper it is always `hidden_dim`
    num_heads=4,
    num_outputs=3, # number of quantiles to be predicted. E.g. [0.5,0.1,0.9] for `num_outputs=3` and [0.5,0.1,0.9,0.2,0.8] for `num_outputs=5`
    num_instance_per_series=100,
    dropout_rate=0.1,
    static_cardinalities={
        info.name: int(info.cardinality) 
        for info in dataset.metadata.feat_static_cat
    }, # cardinalities of static categorical features
    dynamic_cardinalities={
        info.name: int(info.cardinality)
        for info in dataset.metadata.feat_dynamic_cat
    }, # cardinalities of dynamic categorical features
    static_feature_dims={
        info.name: eval(f"{info.name}_dim")
        for info in dataset.metadata.feat_static_real
    }, # dimensions of static real features
    dynamic_feature_dims={
        info.name: eval(f"{info.name}_dim")
        for info in dataset.metadata.feat_dynamic_real
    }, # dimensions of dynamic real features
    past_dynamic_features=feat_past_dynamic,
)

1. Fit the model

predictor = estimator.train(dataset.train)

1. Inference and evaluation

forecast_it, ts_it = make_evaluation_predictions(
    dataset=dataset.test,  # test dataset
    predictor=predictor,  # predictor
    num_samples=100,  # number of sample paths we want for evaluation
)
evaluator = Evaluator(
    quantiles=predictor.prediction_net.quantiles,
    seasonality=24,
)
agg_m, item_m = evaluator(
    ts_iterator=ts_it,
    fcst_iterator=forecast_it,
    num_series=len(dataset.test),
)

[StatMixedML]

@Gandor26 Thanks very much for the example! Very much appreciate.

[MariPlaza]

@Gandor26 @lostella Has the TFT model in gluonts, the implementation of the explanation of variable importance, as pytorch has? Reference: https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/stallion.html

[harusametime]

@Gandor26
What does past_dynamic_features mean?
Does the estimator assume that we cannot use future values of the specified features, but can use only past values in making predictions?

According to the paper, we have two variables z and x:

observed inputs z which can only be measured at each step and are unknown beforehand, and known input x which can be predetermined (e.g. the day-of-week at time t).

Does past_dynamic_features represent z?

[harusametime]

My understanding is probably correct. According to the following implementation, when a variable is specified as past_dynamic_features, the variable moves to list self.past_ts_fields, and only past piece is stored.
https://github.com/awslabs/gluon-ts/blob/fbc00b25c21c2ca9c59e0ae82b0fc6d24a03db5e/src/gluonts/model/tft/_transform.py#L124:L138

[StatMixedML]

@Gandor26 Thanks for providing the detailed explanation!

I do have some questions though on the way the data is prepared:

• the DeepAR estimator expects feat_static_cat, feat_dynamic_real, feat_static_real to be collected in one list/np.array, where the shape of each of them indicates how many features are contained as features for each of them.

• Now for the TemporalFusionTransformer, feat_static_cat_ has as many feat_static_cat_ as there are categorical features, i.e., feat_static_cat_1, feat_static_cat_2, feat_static_cat_x. Why is that? Why can't we provide all of them in one list?

• Also, in your example, you add several static_real_1, static_real_2 and dynamic_cat_1, dynamic_cat_2 and dynamic_real_1, dynamic_real_2. Again, why can't we only provide one static_real & one dynamic_cat & one dynamic_real feature entry to ListDataset?

Thanks for your support!