Gluon-ts: Pandas-backed Dataset types

[lostella]

Pandas is the go-to library to do lots of things with data, and GluonTS should probably provide easy ways to use its data structures directly in models.

One way to do this is to provide an appropriate Dataset type that feeds from Pandas structures, and produces DataEntry objects (the dictionaries that GluonTS models use) when iterated over.

A single DataFrame as a Dataset

The simplest use case would be to be able to use a single DataFrame as Dataset, i.e. to consider each column in it as time series to train, test, or make predictions on. One could even specify a subset of the series to be used, using indices or column names for example.

This has some apparent restrictions:

• All time series share the same index
• It's probably hard to specify per-series features

On the other hand, this would allow doing stuff like

df = pd.read_csv("data.csv")
training_dataset = PandasDataset(df)
estimator.train(training_dataset)

which I think would be quite nice.

One DataFrame per DataEntry

This would allow to overcome some limitations of the previous approach: each DataFrame has its own index, so each DataEntry spans its own time range, and one could indicate which column(s) should be used as target, which as dynamic features.

Some limitations here would remain:

• How to provide static features?

Other observations

Given that we assume uniformly spaced data, accepted DataFrames would probably need to have a DatetimeIndex type of index, such as the ones returned by pd.date_range (see here).

[jfogelberg]

A long format dataframe could include any type of information in a single dataframe.
For parsing it the user would have to specify which columns are feat_static_cat etc.

| date | item_id | target | product_type (feat_static_cat) | product_price (feat_dynamic_real) |
|------------|---------|--------|--------------------------------|-----------------------------------|
| 2019-01-31 | 0 | 15 | a | 9 |
| 2019-02-28 | 0 | 25 | a | 7 |
| 2019-03-31 | 0 | 27 | a | 7 |
| 2019-01-31 | 1 | 195 | b | 1 |
| 2019-02-28 | 1 | 80 | b | 3 |
| 2019-03-31 | 1 | 114 | b | 2 |

[trewaite]

A long format dataframe could include any type of information in a single dataframe.
For parsing it the user would have to specify which columns are feat_static_cat etc.

date item_id target product_type (feat_static_cat) product_price (feat_dynamic_real)
2019-01-31 0 15 a 9
2019-02-28 0 25 a 7
2019-03-31 0 27 a 7
2019-01-31 1 195 b 1
2019-02-28 1 80 b 3
2019-03-31 1 114 b 2

I like this idea. What if the PandasDataset class we are considering took 2 arguments. First being the dataframe and second being a dict mapping from pandas column name to glutonts feature type?

feature_map={'dt':'date','product_id':'item','price':'feat_dynamic_real'}