Gluon-ts: Error using `feat_dynamic_real` from extended tutorial when predicting with DeepAR

[andmib]

Description

Using the extended example, when using dynamic real features, the estimator trains but predictions do not work - I get an error "ValueError: all the input array dimensions except for the concatenation axis must match exactly" even when passing the original training data through estimator.predict().

To Reproduce

from gluonts.model.deepar import DeepAREstimator
from gluonts.model.gp_forecaster import GaussianProcessEstimator
from gluonts.model.wavenet import WaveNetEstimator
from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator
from gluonts.trainer import Trainer
from gluonts.evaluation import Evaluator
from gluonts.evaluation.backtest import make_evaluation_predictions
from gluonts.dataset.common import MetaData
import numpy as np

def create_dataset(num_series, num_steps, period=24, mu=1, sigma=0.3):
    # create target: noise + pattern    
    # noise
    noise = np.random.normal(mu, sigma, size=(num_series, num_steps))

    # pattern - sinusoid with different phase
    sin_minumPi_Pi = np.sin(np.tile(np.linspace(-np.pi, np.pi, period), int(num_steps / period)))
    sin_Zero_2Pi = np.sin(np.tile(np.linspace(0, 2 * np.pi, 24), int(num_steps / period)))

    pattern = np.concatenate((np.tile(sin_minumPi_Pi.reshape(1, -1), 
                                      (int(np.ceil(num_series / 2)),1)), 
                              np.tile(sin_Zero_2Pi.reshape(1, -1), 
                                      (int(np.floor(num_series / 2)), 1))
                             ),
                             axis=0
                            )

    target = noise + pattern

    # create time features: use target one period earlier, append with zeros
    feat_dynamic_real = np.concatenate((np.zeros((num_series, period)), 
                                        target[:, :-period]
                                       ), 
                                       axis=1
                                      )

    # create categorical static feats: use the sinusoid type as a categorical feature
    feat_static_cat = np.concatenate((np.zeros(int(np.ceil(num_series / 2))), 
                                      np.ones(int(np.floor(num_series / 2)))
                                     ),
                                     axis=0
                                    )

    return target, feat_dynamic_real, feat_static_cat

# define the parameters of the dataset
custom_ds_metadata = {'num_series': 100,
                      'num_steps': 24 * 7,
                      'prediction_length': 24,
                      'freq': '1H',
                      'start': [pd.Timestamp("01-01-2019", freq='1H') 
                                for _ in range(100)]
                     }

data_out = create_dataset(custom_ds_metadata['num_series'], 
                          custom_ds_metadata['num_steps'],                                                      
                          custom_ds_metadata['prediction_length']
                         )

target, feat_dynamic_real, feat_static_cat = data_out

train_ds = ListDataset([{FieldName.TARGET: target, 
                         FieldName.START: start,
                         FieldName.FEAT_DYNAMIC_REAL: fdr,
                         FieldName.FEAT_STATIC_CAT: fsc} 
                        for (target, start, fdr, fsc) in zip(target[:, :-custom_ds_metadata['prediction_length']], 
                                                             custom_ds_metadata['start'], 
                                                             feat_dynamic_real[:, :-custom_ds_metadata['prediction_length']], 
                                                             feat_static_cat)],
                      freq=custom_ds_metadata['freq'])


estimator = DeepAREstimator(train_ds,
                            prediction_length=24,
                            context_length=48,
                            freq='W',
                            use_feat_dynamic_real=True,
                            trainer=Trainer(ctx="gpu", 
                                            epochs=100,
                                            num_batches_per_epoch=20),
                        )

estimator = estimator.train(train_ds)

pred = estimator.predict(train_ds)
pred = list(pred)

Error Message

ValueError                                Traceback (most recent call last)
<ipython-input-133-61d817cb5fb5> in <module>
     84 
     85 pred = estimator.predict(test_ds)
---> 86 pred = list(pred)
     87 pred = [np.mean(x.samples, axis=0)[0] for x in pred]

~/anaconda3/envs/gluon/lib/python3.6/site-packages/gluonts/model/predictor.py in predict(self, dataset, num_eval_samples)
    301             float_type=self.float_type,
    302         )
--> 303         for batch in inference_data_loader:
    304             inputs = [batch[k] for k in self.input_names]
    305             outputs = self.prediction_net(*inputs).asnumpy()

~/anaconda3/envs/gluon/lib/python3.6/site-packages/gluonts/dataset/loader.py in __iter__(self)
    225     def __iter__(self) -> Iterator[DataBatch]:
    226         buffer = BatchBuffer(self.batch_size, self.ctx, self.float_type)
--> 227         for data_entry in self.transform(iter(self.dataset), is_train=False):
    228             buffer.add(data_entry)
    229             if len(buffer) >= self.batch_size:

~/anaconda3/envs/gluon/lib/python3.6/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    347     ) -> Iterator:
    348         num_idle_transforms = 0
--> 349         for data_entry in data_it:
    350             num_idle_transforms += 1
    351             try:

~/anaconda3/envs/gluon/lib/python3.6/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    302                 yield self.map_transform(data_entry.copy(), is_train)
    303             except Exception as e:
--> 304                 raise e
    305 
    306     @abc.abstractmethod

~/anaconda3/envs/gluon/lib/python3.6/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
    300         for data_entry in data_it:
    301             try:
--> 302                 yield self.map_transform(data_entry.copy(), is_train)
    303             except Exception as e:
    304                 raise e

~/anaconda3/envs/gluon/lib/python3.6/site-packages/gluonts/transform.py in map_transform(self, data, is_train)
    315 
    316     def map_transform(self, data: DataEntry, is_train: bool) -> DataEntry:
--> 317         return self.transform(data)
    318 
    319     @abc.abstractmethod

~/anaconda3/envs/gluon/lib/python3.6/site-packages/gluonts/transform.py in transform(self, data)
    549             if data[fname] is not None
    550         ]
--> 551         output = np.vstack(r)
    552         data[self.output_field] = output
    553         for fname in self.cols_to_drop:

~/anaconda3/envs/gluon/lib/python3.6/site-packages/numpy/core/shape_base.py in vstack(tup)
    232 
    233     """
--> 234     return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
    235 
    236 def hstack(tup):

ValueError: all the input array dimensions except for the concatenation axis must match exactly

Environment

## Environment - Operating system: Ubuntu 16.04 Conda Environment Dump: name: gluon channels: - anaconda - defaults dependencies: - _anaconda_depends=2019.03=py36_0 - _libgcc_mutex=0.1=main - _mutex_mxnet=0.0.20=gpu_mkl - _tflow_select=2.3.0=mkl - alabaster=0.7.12=py36_0 - anaconda=custom=py36_1 - anaconda-client=1.7.2=py36_0 - anaconda-project=0.8.2=py36_0 - asn1crypto=0.24.0=py36_0 - astroid=2.2.5=py36_0 - astropy=3.1.2=py36h7b6447c_0 - babel=2.6.0=py36_0 - backcall=0.1.0=py36_0 - backports=1.0=py36_1 - backports.os=0.1.1=py36_0 - backports.shutil_get_terminal_size=1.0.0=py36_2 - beautifulsoup4=4.7.1=py36_1 - bitarray=0.8.3=py36h14c3975_0 - bkcharts=0.2=py36h735825a_0 - blas=1.0=mkl - bleach=3.1.0=py36_0 - blosc=1.15.0=hd408876_0 - bokeh=1.0.4=py36_0 - boto=2.49.0=py36_0 - bottleneck=1.2.1=py36h035aef0_1 - bzip2=1.0.6=h14c3975_5 - c-ares=1.15.0=h7b6447c_1001 - ca-certificates=2019.5.15=1 - cairo=1.14.12=h8948797_3 - cffi=1.12.2=py36h2e261b9_1 - chardet=3.0.4=py36_1 - cloudpickle=0.8.0=py36_0 - clyent=1.2.2=py36_1 - colorama=0.4.1=py36_0 - contextlib2=0.5.5=py36h6c84a62_0 - cryptography=2.6.1=py36h1ba5d50_0 - cudatoolkit=9.0=h13b8566_0 - cudnn=7.6.0=cuda9.0_0 - curl=7.64.0=hbc83047_2 - cython=0.29.6=py36he6710b0_0 - cytoolz=0.9.0.1=py36h14c3975_1 - dask=1.1.4=py36_1 - dask-core=1.1.4=py36_1 - dbus=1.13.6=h746ee38_0 - decorator=4.4.0=py36_1 - defusedxml=0.5.0=py36_1 - distributed=1.26.0=py36_1 - docutils=0.14=py36hb0f60f5_0 - entrypoints=0.3=py36_0 - et_xmlfile=1.0.1=py36hd6bccc3_0 - expat=2.2.6=he6710b0_0 - fastcache=1.0.2=py36h14c3975_2 - ffmpeg=3.4=h7985aa0_0 - flask=1.0.2=py36_1 - fontconfig=2.13.0=h9420a91_0 - freeglut=3.0.0=hf484d3e_5 - freetype=2.9.1=h8a8886c_1 - fribidi=1.0.5=h7b6447c_0 - gast=0.2.2=py36_0 - get_terminal_size=1.0.0=haa9412d_0 - gevent=1.4.0=py36h7b6447c_0 - glib=2.56.2=hd408876_0 - gmp=6.1.2=h6c8ec71_1 - gmpy2=2.0.8=py36h10f8cd9_2 - google-pasta=0.1.7=py_0 - graphite2=1.3.13=h23475e2_0 - greenlet=0.4.15=py36h7b6447c_0 - gst-plugins-base=1.14.0=hbbd80ab_1 - gstreamer=1.14.0=hb453b48_1 - h5py=2.9.0=py36h7918eee_0 - harfbuzz=1.8.8=hffaf4a1_0 - hdf5=1.10.2=hba1933b_1 - heapdict=1.0.0=py36_2 - html5lib=1.0.1=py36_0 - icu=58.2=h9c2bf20_1 - idna=2.6=py36h82fb2a8_1 - imageio=2.5.0=py36_0 - imagesize=1.1.0=py36_0 - importlib_metadata=0.8=py36_0 - intel-openmp=2018.0.3=0 - ipykernel=5.1.0=py36h39e3cac_0 - ipython=7.4.0=py36h39e3cac_0 - ipython_genutils=0.2.0=py36hb52b0d5_0 - ipywidgets=7.4.2=py36_0 - isort=4.3.16=py36_0 - itsdangerous=1.1.0=py36_0 - jasper=2.0.14=h07fcdf6_1 - jbig=2.1=hdba287a_0 - jdcal=1.4=py36_0 - jedi=0.13.3=py36_0 - jeepney=0.4=py36_0 - jinja2=2.10=py36_0 - jpeg=9b=h024ee3a_2 - jsonschema=3.0.1=py36_0 - jupyter=1.0.0=py36_7 - jupyter_client=5.2.4=py36_0 - jupyter_console=6.0.0=py36_0 - jupyter_core=4.4.0=py36_0 - jupyterlab=0.35.4=py36hf63ae98_0 - jupyterlab_server=0.2.0=py36_0 - keras-applications=1.0.8=py_0 - keras-preprocessing=1.1.0=py_1 - keyring=18.0.0=py36_0 - krb5=1.16.1=h173b8e3_7 - lazy-object-proxy=1.3.1=py36h14c3975_2 - libcurl=7.64.0=h20c2e04_2 - libedit=3.1.20181209=hc058e9b_0 - libffi=3.2.1=hd88cf55_4 - libgcc-ng=8.2.0=hdf63c60_1 - libgfortran-ng=7.3.0=hdf63c60_0 - libglu=9.0.0=hf484d3e_1 - libmklml=2018.0.3=0 - libmxnet=1.2.1=gpu_mkl_he87abd8_1 - libopencv=3.4.1=h8fa1ad8_3 - libopus=1.3=h7b6447c_0 - libpng=1.6.36=hbc83047_0 - libprotobuf=3.5.2=h6f1eeef_0 - libsodium=1.0.16=h1bed415_0 - libssh2=1.8.0=h1ba5d50_4 - libstdcxx-ng=8.2.0=hdf63c60_1 - libtiff=4.0.10=h2733197_2 - libtool=2.4.6=h7b6447c_5 - libuuid=1.0.3=h1bed415_2 - libvpx=1.7.0=h439df22_0 - libxcb=1.13=h1bed415_1 - libxml2=2.9.9=he19cac6_0 - libxslt=1.1.33=h7d1a2b0_0 - llvmlite=0.28.0=py36hd408876_0 - locket=0.2.0=py36h787c0ad_1 - lxml=4.3.2=py36hefd8a0e_0 - lzo=2.10=h49e0be7_2 - markdown=3.1.1=py36_0 - markupsafe=1.1.1=py36h7b6447c_0 - mccabe=0.6.1=py36_1 - mistune=0.8.4=py36h7b6447c_0 - mkl=2019.3=199 - mkl-dnn=0.14=h6bb024c_0 - mkl-service=1.1.2=py36he904b0f_5 - mkl_fft=1.0.10=py36ha843d7b_0 - mkl_random=1.0.2=py36hd81dba3_0 - more-itertools=6.0.0=py36_0 - mpc=1.1.0=h10f8cd9_1 - mpfr=4.0.1=hdf1c602_3 - mpmath=1.1.0=py36_0 - msgpack-python=0.6.1=py36hfd86e86_1 - multipledispatch=0.6.0=py36_0 - nbconvert=5.4.1=py36_3 - nbformat=4.4.0=py36h31c9010_0 - ncurses=6.1=he6710b0_1 - networkx=2.2=py36_1 - nltk=3.4=py36_1 - nose=1.3.7=py36_2 - notebook=5.7.8=py36_0 - numba=0.43.1=py36h962f231_0 - numexpr=2.6.9=py36h9e4a6bb_0 - numpy-base=1.16.4=py36hde5b4d6_0 - numpydoc=0.8.0=py36_0 - olefile=0.46=py36_0 - openpyxl=2.6.1=py36_1 - openssl=1.1.1=h7b6447c_0 - packaging=19.0=py36_0 - pandoc=2.2.3.2=0 - pandocfilters=1.4.2=py36_1 - pango=1.42.4=h049681c_0 - parso=0.3.4=py36_0 - partd=0.3.10=py36_1 - path.py=11.5.0=py36_0 - pathlib2=2.3.3=py36_0 - patsy=0.5.1=py36_0 - pcre=8.43=he6710b0_0 - pep8=1.7.1=py36_0 - pexpect=4.6.0=py36_0 - pickleshare=0.7.5=py36_0 - pillow=5.4.1=py36h34e0f95_0 - pip=19.0.3=py36_0 - pixman=0.38.0=h7b6447c_0 - pluggy=0.9.0=py36_0 - ply=3.11=py36_0 - prometheus_client=0.6.0=py36_0 - prompt_toolkit=2.0.9=py36_0 - psutil=5.6.1=py36h7b6447c_0 - ptyprocess=0.6.0=py36_0 - py=1.8.0=py36_0 - py-mxnet=1.2.1=py36h6c82189_0 - pycosat=0.6.3=py36h14c3975_0 - pycparser=2.19=py36_0 - pycrypto=2.6.1=py36h14c3975_9 - pycurl=7.43.0.2=py36h1ba5d50_0 - pygments=2.3.1=py36_0 - pylint=2.3.1=py36_0 - pyodbc=4.0.26=py36he6710b0_0 - pyopenssl=19.0.0=py36_0 - pyqt=5.9.2=py36h05f1152_2 - pyrsistent=0.14.11=py36h7b6447c_0 - pysocks=1.6.8=py36_0 - pytables=3.5.1=py36h71ec239_0 - pytest-arraydiff=0.3=py36h39e3cac_0 - pytest-astropy=0.5.0=py36_0 - pytest-doctestplus=0.3.0=py36_0 - pytest-openfiles=0.3.2=py36_0 - pytest-remotedata=0.3.1=py36_0 - python=3.6.8=h0371630_0 - python-dateutil=2.8.0=py36_0 - pytz=2018.9=py36_0 - pywavelets=1.0.2=py36hdd07704_0 - pyyaml=5.1=py36h7b6447c_0 - pyzmq=18.0.0=py36he6710b0_0 - qt=5.9.7=h5867ecd_1 - qtawesome=0.5.7=py36_1 - qtconsole=4.4.3=py36_0 - qtpy=1.7.0=py36_1 - readline=7.0=h7b6447c_5 - rope=0.12.0=py36_0 - ruamel_yaml=0.15.46=py36h14c3975_0 - scikit-image=0.14.2=py36he6710b0_0 - scikit-learn=0.20.3=py36hd81dba3_0 - seaborn=0.9.0=py36_0 - secretstorage=3.1.1=py36_0 - send2trash=1.5.0=py36_0 - simplegeneric=0.8.1=py36_2 - singledispatch=3.4.0.3=py36h7a266c3_0 - sip=4.19.8=py36hf484d3e_0 - snappy=1.1.7=hbae5bb6_3 - snowballstemmer=1.2.1=py36h6febd40_0 - sortedcollections=1.1.2=py36_0 - sortedcontainers=2.1.0=py36_0 - soupsieve=1.8=py36_0 - sphinxcontrib=1.0=py36_1 - sphinxcontrib-websupport=1.1.0=py36_1 - spyder=3.3.3=py36_0 - spyder-kernels=0.4.2=py36_0 - sqlite=3.27.2=h7b6447c_0 - statsmodels=0.9.0=py36h035aef0_0 - sympy=1.3=py36_0 - tblib=1.3.2=py36h34cf8b6_0 - tensorboard=1.14.0=py36hf484d3e_0 - tensorflow=1.14.0=mkl_py36h2526735_0 - tensorflow-base=1.14.0=mkl_py36h7ce6ba3_0 - tensorflow-estimator=1.14.0=py_0 - termcolor=1.1.0=py36_1 - terminado=0.8.1=py36_1 - testpath=0.4.2=py36_0 - tk=8.6.8=hbc83047_0 - toolz=0.9.0=py36_0 - traitlets=4.3.2=py36h674d592_0 - unicodecsv=0.14.1=py36ha668878_0 - unixodbc=2.3.7=h14c3975_0 - urllib3=1.22=py36hbe7ace6_0 - wcwidth=0.1.7=py36hdf4376a_0 - webencodings=0.5.1=py36_1 - werkzeug=0.14.1=py36_0 - wheel=0.33.1=py36_0 - widgetsnbextension=3.4.2=py36_0 - wrapt=1.11.1=py36h7b6447c_0 - wurlitzer=1.0.2=py36_0 - xlrd=1.2.0=py36_0 - xlsxwriter=1.1.5=py36_0 - xlwt=1.3.0=py36h7b00a1f_0 - xz=5.2.4=h14c3975_4 - yaml=0.1.7=had09818_2 - zeromq=4.3.1=he6710b0_3 - zict=0.1.4=py36_0 - zipp=0.3.3=py36_1 - zlib=1.2.11=h7b6447c_3 - zstd=1.3.7=h0b5b093_0 - pip: - absl-py==0.7.0 - alembic==1.0.10 - amqp==2.5.0 - apipkg==1.5 - appdirs==1.4.3 - argh==0.26.2 - argparse==1.4.0 - astor==0.7.1 - atomicwrites==1.2.1 - attrs==18.2.0 - black==18.6b4 - boto3==1.9.163 - botocore==1.12.164 - cachetools==3.0.0 - certifi==2018.11.29 - click==6.7 - commonmark==0.9.0 - convertdate==2.1.3 - coverage==4.5.3 - cx-oracle==7.0.0 - cycler==0.10.0 - dataclasses==0.6 - debtcollector==1.21.0 - dnspython==1.16.0 - dogpile-cache==0.7.1 - ephem==3.7.6.0 - eventlet==0.25.0 - execnet==1.6.0 - extras==1.0.0 - fasteners==0.15 - fbprophet==0.5 - feather-format==0.4.0 - fixtures==3.0.0 - flake8==3.4.1 - future==0.17.1 - futurist==1.8.1 - gluon==1.1.0 - gluonts==0.3.4.dev5+g4d044a2 - google-api-core==1.14.2 - google-auth==1.6.2 - google-auth-oauthlib==0.2.0 - google-cloud-bigquery==1.18.0 - google-cloud-core==1.0.3 - google-resumable-media==0.3.2 - googleapis-common-protos==1.6.0 - grpcio==1.18.0 - holidays==0.9.11 - iso8601==0.1.12 - jmespath==0.9.4 - keras==2.2.5 - keystoneauth1==3.14.0 - keystonemiddleware==6.0.0 - kiwisolver==1.1.0 - kombu==4.6.1 - linecache2==1.0.0 - livereload==2.6.1 - logutils==0.3.5 - lunardate==0.2.0 - mako==1.0.12 - matplotlib==3.1.1 - merf==0.3 - monotonic==1.5 - mxnet==1.4.1 - mxnet-cu101==1.4.1 - mxtheme==0.2 - mypy==0.630 - mypy-extensions==0.4.1 - nbsphinx==0.3.5 - netaddr==0.7.19 - netifaces==0.10.9 - numpy==1.14.6 - oauthlib==3.0.1 - os-service-types==1.7.0 - oslo-cache==1.35.0 - oslo-concurrency==3.29.1 - oslo-config==6.9.0 - oslo-context==2.22.1 - oslo-db==5.0.0 - oslo-i18n==3.23.1 - oslo-log==3.44.0 - oslo-messaging==9.7.1 - oslo-middleware==3.38.0 - oslo-policy==2.2.0 - oslo-serialization==2.29.1 - oslo-service==1.40.0 - oslo-utils==3.41.0 - oslo-versionedobjects==1.36.0 - pandas==0.25.1 - pandas-gbq==0.10.0 - paste==3.0.8 - pastedeploy==2.0.1 - pathtools==0.1.2 - pbr==5.2.1 - pecan==1.3.3 - port-for==0.3.1 - prettytable==0.7.2 - protobuf==3.8.0 - pyarrow==0.13.0 - pyasn1==0.4.5 - pyasn1-modules==0.2.3 - pycadf==2.9.0 - pycodestyle==2.3.1 - pydantic==0.28 - pydata-google-auth==0.1.3 - pyflakes==1.5.0 - pyinotify==0.9.6 - pymssql==2.1.4 - pyparsing==2.4.2 - pystan==2.19.0.0 - pytest==3.10.1 - pytest-cov==2.6.1 - pytest-forked==1.0.2 - pytest-runner==2.11.1 - pytest-timeout==1.2.1 - pytest-xdist==1.27.0 - python-editor==1.0.4 - python-etcd==0.4.5 - python-graphviz==0.8.4 - python-keystoneclient==3.19.0 - python-mimeparse==1.6.0 - recommonmark==0.5.0 - repoze-lru==0.7 - requests==2.22.0 - requests-oauthlib==1.2.0 - rfc3986==1.3.2 - routes==2.4.1 - rpy2==3.0.4 - rsa==4.0 - s3transfer==0.2.1 - scipy==1.2.0 - setuptools==41.2.0 - setuptools-git==1.2 - six==1.12.0 - sphinx==1.7.9 - sphinx-autobuild==0.7.1 - sphinx-autorun==1.1.0 - sphinx-gallery==0.3.1 - sphinx-rtd-theme==0.4.3 - sqlalchemy==1.0.19 - sqlalchemy-migrate==0.12.0 - sqlparse==0.3.0 - statsd==3.3.0 - stevedore==1.30.1 - tempita==0.5.2 - testresources==2.0.1 - testscenarios==0.5.0 - testtools==2.3.0 - toml==0.10.0 - tornado==5.1.1 - tqdm==4.35.0 - traceback2==1.4.0 - tsfresh==0.11.2 - typed-ast==1.1.2 - tzlocal==1.5.1 - ujson==1.35 - unittest2==1.1.0 - vine==1.3.0 - waitress==1.3.0 - watchdog==0.9.0 - webob==1.8.5 - webtest==2.0.33 - wsme==0.9.3 - xgboost==0.90 - yappi==1.0

[benidis]

Hi andmib,

The issue is that you have set use_feat_dynamic_real=True and then you invoke the predict method directly which does not apply the correct transformations to the fields of the dataset. To solve this you can use the following:

predictor = estimator.train(train_ds)

forecast_it, ts_it = make_evaluation_predictions(
    dataset=train_ds,  # test dataset
    predictor=predictor,  # predictor
    num_eval_samples=100,  # number of sample paths we want for evaluation
)

tss = list(ts_it)
forecasts = list(forecast_it)

The make_evaluation_predictions function applies the transformations and then invokes the predict method.

You can find all of the above in the tutorial notebooks.

Some side notes: you pass the train_ds as an input to DeepAREstimator. This is not needed and is ignored. Also, you use the same dataset train_ds for training and prediction.

[andmib]

Hey @benidis - thank you for the prompt reply. That makes perfect sense!

(passing train_ds to the predict method was just to show that all of the dynamic feature dimensions were the same and I was still getting the erro)

[andmib]

@benidis Side note, somewhat related - when I run train_ds.calc_stats() from the extended example, it says that the num_dynamic_feat_real is 0, even though the example clearly has dynamic features.

[gabrielcrds]

Hello,
I'm getting the same error, but at train... Can you help me?

Code:

print(">>>>DEBUG: Dataset columns:", temp_df.columns)
print("\n>>>>DEBUG: train_e (used to get pred_lenght)", train_e)
print("\n>>>>DEBUG: Dataset head:\n", temp_df.head())

pred_lenght = temp_df[temp_df.ds >= train_e].shape[0]

print("\n>>>>DEBUG: pred_lenght:", pred_lenght)
start = pd.Timestamp(temp_df.iloc[0]['ds'], freq='D')

print(">>>>DEBUG: Series shape:", temp_df.y.values.shape)
print(">>>>DEBUG: feat_dynamic_real shape:", feat.values.shape)

# train dataset: cut the last window of length "prediction_length", add "target" and "start" fields
train_ds = common.ListDataset(
    [{
        "start": start,
        "target": temp_df[:-pred_lenght].y.values,
        "feat_dynamic_real": feat[:-pred_lenght].values
    }],
    freq="D"
)

# test dataset: use the whole dataset, add "target" and "start" fields
test_ds = common.ListDataset(
    [{
        "start": start,
        "target": temp_df.y.values,
        "feat_dynamic_real": feat.values
    }],
    freq="D"
)    

estimator = deepar.DeepAREstimator(
    freq="D",
    prediction_length=pred_lenght,
    trainer=Trainer(epochs=10),
    num_layers=10,
    num_cells=40,
    use_feat_dynamic_real=True
)

predictor = estimator.train(train_ds)

Error

```
INFO:root:Using CPU
INFO:root:Start model training
INFO:root:Number of parameters in DeepARTrainingNetwork: 67223
INFO:root:Epoch[0] Learning rate is 0.001
0%| | 0/50 [00:00

DEBUG: Dataset columns: Index(['ds', 'y'], dtype='object')

DEBUG: train_e (used to get pred_lenght) 2018-01-01

DEBUG: Dataset head:
ds y
0 2014-06-01 78
1 2014-06-02 192
2 2014-06-03 181
3 2014-06-04 175
4 2014-06-05 141

DEBUG: pred_lenght: 31
DEBUG: Series shape: (1341,)
DEBUG: feat_dynamic_real shape: (1341, 4)

---

ValueError Traceback (most recent call last)
in
40 )
41
---> 42 predictor = estimator.train(train_ds)

~/anaconda3/lib/python3.7/site-packages/gluonts/model/estimator.py in train(self, training_data)
201 def train(self, training_data: Dataset) -> Predictor:
202
--> 203 return self.train_model(training_data).predictor

~/anaconda3/lib/python3.7/site-packages/gluonts/model/estimator.py in train_model(self, training_data)
187 net=trained_net,
188 input_names=get_hybrid_forward_input_names(trained_net),
--> 189 train_iter=training_data_loader,
190 )
191

~/anaconda3/lib/python3.7/site-packages/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter)
243
244 with tqdm(train_iter) as it:
--> 245 for batch_no, data_entry in enumerate(it, start=1):
246 if self.halt:
247 break

~/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py in __iter__(self)
1003 """), fp_write=getattr(self.fp, 'write', sys.stderr.write))
1004
-> 1005 for obj in iterable:
1006 yield obj
1007 # Update and possibly print the progressbar.

~/anaconda3/lib/python3.7/site-packages/gluonts/dataset/loader.py in __iter__(self)
186 assert self._cur_iter is not None
187 while True:
--> 188 data_entry = next(self._cur_iter)
189 self._buffer.add(data_entry)
190 if (

~/anaconda3/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
345 ) -> Iterator:
346 num_idle_transforms = 0
--> 347 for data_entry in data_it:
348 num_idle_transforms += 1
349 try:

~/anaconda3/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
300 yield self.map_transform(data_entry.copy(), is_train)
301 except Exception as e:
--> 302 raise e
303
304 @abc.abstractmethod

~/anaconda3/lib/python3.7/site-packages/gluonts/transform.py in __call__(self, data_it, is_train)
298 for data_entry in data_it:
299 try:
--> 300 yield self.map_transform(data_entry.copy(), is_train)
301 except Exception as e:
302 raise e

~/anaconda3/lib/python3.7/site-packages/gluonts/transform.py in map_transform(self, data, is_train)
313
314 def map_transform(self, data: DataEntry, is_train: bool) -> DataEntry:
--> 315 return self.transform(data)
316
317 @abc.abstractmethod

~/anaconda3/lib/python3.7/site-packages/gluonts/transform.py in transform(self, data)
547 if data[fname] is not None
548 ]
--> 549 output = np.vstack(r)
550 data[self.output_field] = output
551 for fname in self.cols_to_drop:

~/anaconda3/lib/python3.7/site-packages/numpy/core/shape_base.py in vstack(tup)
232
233 """
--> 234 return _nx.concatenate([atleast_2d(_m) for _m in tup], 0)
235
236 def hstack(tup):

ValueError: all the input array dimensions except for the concatenation axis must match exactly```

[benidis]

Hi gabrielcrds,

I reproduced your error. The issue is in the shape of feat_dynamic_real. The correct format should be:

target shape: (num_series, ts_length)
feat_dynamic_real shape: (num_series, num_features, ts_length)

In your case you have only one time series so num_series=1 and you can ignore this axis (i.e., no need to have a shape of the form (1, ...)). So, target should have shape (ts_length,) which is correct in your example, and feat_dynamic_real should have shape (num_features, ts_length) which is wrong in your example since you have (ts_length, num_features).

I guess something like "feat_dynamic_real": feat.values.reshape(4, -1) would do the job.

Note that the tutorials had a mistake (in the text) and indicated a wrong format (the one you used). However this was fixed in PR #269 and the website will be updated accordingly.

[benidis]

@andmib I am looking into the train_ds.calc_stats(). If there is a bug I will probably open a new issue.

[dai-ichiro]

I think this is not a problem with shape of 'feat_dynamic_real', but a problem with length of that.

A code below works fine.

import pandas as pd
import numpy as np
from gluonts.dataset.common import ListDataset
from gluonts.transform import FieldName
from gluonts.dataset.util import to_pandas

train_target = np.random.rand(48,168)
train_feat_dynamic_real = np.random.rand(48,168)

start = pd.Timestamp("01-01-2019")

train_ds = ListDataset([{FieldName.TARGET: target,
                         FieldName.START: start,
                         FieldName.FEAT_DYNAMIC_REAL: fdr}
                        for (target, fdr) in zip(train_target, train_feat_dynamic_real)],
                      freq= '1H')

test_target = np.random.rand(2,168)
test_feat_dynamic_real = np.random.rand(2,168)

test_ds = ListDataset([{FieldName.TARGET: target,
                        FieldName.START: start,
                        FieldName.FEAT_DYNAMIC_REAL: fdr}
                        for (target, fdr) in zip(test_target, test_feat_dynamic_real)],
                      freq= '1H')

from gluonts.model.deepar import DeepAREstimator
from gluonts.trainer import Trainer

estimator = DeepAREstimator(freq='1H', 
                            prediction_length=24, 
                            context_length=48, 
                            use_feat_dynamic_real = True,
                            trainer=Trainer(epochs=5))
predictor = estimator.train(training_data=train_ds)

test_target = np.random.rand(2,168)
test_feat_dynamic_real = np.random.rand(2,168+24)

test_ds = ListDataset([{FieldName.TARGET: target,
                        FieldName.START: start,
                        FieldName.FEAT_DYNAMIC_REAL: fdr}
                        for (target, fdr) in zip(test_target, test_feat_dynamic_real)],
                      freq= '1H')

pred = predictor.predict(test_ds)
pred = list(pred)

[benidis]

@dai-ichiro thank you for your input. The previous issue was indeed due to the shape of the features.

Your code works fine. Here is why:

1. TRAINING: the target and the dynamic features should have the same length. As I mentioned earlier the correct shapes should be the following:

target shape: (num_series, ts_length)
feat_dynamic_real shape: (num_series, num_features, ts_length)

In your case you have used

train_target = np.random.rand(48,168)
train_feat_dynamic_real = np.random.rand(48,168)

which implies that num_series=48, num_features=1 (ignored dimension) and ts_length=168.

1. INFERENCE: note that during inference the target in the future is not known but the features in the future are assumed to be known. Therefore, the features should have length equal to ts_length + prediction_range. Now, you have two options.

_First option:_ you have a test dataset that the target and the dynamic features have the same length and you want to predict the last (known) window of target in order to compare it with the true values and evaluate your model. In this case the easiest option is to use the make_evaluation_predictions function. This function initially removes the last window of length prediction_length from the target considering it the future window that should be unknown and needs to be predicted. Notice that by doing this the features now have a larger length (they expand into the future window). Then, the function calls the predict method to predict the future values that it has earlier removed. You can see this in the extended tutorial (section 4.1).

_Second option:_ you have a test dataset where the length of the features is larger than the target length (assuming that both target and features start in the same date and the features expand more into the future). In this case you can use directly the predict method since the future window of your target is not there eitherway.

In your case you have

test_target = np.random.rand(2,168)
test_feat_dynamic_real = np.random.rand(2,168+24)

test_ds = ListDataset([{FieldName.TARGET: target,
                        FieldName.START: start,
                        FieldName.FEAT_DYNAMIC_REAL: fdr}
                        for (target, fdr) in zip(test_target, test_feat_dynamic_real)],
                      freq= '1H')

pred = predictor.predict(test_ds)

which falls under the second option. Note that the difference in length of the target and features should be equal to the prediction_length of predictor else you will get an error.

Also, in your code you define the test_ds twice (the second version is used). You define the first test_ds as

test_target = np.random.rand(2,168)
test_feat_dynamic_real = np.random.rand(2,168)

test_ds = ListDataset([{FieldName.TARGET: target,
                        FieldName.START: start,
                        FieldName.FEAT_DYNAMIC_REAL: fdr}
                        for (target, fdr) in zip(test_target, test_feat_dynamic_real)],
                      freq= '1H')

This should not work with the predict method but it should work with the make_evaluation_predictions function.

We are working on standarizing these points that cause confusion and give the correct feedback to the user. We will make clear all of these things in the following weeks and update the tutorials accordingly so please always check the latest versions.

[dai-ichiro]

@benidis many tanks for your quick reply.

[QiuSYang]

@dai-ichiro thank you for your input. The previous issue was indeed due to the shape of the features.

Your code works fine. Here is why:

1. TRAINING: the target and the dynamic features should have the same length. As I mentioned earlier the correct shapes should be the following:

target shape: (num_series, ts_length)
feat_dynamic_real shape: (num_series, num_features, ts_length)

In your case you have used

train_target = np.random.rand(48,168)
train_feat_dynamic_real = np.random.rand(48,168)

which implies that num_series=48, num_features=1 (ignored dimension) and ts_length=168.

1. INFERENCE: note that during inference the target in the future is not known but the features in the future are assumed to be known. Therefore, the features should have length equal to ts_length + prediction_range. Now, you have two options.

_First option:_ you have a test dataset that the target and the dynamic features have the same length and you want to predict the last (known) window of target in order to compare it with the true values and evaluate your model. In this case the easiest option is to use the make_evaluation_predictions function. This function initially removes the last window of length prediction_length from the target considering it the future window that should be unknown and needs to be predicted. Notice that by doing this the features now have a larger length (they expand into the future window). Then, the function calls the predict method to predict the future values that it has earlier removed. You can see this in the extended tutorial (section 4.1).

_Second option:_ you have a test dataset where the length of the features is larger than the target length (assuming that both target and features start in the same date and the features expand more into the future). In this case you can use directly the predict method since the future window of your target is not there eitherway.

In your case you have

test_target = np.random.rand(2,168)
test_feat_dynamic_real = np.random.rand(2,168+24)

test_ds = ListDataset([{FieldName.TARGET: target,
                        FieldName.START: start,
                        FieldName.FEAT_DYNAMIC_REAL: fdr}
                        for (target, fdr) in zip(test_target, test_feat_dynamic_real)],
                      freq= '1H')

pred = predictor.predict(test_ds)

which falls under the second option. Note that the difference in length of the target and features should be equal to the prediction_length of predictor else you will get an error.

Also, in your code you define the test_ds twice (the second version is used). You define the first test_ds as

test_target = np.random.rand(2,168)
test_feat_dynamic_real = np.random.rand(2,168)

test_ds = ListDataset([{FieldName.TARGET: target,
                        FieldName.START: start,
                        FieldName.FEAT_DYNAMIC_REAL: fdr}
                        for (target, fdr) in zip(test_target, test_feat_dynamic_real)],
                      freq= '1H')

This should not work with the predict method but it should work with the make_evaluation_predictions function.

We are working on standarizing these points that cause confusion and give the correct feedback to the user. We will make clear all of these things in the following weeks and update the tutorials accordingly so please always check the latest versions.
@benidis When we make time series prediction, we can't get the feat_dynamic_real of the future time. Is there a good way to solve this problem?