Gluon-ts: Multiprocessing broken code

[ehsanmok]

Description

The very recently added parallelized_loaded breaks my already working code in SageMaker

def evaluate(
    model: Predictor, test_data: ListDataset, num_samples: int = 10
) -> Tuple[List[Forecast], List[pd.Series], Dict[str, float], pd.DataFrame]:
    forecast_it, ts_it = make_evaluation_predictions(
        dataset=test_data, predictor=model, num_samples=num_samples
    )
    forecasts = list(forecast_it)
    tss = list(ts_it)
    evaluator = MultivariateEvaluator()
    agg_metrics, item_metrics = evaluator(
        iter(tss), iter(forecasts), num_series=len(test_data)
    )
    return forecasts, tss, agg_metrics, item_metrics

Error message or code output

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/ml/code/train.py", line 137, in <module>
    forecasts, tss, agg_metrics, item_metrics = evaluate(predictor, dataset.test)
  File "/opt/ml/code/utils.py", line 30, in evaluate
    forecasts = list(forecast_it)
  File "/usr/local/lib/python3.6/site-packages/gluonts/model/predictor.py", line 317, in predict
    num_samples=num_samples,
  File "/usr/local/lib/python3.6/site-packages/gluonts/model/forecast_generator.py", line 195, in __call__
    for batch in inference_data_loader:
  File "/usr/local/lib/python3.6/site-packages/gluonts/dataset/loader.py", line 101, in __iter__
    yield from self.parallel_data_loader
  File "/usr/local/lib/python3.6/site-packages/gluonts/dataset/parallelized_loader.py", line 572, in same_process_iter
    for k, v in batch.items()
  File "/usr/local/lib/python3.6/site-packages/gluonts/dataset/parallelized_loader.py", line 572, in <dictcomp>
    for k, v in batch.items()
  File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2765, in as_in_context
    return self.copyto(context)
  File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2632, in copyto
    return _internal._copyto(self, out=hret)
  File "<string>", line 27, in _copyto
  File "/usr/local/lib/python3.6/site-packages/mxnet/_ctypes/ndarray.py", line 107, in _imperative_invoke
    ctypes.byref(out_stypes)))
  File "/usr/local/lib/python3.6/site-packages/mxnet/base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [18:26:25] src/imperative/./imperative_utils.h:146: Operator _copyto inferring shapes failed.
input shapes:
[1,321,-1]
output shapes:
[1,321,-1]
operator attributes:

Stack trace:
  [bt] (0) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x6dfb0b) [0x7f99a9e28b0b]
  [bt] (1) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, mxnet::DispatchMode*)+0x363b) [0x7f99ad0da57b]
  [bt] (2) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0x1db) [0x7f99ad0e24bb]
  [bt] (3) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3847f3f) [0x7f99acf90f3f]
  [bt] (4) /usr/local/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x62) [0x7f99acf91502]
  [bt] (5) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7f9a01ccf4e6]
  [bt] (6) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(ffi_call+0x3f1) [0x7f9a01cce241]
  [bt] (7) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2cf) [0x7f9a01cc58ff]
[2020-04-02 18:26:25.315 ip-10-0-168-31.us-west-2.compute.internal:213 INFO utils.py:25] The end of training job file will not be written for jobs running under SageMaker.
  [bt] (8) /usr/local/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x9829) [0x7f9a01cbc829]

Environment

• Operating system: Amazon linux (amzn1.x86_64 SMP Thu Feb 27 23:49:15 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux)
• Python version: 3.6
• GluonTS version: master
• MXNet: 1.6.0

[AaronSpieler]

Hi, thanks for the bug report!
Cold you maybe provide a minimal working example that reproduces the error?

[AaronSpieler]

Hi, I looked into it, and while I was not able so far to recreate the problem, I think I might have found a solution nonetheless.

Could you please confirm whether the problem was fixed by trying out this version:
!pip install git+https://github.com/AaronSpieler/gluon-ts.git@mp_data_loader_updates

[AtsunoriFujita]

Thanks, @AaronSpieler

I had same problem.

• GluonTS version:0.4.3
• MXNet: 1.6.0

But your version works well on my env.
!pip install git+https://github.com/AaronSpieler/gluon-ts.git@mp_data_loader_updates

[ehsanmok]

Hi @AaronSpieler, thanks for the fix! It works for me now.

[AaronSpieler]

Thx for the quick feedback and bug report @ehsanmok , and thank you too @AtsunoriFujita !