Gluon-ts: Problems using GPU with Amazon SageMaker on AWS-instance "ml.p2.xlarge".

[tm1611]

Description

Using Amazon SageMaker on an AWS GPU-instance "ml.p2.xlarge", I was not able to run the example benchmark_m4.py script (copy/pasted in SageMaker) on GPU.

To Reproduce

After starting the instance:

!pip install gluonts

Next cell: paste the slightly modified script benchmark_m4.py with a little modification:

estimators = [
    partial(
        DeepAREstimator,
        trainer=Trainer(
            epochs=epochs, 
            num_batches_per_epoch=num_batches_per_epoch,
            ctx="gpu"
        ),
    ),
]

(without specifying the context this works fine, but is only running on CPU)

Error Message

INFO:root:using dataset already processed in path /home/ec2-user/.mxnet/gluon-ts/datasets/m4_quarterly.
INFO:root:Start model training
INFO:root:using dataset already processed in path /home/ec2-user/.mxnet/gluon-ts/datasets/m4_yearly.
INFO:root:Start model training
evaluating gluonts.model.deepar._estimator.DeepAREstimator(cardinality=[24000], cell_type="lstm", context_length=None, distr_output=gluonts.distribution.student_t.StudentTOutput(), dropout_rate=0.1, embedding_dimension=20, freq="3M", lags_seq=None, num_cells=40, num_layers=2, num_parallel_samples=100, prediction_length=8, scaling=True, time_features=None, trainer=gluonts.trainer._base.Trainer(batch_size=32, clip_gradient=10.0, ctx=mxnet.context.Context("gpu", 0), epochs=100, hybridize=True, init="xavier", learning_rate=0.001, learning_rate_decay_factor=0.5, minimum_learning_rate=5e-05, num_batches_per_epoch=200, patience=10, weight_decay=1e-08), use_feat_dynamic_real=False, use_feat_static_cat=True) on TrainDatasets(metadata=<MetaData freq='3M' target=None feat_static_cat=[<CategoricalFeatureInfo name='feat_static_cat' cardinality='24000'>] feat_static_real=[] feat_dynamic_real=[] feat_dynamic_cat=[] prediction_length=8>, train=<gluonts.dataset.common.FileDataset object at 0x7f9377c9e748>, test=<gluonts.dataset.common.FileDataset object at 0x7f9377c53208>)
[22:17:01] src/ndarray/ndarray.cc:1279: GPU is not enabled

Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23d55a) [0x7f93951c155a]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23dbc1) [0x7f93951c1bc1]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::CopyFromTo(mxnet::NDArray const&, mxnet::NDArray const&, int, bool)+0x723) [0x7f9397cf7623]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::PushFComputeEx(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)+0x47e) [0x7f9397bad59e]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::InvokeOp(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, mxnet::DispatchMode, mxnet::OpStatePtr)+0x839) [0x7f9397bb28f9]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0x38c) [0x7f9397bb317c]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2b34989) [0x7f9397ab8989]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x6f) [0x7f9397ab8f7f]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f93d58efec0]
[bt] (9) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f93d58ef87d]


evaluating gluonts.model.deepar._estimator.DeepAREstimator(cardinality=[23000], cell_type="lstm", context_length=None, distr_output=gluonts.distribution.student_t.StudentTOutput(), dropout_rate=0.1, embedding_dimension=20, freq="12M", lags_seq=None, num_cells=40, num_layers=2, num_parallel_samples=100, prediction_length=6, scaling=True, time_features=None, trainer=gluonts.trainer._base.Trainer(batch_size=32, clip_gradient=10.0, ctx=mxnet.context.Context("gpu", 0), epochs=100, hybridize=True, init="xavier", learning_rate=0.001, learning_rate_decay_factor=0.5, minimum_learning_rate=5e-05, num_batches_per_epoch=200, patience=10, weight_decay=1e-08), use_feat_dynamic_real=False, use_feat_static_cat=True) on TrainDatasets(metadata=<MetaData freq='12M' target=None feat_static_cat=[<CategoricalFeatureInfo name='feat_static_cat' cardinality='23000'>] feat_static_real=[] feat_dynamic_real=[] feat_dynamic_cat=[] prediction_length=6>, train=<gluonts.dataset.common.FileDataset object at 0x7f937812ce48>, test=<gluonts.dataset.common.FileDataset object at 0x7f9377c53208>)
[22:17:01] src/ndarray/ndarray.cc:1279: GPU is not enabled

Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23d55a) [0x7f93951c155a]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23dbc1) [0x7f93951c1bc1]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::CopyFromTo(mxnet::NDArray const&, mxnet::NDArray const&, int, bool)+0x723) [0x7f9397cf7623]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::imperative::PushFComputeEx(std::function<void (nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> > const&)> const&, nnvm::Op const*, nnvm::NodeAttrs const&, mxnet::Context const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::engine::Var*, std::allocator<mxnet::engine::Var*> > const&, std::vector<mxnet::Resource, std::allocator<mxnet::Resource> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&)+0x47e) [0x7f9397bad59e]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::InvokeOp(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, mxnet::DispatchMode, mxnet::OpStatePtr)+0x839) [0x7f9397bb28f9]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&)+0x38c) [0x7f9397bb317c]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2b34989) [0x7f9397ab8989]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x6f) [0x7f9397ab8f7f]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f93d58efec0]
[bt] (9) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f93d58ef87d]


---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-15-b3fbc3bdf424> in <module>()
     88             "MASE",
     89             "sMAPE",
---> 90             "MSIS",
     91         ]
     92     ]

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2999             if is_iterator(key):
   3000                 key = list(key)
-> 3001             indexer = self.loc._convert_to_indexer(key, axis=1, raise_missing=True)
   3002 
   3003         # take() does not accept boolean indexers

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pandas/core/indexing.py in _convert_to_indexer(self, obj, axis, is_setter, raise_missing)
   1283                 # When setting, missing keys are not allowed, even with .loc:
   1284                 kwargs = {"raise_missing": True if is_setter else raise_missing}
-> 1285                 return self._get_listlike_indexer(obj, axis, **kwargs)[1]
   1286         else:
   1287             try:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pandas/core/indexing.py in _get_listlike_indexer(self, key, axis, raise_missing)
   1090 
   1091         self._validate_read_indexer(
-> 1092             keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing
   1093         )
   1094         return keyarr, indexer

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pandas/core/indexing.py in _validate_read_indexer(self, key, indexer, axis, raise_missing)
   1175                 raise KeyError(
   1176                     "None of [{key}] are in the [{axis}]".format(
-> 1177                         key=key, axis=self.obj._get_axis_name(axis)
   1178                     )
   1179                 )

KeyError: "None of [Index(['dataset', 'estimator', 'RMSE', 'mean_wQuantileLoss', 'MASE', 'sMAPE',\n       'MSIS'],\n      dtype='object')] are in the [columns]"

Other

In addition, before installing gluonts (from https://beta.mxnet.io/guide/crash-course/6-use_gpus.html):

x = nd.ones((3,4), ctx=gpu())
x

[[1. 1. 1. 1.]
 [1. 1. 1. 1.]
 [1. 1. 1. 1.]]
<NDArray 3x4 @gpu(0)>



md5-ef527f94575bc1711a927f96fb5ad6be

---

MXNetError Traceback (most recent call last)
in ()
5
6
----> 7 x = nd.ones((3,4), ctx=gpu())
8 x

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py in ones(shape, ctx, dtype, *kwargs)
2419 dtype = mx_real_t if dtype is None else dtype
2420 # pylint: disable= no-member, protected-access
-> 2421 return _internal._ones(shape=shape, ctx=ctx, dtype=dtype, *kwargs)
2422 # pylint: enable= no-member, protected-access
2423

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/register.py in _ones(shape, ctx, dtype, out, name, **kwargs)

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/_ctypes/ndarray.py in _imperative_invoke(handle, ndargs, keys, vals, out)
90 c_str_array(keys),
91 c_str_array([str(s) for s in vals]),
---> 92 ctypes.byref(out_stypes)))
93
94 if original_output is not None:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/base.py in check_call(ret)
250 """
251 if ret != 0:
--> 252 raise MXNetError(py_str(_LIB.MXGetLastError()))
253
254

MXNetError: [22:29:51] src/imperative/imperative.cc:79: Operator _ones is not implemented for GPU.

Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23d55a) [0x7f93951c155a]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x23dbc1) [0x7f93951c1bc1]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::InvokeOp(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector > const&, std::vector > const&, std::vector > const&, mxnet::DispatchMode, mxnet::OpStatePtr)+0x9fb) [0x7f9397bb2abb]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Imperative::Invoke(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector > const&, std::vector > const&)+0x38c) [0x7f9397bb317c]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2b34989) [0x7f9397ab8989]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXImperativeInvokeEx+0x6f) [0x7f9397ab8f7f]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f93d58efec0]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f93d58ef87d]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f93d5b04e2e]
[bt] (9) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x12865) [0x7f93d5b05865]
```

Environment

• Amazon SageMaker, running on AWS instance "ml.p2.xlarge".
• GluonTS version: 0.3.3 installed using pip.
• Kernel: conda_mxnet_p36

[lostella]

After installing GluonTS, could you try running in a cell

!pip show mxnet

and report the result?

[tm1611]

!pip show mxnet results in:

Name: mxnet
Version: 1.4.1
Summary: MXNet is an ultra-scalable deep learning framework. This version uses openblas.
Home-page: https://github.com/apache/incubator-mxnet
Author: UNKNOWN
Author-email: UNKNOWN
License: Apache 2.0
Location: /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages
Requires: numpy, graphviz, requests
Required-by: gluonts
You are using pip version 10.0.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

!pip install mxnet --upgrade led to conflicts with gluonts and numpy and the same error message with GPU is not enabled.

[lostella]

@tm1611 hopefully this is solved with the upcoming 0.4 release (to be relased very soon), since #428 was merged. I think the problem is that with gluonts<0.4 the vanilla mxnet package gets installed and replaces the one with built-in cuda for GPU processing.

[lostella]

@tm1611 can you check if the problem is gone now when you install GluonTS from scratch on your instance?

[tm1611]

Yes, the problem with mxnet and dependencies was removed. Thanks a lot for the quick fix.

Unrelated to this issue, but related to m4_benchmark.py and the sake of completeness: Using gluonts-version 0.4., one has to change num_eval_samples to num_sampels (see #421 ).