Sagemaker-python-sdk: [bug] mxnet estimator requirements.txt stopped working

[yifeim]

Describe the bug
According to the documents,
https://github.com/aws/sagemaker-python-sdk/blob/master/doc/using_mxnet.rst#use-third-party-libraries
requirements.txt could be loaded in mxnet estimators. This was the behavior when framework_version='1.4.1', but will trigger errors in framework_version='1.6.0'.

Screenshots or logs
Error because the package is not installed, despite being specified in requirements.txt. No errors are observed with framework_version='1.4.1'.

mxnet 1.6.0
Traceback (most recent call last):
  File "sagemaker_wrapper.py", line 18, in <module>
    from sm_train import train, save
  File "/opt/ml/code/sm_train.py", line 33, in <module>
    from data_io import walk_s3_or_local, read_s3_or_local, ReadAllAndStream
  File "/opt/ml/code/data_io.py", line 19, in <module>
    from dependencies import lru_cache, AttrDict, boto3
  File "/opt/ml/code/dependencies.py", line 31, in <module>
    from attrdict import AttrDict
ModuleNotFoundError: No module named 'attrdict'

System information
A description of your system. Please provide:

• SageMaker Python SDK version: latest as of 2/29/2020
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): mxnet
• Framework version: 1.6.0
• Python version: py3
• CPU or GPU: local_gpu and ml.p3.xlarge
• Custom Docker image (Y/N): N

Additional context
I will be using framework_version='1.4.1' for the time being. A related issue is that framework_version='1.3.1' could not be loaded. Mxnet-1.3 was announced on 11/2018. I think there should be sufficiently many people wanting to use this version to reproduce old experiments. Why did we stop supporting already?

https://aws.amazon.com/about-aws/whats-new/2018/11/amazon-sagemaker-now-supports-apache-mxnet-1-3-and-tensorflow-1-/

[otter-bunny]

Hi @yifeim, thanks for using SageMaker. There was a bug in our repack_model code that if your requirements.txt is tarred into your model data in training step and you changed the entry_point in inference stage, the requirements.txt will get removed. This issue was fixed in this change: https://github.com/aws/sagemaker-python-sdk/pull/1352

You can install the latest version of SageMaker Python SDK and try again. At the same time, you could also try putting your new entry_point script and requirements.txt under source_dir and provide its path.

[laurenyu]

1352 was released in v1.51.3

[austinmw]

Hi @laurenyu , my requirements.txt is also not working now:

code/
  main.py
  requirements.txt

m = MXNet(entry_point="main.py",
          source_dir="code",
          role=role,
          train_instance_count=1,
          train_instance_type="local_gpu",
          framework_version="1.6.0",
          py_version="py3",
          hyperparameters={'batch-size': 64,
                           'epochs': 200,
                           'lr': 0.001})

Not sure if I'm following the recommended solution here. I'm on sagemaker=='1.51.3'.

[laurenyu]

@austinmw we recently released a bug in the images themselves - I've opened a PR to fix this: https://github.com/aws/sagemaker-containers/pull/261

[harusametime]

When can we use the latest image including the bug fix? I am still seeing the error; framework_version='1.4.1' works, but framework_version='1.6.0' does not work.

[laurenyu]

we're are still waiting on the fix to be released. should be around the same time as https://github.com/aws/sagemaker-python-sdk/issues/1371. Apologies for the delay.

in the meantime, as a workaround, you can use subprocess to make a pip call in your training script.

[laurenyu]

The fixed images have been released (and verified while writing https://github.com/aws/sagemaker-mxnet-container/pull/165). Thank you for your patience!