Sagemaker-python-sdk: Add support for specifying env/requirements for sagemaker.mxnet.MXNet

[YashalShakti]

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): MXNet
• Framework Version: Latest
• Python Version: 3
• CPU or GPU: CPU and GPU
• Python SDK Version: Latest
• Are you using a custom image: No

Feature Request

It will be great to either specify the env dictionary or path to requirements file for the Sagemaker MxNet API. https://github.com/aws/sagemaker-python-sdk/blob/8b3d113a23c09995c6a6a5d12d4364e27bfd549d/src/sagemaker/mxnet/estimator.py#L26

It is already available in the Tensorflow API #52

We need to add additional dependencies and having the ability to specify requirements is much much easier than creating our own Docker image.

[mvsusp]

Hi @YashalShakti,

This feature is already in the backlog and will be available later this year. In the meanwhile you can install the modules yourself (example extracted from https://github.com/aws/sagemaker-python-sdk/issues/289):

import subprocess
import sys

subprocess.call([sys.executable, '-m', 'pip', 'install', '-U', 'scikit-image') 

Thanks for using SageMaker!

Best,

Márcio

[yifeim]

Hope people still have this in their plans?

What I really appreciate about sagemaker is its clean APIs that contribute to good coding habits, as well as the rich set of examples. I wish to see things in their proper places. Having pip install inside the codes is a bit of a stretch.

Would it just be a one-line code change in the backend?

Thanks.

[yifeim]

Btw, one place where this idea fails in during inference with multiple processes. In that case, all parallel processes try to install at the same time.

[laurenyu]

Sorry not to update here with the completion of #446. As of now, you can now include a requirements.txt file in the same directory as the entry script, and it will be installed - but only if using our MXNet 1.3 images.

[kirkanderson]

Sorry not to update here with the completion of #446. As of now, you can now include a requirements.txt file in the same directory as the entry script, and it will be installed - but only if using our MXNet 1.3 images.

Can you confirm this has been implemented? This functionality doesn't appear to be present in the code and doesn't work for me.

[YashalShakti]

Sorry not to update here with the completion of #446. As of now, you can now include a requirements.txt file in the same directory as the entry script, and it will be installed - but only if using our MXNet 1.3 images.

Can you confirm this has been implemented? This functionality doesn't appear to be present in the code and doesn't work for me.

@kirkanderson It does work with MXNet 1.3 images (framework_version='1.3.0'). Be sure to specify the "source_dir" parameter. It should point to the source dir that contains the 'requirements.txt' file. I have verified that it works.

[yifeim]

Yes. This has been a very useful feature. Another notable convenience feature is hyperparameter in environmental variables. Thanks to everyone!