Sagemaker-python-sdk: Parser of framework and version from image URI does not work with TF script mode images

[jonsnowseven]

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): TensorFlow/Neural Network
• Framework Version: 1.12
• Python Version: 3.6
• CPU or GPU: CPU
• Python SDK Version: 1.16.3 (and 1.18.3 also)
• Are you using a custom image: No

Describe the problem

https://github.com/aws/sagemaker-python-sdk/blob/2af36ade01fbd95cf1d1cc674a38662f71e41314/src/sagemaker/fw_utils.py#L223

This function does not parse container images in the form of "sagemaker-tensorflow-scriptmode".

• Exact command to reproduce:

from sagemaker.fw_utils import framework_name_from_image

framework_name_from_image("520713654638.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-tensorflow-scriptmode:1.12-cpu-py3")

(None, None, None)

[iquintero]

Hi @jonsnowseven

Thanks for reporting this issue. We add a fix for this to our backlog, other than calling it directly have you identified any other case within the SDK that is affected by this?

[jonsnowseven]

Hello @iquintero.

Thank you for answering.

Yes, this parse causes, for instance, TensorFlow.attach(trainingJob) to not work because when initializing the parameters, the image is parsed using that until and the object does not have the correct attributes comparing with the training job.

[nadiaya]

Adding bug label and leaving the issue open for further references.

To clarify - attach should work correctly, except it would change the logic to treat the image as 'custom' image and would cause issues for deployment (it would try to use the same script mode image instead of correct TFS image). More details in: https://github.com/aws/sagemaker-python-sdk/issues/663

[jonsnowseven]

Hello @nadiaya.

Thank you for your answer.

I'll be waiting for a solution then.

Best regards,

João Neves

[cbreathnach]

Hi, I've been working with a SageMaker Hyperparameter Tuning Job and I'm running into a problem that seems to be based on this issue as well. The training jobs run without issue, but when calling the .deploy() method, I get a similar error as mentioned in #663, which links to here, showing in the Endpoint logs:

FileNotFoundError: [Errno 2] No such file or directory: 'nginx': 'nginx'.

The deployment fails with the message:

The primary container for production variant default-variant-name did not pass the ping health check.

I am also using custom entrypoint scripts, so I am wondering can I attribute this to the same error as listed here, given that the .attach() method is called inside of the .deploy() method for the HyperparameterTuner? If so, is there any estimate on when a fix could be available for this or what would be the best way to get the deployment working here? Thanks.

[jonsnowseven]

Hello @cbreathnach.

I believe so. That error occurs because the container image string is wrongly parsed! After all, the Hyperparameter Tuning Job is a set of Training Jobs so it is only natural that you experience the same behaviour. But it is probably better if some developer from Sagemaker Python SDK team can confirm this.

[markproy]

Is there a workaround? I'm hitting this today as well.

[laurenyu]

apologies for the inconvenience. we do have an open item in our backlog for working on this issue.

In the meantime, there are a couple workarounds:

1. set the image name explicitly to 520713654638.dkr.ecr.<region>.amazonaws.com/sagemaker-tensorflow-serving:1.12.0-<cpu|gpu>, specifying your desired region and processor type (cpu or gpu)
2. use the TFS Model class: https://sagemaker.readthedocs.io/en/stable/sagemaker.tensorflow.html#tensorflow-serving-model

edit: it looks like this should have been fixed in https://github.com/aws/sagemaker-python-sdk/pull/692. To those seeing the error, are you on version 1.18.6 or higher?

[cbreathnach]

Increasing the version was the solution in my case for the deployment of an endpoint from a Hyperparameter Tuning job. I had been previously using version 1.16.1, but swapping to test in an environment with version 1.18.14.post0 has fixed the problem.

In the case it's relevant for any trouble-shooting, the error was only occurring for me while using script mode for training scripts, for example: https://sagemaker.readthedocs.io/en/stable/using_tf.html#preparing-a-script-mode-training-script

The deployment in my case worked as expected when using an older approach with the older sdk version, eg: https://github.com/aws/sagemaker-python-sdk/tree/v1.12.0/src/sagemaker/tensorflow#tensorflow-sagemaker-estimators-and-models

[aninoy]

I'm having similar problems still with sagemaker version 1.51.4.
My training image name is 763104351884.dkr.ecr.us-west-2.amazonaws.com/tensorflow-training:1.14.0-gpu-py3 and when doing an attach and deploy, I get the same the error:
FileNotFoundError: [Errno 2] No such file or directory: 'nginx': 'nginx'

Is there a reason why my training image name is "tensorflow-training..." and not "sagemaker-tensorflow-training..."? The regex in the fw_utils doesn't extract the image name correctly in this instance.

[laurenyu]

@aninoy the training image name comes from the AWS Deep Learning Containers: https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-images.html

could you open a new issue and include the code you're using to fit, attach, and deploy? (opening a new issue will help with our internal tracking)

[aninoy]

Yes, will do, thank you for your response.

[aninoy]

Linking the related issue here for reference.