Sagemaker-python-sdk: The trained model can be deployed on the other platform without dependency of sagemaker or aws service?

[giggle0312]

The output of the training model operation is model artifacts stored on S3 bucket, I did not find the path of docker image. Therefore, I want to ask a question that if the trained model can be deployed on the other platform without sagemaker or aws service,for example I download the artifacts and docker image to deploy on local environment?

[juliensimon]

Yes, you can download the model from S3 and use it anywhere else.

For Docker images, only the Tensorflow and MXNet containers have been open sourced (you'll find them on Github) but you don't need them to use a model trained on SageMaker. You can use vanilla Tensorflow and MXNet.

[giggle0312]

@juliensimon Thank you very much. I find the model artifacts on the S3 bucket which can be download. But I unzip the model.tar.gz , I don't know how to use it for predict. Can you help give me a explanation about how to deply a trained model on local environment and use it to make some predicts with the local data?

[winstonaws]

Hi @giggle0312 -

1. If you want to make a prediction locally using the docker image, the easiest way to do so is to use the SageMaker Python SDK's local mode - https://github.com/aws/sagemaker-python-sdk#sagemaker-python-sdk-overview (see the section labeled "Local Mode")

What this will do for you is download the appropriate image, set up the environment, and run the image as if it was on sagemaker.

1. As @juliensimon said, you can also take the model artifacts and use them natively in the framework that trained them. The specific code needed to do this would depend on whether you are using MXNet or TensorFlow, and what you did to train the models. It'd be possible to figure this out just from reading MXNet/TensorFlow documentation, but wouldn't be super simple, so hopefully option 1 works for you.

[giggle0312]

@winstonaws Thank you very much. I have a question that what is the difference and relation between model artifacts stored on s3 and docer image stored on EC2?

[winstonaws]

@giggle0312

The recommended way to use SageMaker hosting is to build a docker image with the code and dependencies you need to make predictions using your model, but have the actual model data itself (the weights) stored in S3.

When you create an SageMaker model to host in an endpoint, you specify both the image and the S3 location of the model data: https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateModel.html

When the endpoint is created, SageMaker will download the model data from S3 and make it available in the docker image in the local /opt/ml/model directory. The code in the docker image should know how to load the model data and make predictions.

[jesterhazy]

This issue has been closed due to inactivity. If you still have questions, please re-open it.

[kshitij-kant]

Hi. As answered above , we can use trained 3rd party models (MXnet/tensorflow) natively. But can we also do the same for the 13 core sagemaker models like deepAR or linear-learner etc.
thanks

[nickjtch]

Hello. As @kshitij-kant pointed, is hosting Sagemaker models like DeepAR possible on premises using any open source software?

[mfkenson]

I have the same question actually after attending AWS summit. @nicolasmichaeleljamal
@juliensimon Do you have any idea?

[NotSoShaby]

@winstonaws can you please elaborate on how to load tar.gz file that was created by sagemaker to MXNet/TensorFlow? I understand it changes, but even a simple example will be very helpful

[Serhiy-Shekhovtsov]

Also wondering about extracting core SageMaker models(like DeepAR). @jesterhazy how can someone reopen this issue?

[laurenyu]

@NotSoShaby here are some examples for MXNet/TFS:

• MXNet: https://github.com/aws/sagemaker-mxnet-serving-container/blob/master/src/sagemaker_mxnet_serving_container/default_inference_handler.py#L36-L68
• TFS: https://github.com/aws/sagemaker-tensorflow-serving-container/blob/master/container/sagemaker/proxy_client.py#L30-L62

to those wondering about DeepAR, can you open a new issue? That'll help us track it and get it to the right team to answer your question.

[wangzhenj321]

@laurenyu

I have trained a model artifact (model.tar.gz stored on S3) with the built-in algorithm Semantic Segmentation on AWS SageMaker. I want to download this model and make inference with mxnet and gluon-cv on my local PC. However, after downloading and unzipping this model file, there are only three files as follows, but there is no json of reconstructing the network model.

• hyperparams.json
• model_algo-1
• model_best.params

And I can't see the detail of the image of the Semantic Segmentation algorithm about how the network model is constructed.

How can I make inference with trained model of Semantic Segmentation on my local PC?

I have attempted to use the following code to make inference, but failed...

import mxnet as mx
import gluoncv
from mxnet import image
from gluoncv.data.transforms.presets.segmentation import test_transform

model = gluoncv.model_zoo.PSPNet(2, crop_size=640)
model.load_parameters('./model/model_algo-1') # extract from model.tar.gz
img = image.imread('./img/IMG_4015.jpg') # the test image
ctx = mx.cpu(0)
img = test_transform(img, ctx)
img = img.astype('float32')
output = model(img)

Error message from the above code:

terminate called after throwing an instance of 'dmlc::Error'
  what():  [16:00:55] src/operator/nn/concat.cc:67: Check failed: shape_assign(&(*in_shape)[i], dshape): Incompatible input shape: expected [1,-1,75,113], got [1,512,80,80]
Stack trace:
  [bt] (0) /home/wang/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x307d3b) [0x7fc32af58d3b]
  [bt] (1) /home/wang/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x8e67fb) [0x7fc32b5377fb]
  [bt] (2) /home/wang/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, std::vector<mxnet::NDArray*, std::allocator<mxnet::NDArray*> > const&, mxnet::DispatchMode*)+0x1d27) [0x7fc32e0f2aa7]

[laurenyu]

@wangzhenj321 Apologies as I'm not super familiar with MXNet, but I'm guessing you might need to provide more information when loading the model. Can you create a new GitHub issue? It'll help us track your question and pass it along to the team that owns Semantic Segmentation.

[wangzhenj321]

@laurenyu

Sorry for forgetting to update my status about the solution. After struggling for three days, I found the way to make inference on my local PC with the trained model from the built-in algorithm Semantic Segmentation.

import mxnet as mx
from mxnet import image
from gluoncv.data.transforms.presets.segmentation import test_transform
import gluoncv

# use cpu
ctx = mx.cpu(0)

# load test image
img = image.imread('./img/IMG_4015.jpg')
img = test_transform(img, ctx)
img = img.astype('float32')

# reconstruct the PSP network model
model = gluoncv.model_zoo.PSPNet(2)

# load the trained model
model.load_parameters('./model/model_algo-1')

# make inference
output = model.predict(img)
predict = mx.nd.squeeze(mx.nd.argmax(output, 1)).asnumpy()