Sagemaker-python-sdk: Model deploy instance_type modification failed

[Wei-1]

Reference:

• 0409412934
• MLFW-1638

System Information

• Framework / Algorithm: PCA
• Framework Version: SageMaker Aug 12, 2019 09:03 UTC
• Python Version: 3
• CPU or GPU: CPU
• Python SDK Version:
• Are you using a custom image: No

Describe the problem

Running SageMaker example: PCA for MNIST
If a user try to deploy a model with an instance_type that is not available,
the user won't be able to simply replace the instance_type and deploy again.

Minimal repro / logs

While running PCA for MNIST in the example project, and executing the following script:

pca_predictor = pca.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge') 

When a user doesn't have any extra resource to launch new instance with the assigned instance_type. The user will get a ResourceLimitExceeded error.
If the user desides to change the instance_type and execute the script again.

pca_predictor = pca.deploy(initial_instance_count=1, instance_type='ml.t2.medium') 

The the user will get this error:

--------------------------------------------------------------------------- 
ResourceLimitExceeded Traceback (most recent call last) 
<ipython-input-18-b2bb257b120c> in <module>() 
1 pca_predictor = pca.deploy(initial_instance_count=1, 
----> 2 instance_type='ml.t2.medium') 
...... 
ResourceLimitExceeded: An error occurred (ResourceLimitExceeded) when calling the CreateEndpoint operation: The account-level service limit 'ml.m4.xlarge for endpoint usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances... 
--------------------------------------------------------------------------- 

From this error message,
we can see that although we are trying to set a new instance_type,
the instance_type is not really reset.

[ChoiByungWook]

Hello @Wei-1,

Thank you for bringing this to our attention.

Let me look into this and figure out why the instance type isn't being propagated correctly.

Thank you for your patience.

[ChoiByungWook]

Can you paste how you instantiate the PCA algorithm?

Do you specify a name in the constructor, as I am suspecting this line causes an existing endpoint configuration name to be used: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/session.py#L1306

That line is fed when attempting to deploy your model: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/model.py#L427

[Wei-1]

I follow the lab tutorial: link
I don't think the name is stated.
I think it makes sense that the instance_type is passed through the model but the instance_type in the session is not replaced. Cus the error log states this behavior.

pca = sagemaker.estimator.Estimator(container,
                                    role, 
                                    train_instance_count=1, 
                                    train_instance_type='ml.c4.xlarge',
                                    output_path=output_location,
                                    sagemaker_session=sess)
pca.set_hyperparameters(feature_dim=50000,
                        num_components=10,
                        subtract_mean=True,
                        algorithm_mode='randomized',
                        mini_batch_size=200)
pca.fit({'train': s3_train_data})

pca_predictor = pca.deploy(initial_instance_count=1,
                           instance_type='ml.m4.xlarge')

[ChoiByungWook]

@Wei-1,

Thank you for the clarification.

Can you do me a quick favor and check your endpoint configurations in the AWS console?

AWS Console -> Amazon SageMaker -> Endpoint configurations

Or you can use the cli: https://docs.aws.amazon.com/cli/latest/reference/sagemaker/list-endpoint-configs.html

Can you tell me if you see any corresponding endpoint configurations that have the expected "ml.t2.medium", because based on the code links provided the session object should be propagating correctly.

Thanks!

[Wei-1]

I had rerun the whole estimator initiation process so I can see both t2.medium and m4.xlarge.
Let me find a time to remove both endpoint-config and run the whole notebook again to check.

[Wei-1]

Can you tell me if you see any corresponding endpoint configurations that have the expected "ml.t2.medium", because based on the code links provided the session object should be propagating correctly.

While running the code the first time with ml.m4.xlarge, we will be able to see the corresponding endpoint config.
However, after we get the error message and rerun the code with ml.t2.medium, a new endpoint config will NOT be generated.

[ChoiByungWook]

Gotcha, so from a notebook context this is still failing.

Hmmm... I wonder if there is something going on with the Python cache in the notebook environment. Thank you so much for all of this information.

I think this will require a bit of dedicated investigation. I'll bring this up with the team.

Thanks!

[laurenyu]

hi @Wei-1, can you try adding update_endpoint=True to the deploy call?

reference: https://sagemaker.readthedocs.io/en/stable/pca.html?highlight=update_endpoint#sagemaker.PCA.deploy

[Wei-1]

@laurenyu, Now it is even more interesting...
If I modify the deploy script to this:

pca_predictor = pca.deploy(initial_instance_count=1,
                           instance_type='ml.t2.medium',
                           update_endpoint=True)

I will get this error message:

ClientError: An error occurred (ValidationException) when calling the CreateEndpointConfig operation: Cannot create already existing endpoint configuration "arn:aws:sagemaker:us-east-2:XXXXXXXXXXXX:endpoint-config/pca-2019-09-04-01-01-54-841".

It will show that the endpointConfig already exist,
while the existing one is still in the old instance_type.

It will NOT create a new endpointConfig with ml.t2.medium.

[laurenyu]

@Wei-1 hmm, can you try specifying a new endpoint name?

[Wei-1]

pca_predictor = pca.deploy(initial_instance_count=1,
                           instance_type='ml.t2.medium',
                           endpoint_name='NewEndpointName')

@laurenyu, with a new endpoint name, the endpointCondfig with t2.medium can successfully be created and there will have no other errors for the entire process.

[mvsusp]

Hi @Wei-1,

Thanks for reporting this bug. I am adding to our roadmap changes in the behaviour of pca.deploy fixing your use case. We will update this ticket as soon as the changes are pushed.

Thanks for using SageMaker!

[Wei-1]

Cool @mvsusp!
the problem seems to be in the model and session, so it should be a general model deploy problem, not just PCA.
I can also draft a PR if you need any help.

[mvsusp]

You are right. I believe that the issue is that session.create_model tries to create a new model with the same previous name and fail - see https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/session.py#L774

One possible solution to fix this issue is to always generate models with a new name, perhaps in the format of name + '_' + timestamp.

Any contribution will be highly appreciated. Thanks @Wei-1

[Wei-1]

I will check if I am able to solve the issue this weekend.
Will let you know if I have any result.

[icywang86rui]

@Wei-1 the update_endpoint arg is designed to handle this case. Are you still seeing error or unexpected behavior when update_endpoint is set to True and and a new endpoint name is given in the second call?

[Wei-1]

@icywang86rui, if we use a new endpoint name, the module will behave as designed.
However, update_endpoint didn't act as expected as my investigation here: https://github.com/aws/sagemaker-python-sdk/issues/987#issuecomment-527698917

[mvsusp]

Thanks for start working on this PR @Wei-1 . Let us know if you have any doubts.

[Wei-1]

I changed some naming of the model -> endpoint config -> endpoint
instead of following the name of model, I change endpoint config to follow the name of endpoint since instance_type is not a part of model.
Please help to check if the modification in #1058 make sense.

[mvsusp]

I will answer your observations in the PR. Thanks.

[Wei-1]

@mvsusp, I just notice that the PR is reverted because of getting an error in the canaries in #1070.
The test seem to be failing when using hardcoded endpoint name,
but those tests section doesn't seem to be included in this repository.
You got an idea how we can proceed with this?

[ChoiByungWook]

Hello @Wei-1,

@mvsusp just recently left Amazon.

I'll take over this issue now. I'll begin by looking at #1070 .

Reference:

• 0409412934
• MLFW-1638

[Wei-1]

Hello @ChoiByungWook, any follow up on this?
Or is there any help that I can offer?

[AkashShukla199]

getting an error while I was deploying the trained model. It is just failing in between.

[laurenyu]

I've merged changes that were released as part of v2.0.0.rc1 to address this issue.

[Wei-1]

Nice nice, should I close this issue?

[laurenyu]

Yeah, we can close this issue. Feel free to reopen/create a new issue if further issues arise :)