Sagemaker-python-sdk: scikit learn: wrong image used when attaching and creating a transformer

[saadtazi]

Describe the bug

When I attach a model that was trained with framework version 0.23-1 and create a transformer, the transformer uses framework version 0.20.0.

To reproduce
Assuming you have a model named MODEL_NAME = "my-model-2020-08-12-20-46-51-723" that was trained with scikit learn image v.0.23-1:

sklearn_estimator = SKLearn(
    entry_point='svm_script.py',
    source_dir='src',
    role = get_execution_role(),
    train_instance_count=1,
    # does not work with 'local' (could have been faster) because of memory error
    train_instance_type='ml.m5.2xlarge',
    framework_version='0.20.0',
    # not able to start a scikit learn transform with 0.23-1
    # framework_version='0.23-1',
    base_job_name='saad-ecom-binary-svm',
)

Note that the training process is not documented, just the estimator instance creation used before calling fit()...

Then if I want to attach that model to create a transformer, using the following lines:

sklearn = SKLearn.attach(MODEL_NAME)
transformer = sklearn.transformer(instance_count=1, instance_type='ml.m5.12xlarge', accept="text/csv", assemble_with="Line")

Then a new model is created then attached to the transformer, that model uses image 0.20.0 (683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3) instead of 0.23-1.

Expected behavior

I was expecting the transformer to use a model that uses 0.23-1.

Note that the proper image is listed when I run the following using the model name:

job_details = sagemaker_session.sagemaker_client.describe_training_job(
    TrainingJobName=MODEL_NAME
)

Screenshots or logs
If applicable, add screenshots or logs to help explain your problem.

System information
A description of your system. Please provide:

• SageMaker Python SDK version: sagemaker==1.71.0 (from sagemaker notebook)
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): scikit learn
• Framework version: 0.23-1
• Python version: 3
• CPU or GPU: CPU only
• Custom Docker image (Y/N): no (custom entry_point)

Additional context

I have 2 dependencies in requirements.txt:

nltk==3.4.5
joblib==0.15.1

[metrizable]

@saadtazi Thanks for filing the issue. I have a clarification question. You mention above that your MODEL_NAME model was trained with scikit-learn image v.0.23-1:

Assuming you have a model named MODEL_NAME = "my-model-2020-08-12-20-46-51-723" that was trained with scikit learn image v.0.23-1:

sklearn_estimator = SKLearn(
    entry_point='svm_script.py',
    source_dir='src',
    role = get_execution_role(),
    train_instance_count=1,
    # does not work with 'local' (could have been faster) because of memory error
    train_instance_type='ml.m5.2xlarge',
    framework_version='0.20.0',
    # not able to start a scikit learn transform with 0.23-1
    # framework_version='0.23-1',
    base_job_name='saad-ecom-binary-svm',
)

From your included code, it looks like you're selecting framework version 0.20.0. Is this right?

[saadtazi]

Oops, I copied the code I am currently using to by-pass the limitation, not the code that was causing the issue. This code ahould read:

sklearn_estimator = SKLearn(
    entry_point='svm_script.py',
    source_dir='src',
    role = get_execution_role(),
    train_instance_count=1,
    train_instance_type='ml.m5.2xlarge',
    framework_version='0.23-1',
    base_job_name='saad-ecom-binary-svm',
)

Sorry for the confusion.

[saadtazi]

I tried a couple of things, here are some additional info, that narrows down the problem I think:

1. I train a model using the SKLearn estimator with framework_version='0.23-1' (TRAINING_JOB_NAME). I can confirm in the AWS console that the image asssociated with that trainging job uses the right docker image (0.23-1).
2. Right after, if I do the following:

sklearn = SKLearn.attach(TRAINING_JOB_NAME)
print(sklearn.image_name)

this outputs 683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.20.0-cpu-py3
but I was expecting 683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3

I was able to prevent the issue by setting the image_name right after attach(). The transformer gets the right image:

sklearn = SKLearn.attach(TRAINING_JOB_NAME)
sklearn.image_name = "683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3"

Note: I am new to sagemaker, so I think I used the wrong term in my initial description: MODEl_NAME should be TRAINING_JOB_NAME...

[metrizable]

@saadtazi Thanks for the clarification.

I was initially unable to reproduce the issue, until I realized you were running in a notebook under the older sagemaker==1.71.0.

This bug has been fixed in later versions of sagemaker. As you can see in the older version here, the image tag is not picked up, nor used, to determine the framework_version for the init_params. However, from the newer version here, the image tag is picked up and used to establish the correct framework_version for the init_params.

Of particular note, since v2 is a semantic versioning major version upgrade, there are breaking changes that may affect your code. You can address any changes individually, or you can apply the automated upgrade tool to your ipynb file to make your 1.x notebook code 2.x compatible.

To close, if you are running in a SageMaker Notebook instance, and you decide to upgrade the notebooks in your notebook instance to be 2.x compliant, be sure to add a cell to the top of them:

pip install -U sagemaker

and restart your kernel.

[saadtazi]

Thank you @metrizable for your detailed and quick answer. Sorry for the duplicate then. I searched the github issue before adding my issue, I swear :-). I probably didn't seach enough...

I will stick with 1.71.0 for now, and do the upgrade later. In the meantime, I'm closing the issue.

Update

I couldn't resist: i updated to sagemaker 2.3.0 and it works!