Azure-docs: Deploy model, image and service inside pipeline

[YutongTie-MSFT]

This issue is moved from MSDN forum. https://social.msdn.microsoft.com/Forums/en-US/af1c2b3a-4d0e-4af7-8902-f827b7207148/deploy-model-image-and-service-inside-pipeline?forum=AzureMachineLearningService

According to the pipeline documentation, it seems that it should be possible to deploy models, images and services inside the pipeline infrastructure. However, the guides and example notebooks doesn't (at the moment) touch this. I believe I would be able to replicate my current deployment setup I have in my notebooks, but I am worried about how to connect to the workspace in a secure manner. Is this possible, or do I have to use my old notebooks for this purpose?

I am trying to automate the entire process from data preparation, through training and all the way to deployment of the model and updating my service, all in one simple REST call. If I have to manually deploy and update my service, I miss a lot of the automation I am trying to achieve.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: c8fc8bdb-1ee0-b978-e42f-de875b8cb50f
• Version Independent ID: dbad235e-f8d6-361b-d337-f9d2a3573059
• Content: Build machine learning pipelines - Azure Machine Learning service
• Content Source: articles/machine-learning/service/concept-ml-pipelines.md
• Service: machine-learning
• GitHub Login: @sanpil
• Microsoft Alias: sanpil

[YutongTie-MSFT]

@sanpil Hi, can you please take a look of this issue and update the document as necessary? Thanks a lot!

[j-martens]

@jpe316

[ArvidBaa]

I am the author of the original post in the MSDN forum, and I just wanted to make myself known. Also, I write this post so I will get notifications on further development.

Has there been any development regarding this? I would be happy to be able to automate the deployment process using the pipelines!

[jpe316]

Hi @ArvidBaa - yes, you can register models and perform other model management commands using an Azure ML Pipeline!

In order to do this, you simply need to get access to the workspace context object (so you have permissions to perform these activities).

You can do this from run.register_model or Model.register

Option A:

run = Run.get_context()
run.register_model(model_name='{model_name}')

Option B:

 Model.register(Run.get_context().experiment.workspace, model_path, model_name, tags=None, properties=None, description=None)

[ArvidBaa]

Hi @jpe316 - Thank you so much! Run.get_context().experiment.workspace was just what I needed!

[ArvidBaa]

@jpe316 - After some more testing, I realize that it doesn't take me the entire way. First of all, deployment of models and images works perfect. The workspace object was enough to make it work. Thank you again for the help there!

But, when deploying services, it doesn't work as smoothly as I had hoped. I am able to create a 'default' AciWebservice by calling

aci_config = AciWebservice.deploy_configuration()
service = Webservice.deploy_from_image(workspace=ws, name=service_name, deployment_config=aci_config, image=image)

and I am also able to retrieve an already existing AciWebservice by calling

service = AciWebservice(ws, service_name)

So far so good. But when trying the same process with an AksWebservice like

prov_config = AksCompute.provisioning_configuration()
aks_target = ComputeTarget.create(workspace=ws, name=aks_name, provisioning_configuration=prov_config)
aks_config = AksWebservice.deploy_configuration()
service = Webservice.deploy_from_image(workspace=ws, name=service_name, image=image, deployment_config=aks_config, deployment_target=aks_target)

I only get authorization error for invalid token.

How do I solve this? I have no problem creating and fetching an AksCompute or an AksWebservice when working locally, either in scripts or in notebooks. But inside the pipeline it doesn't work.

[salilbapat]

Given that your ACI deployments are working from within the pipeline, AKS should also have worked. Can you share some more details so we can look at the exact failure? Following will be helpful:
1) Name of the AKS compute that you created
2) Name of the AKS webservice whose creation failed
3) Name of the ACI webservice which was successful
4) Subscription id for all these operations
5) The exact error message you are seeing

[ArvidBaa]

@salilbapat - Of course. But should I post it here, or in some more private discussion? I'm not sure if there are parts in the error message that I should keep secret.

[salilbapat]

Ideally the error message should not have any secrets and any of the other info I asked is not secret either. But you have any concerns, then feel free to email me the details I asked for. salil.[email protected]

[ArvidBaa]

Ok, here goes.

For all of the following examples I get the ws object using the code given earlier:

from azureml.core.run import Run
ws = Run.get_context().experiment.workspace

1. I have an existing AksCompute with name AKS-compute. I have created it using Jupyter notebook and I am able to both fetch and use it in the notebook. When trying to fetch it inside the pipeline with the following script:

from azureml.core.compute import AksCompute
aks_name = 'AKS-compute'
aks_target = AksCompute(ws, aks_name)

I get the following error:

Received bad response from Resource Provider:
Response Code: 401
Headers: {'Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Type': 'application/json; charset=utf-8', 'Expires': '-1', 'WWW-Authenticate': 'Bearer authorization_uri="https://login.windows.net/6bafac43-30e0-4371-849c-ff42ba3d17e6", error="invalid_token", error_description="Could not find identity for access token."', 'x-ms-failure-cause': 'gateway', 'x-ms-request-id': '6fa89474-a06a-4562-90f8-d779dd1bd59a', 'x-ms-correlation-request-id': '6fa89474-a06a-4562-90f8-d779dd1bd59a', 'x-ms-routing-request-id': 'WESTEUROPE:20190124T101419Z:6fa89474-a06a-4562-90f8-d779dd1bd59a', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 24 Jan 2019 10:14:19 GMT', 'Connection': 'close', 'Content-Length': '76'}
Content: b'{"error":{"code":"AuthenticationFailed","message":"Authentication failed."}}'

If I try to create a new AksCompute with name AKS-compute-new inside the pipeline script with the following script:

from azureml.core.compute import AksCompute, ComputeTarget
aks_name = 'AKS-compute-new'
prov_config = AksCompute.provisioning_configuration()
aks_target = ComputeTarget.create(ws, aks_name, prov_config)
aks_target.wait_for_completion(show_output=True)

I get the following error:

Received bad response from Resource Provider:
Response Code: 401
Headers: {'Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Type': 'application/json; charset=utf-8', 'Expires': '-1', 'WWW-Authenticate': 'Bearer authorization_uri="https://login.windows.net/6bafac43-30e0-4371-849c-ff42ba3d17e6", error="invalid_token", error_description="Could not find identity for access token."', 'x-ms-failure-cause': 'gateway', 'x-ms-request-id': '0f296e0a-10e0-4c23-a7d1-9486c0d97a1b', 'x-ms-correlation-request-id': '0f296e0a-10e0-4c23-a7d1-9486c0d97a1b', 'x-ms-routing-request-id': 'WESTEUROPE:20190124T101419Z:0f296e0a-10e0-4c23-a7d1-9486c0d97a1b', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 24 Jan 2019 10:14:18 GMT', 'Connection': 'close', 'Content-Length': '76'}
Content: b'{"error":{"code":"AuthenticationFailed","message":"Authentication failed."}}'

1. I have created an AksWebservice with name aks-service using Jupyter notebook, and I am able to both fetch it and use it (e.g. update it with a new image) in the notebook. However, since I can't either fetch or create an AksCompute inside the pipeline, I can't test to create the AksWebservice inside the pipeline. However, when trying to fetch the existing AksWebservice inside the pipeline using the following script:

from azureml.core.webservice import AksWebservice
service_name = 'aks-service'
service = AksWebservice(ws, service_name)

I get the following error:

Received bad response from Model Management Service:
Response Code: 401
Headers: {'Date': 'Thu, 24 Jan 2019 10:14:19 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'api-supported-versions': '2018-03-01-preview, 2018-11-19', 'x-ms-client-request-id': 'a655002c5a84477dae038735f4a47f6f', 'x-ms-client-session-id': '', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'}
Content: b'{"code":"Unauthorized","statusCode":401,"message":"Unauthorized","details":[{"code":"InvalidOrExpiredToken","message":"The request token was either invalid or expired. Please try again with a valid token."}]}'

1. I have been able to fetch an existing AciWebservice with name aci-service inside the pipeline. I use the following script for this process:

from azureml.core.webservice import AciWebservice
service_name = 'aci-service'
service = AciWebservice(ws, service_name)

I have also been able to create a new AciWebservice with name aci-service-new inside the pipeline using the following script:

from azureml.core.webservice import Webservice, AciWebservice
service_name = 'aci-service-new'
aci_config = AciWebservice.deploy_configuration()
service = Webservice.deploy_from_image(ws, service_name, image, aci_config)

1. My subscription id is 291faa0e-d03f-41ee-883e-44000cf2eac0

2. The error messages connected to each issue is given above

[salilbapat]

Thanks for the information. For some reasons I thought that you are able to create AKS compute but not deploy services to it via pipelines. But looks like you are not able to do any AKS related operations in the pipeline. I checked internally and this is an expected behavior right now. Using the compute creation/attach classes and accessing AKS services need a slightly different auth mechanism which is currently not implemented in pipelines. We will look to add native support in pipelines for this functionality. If I am able to find a timeframe around when this will be available, then I will post on the thread.

[ArvidBaa]

Ah, I can see why you thought that. I wasn't very clear on the fact from the beginning. But with your help we got to the root of the issue. I would be very grateful if you could provide a timeframe around when this will be available, it you are able to find it. Until then we will have to stick with manual deployment and updating of the services.

Thank you so much for your help! I know that the AML services are a very new service and that it will take some time to implement all wanted features, and I am both grateful and impressed that you dedicate so much time to discussing features and issues with the users.

[hning86]

it should be possible to create a service principle and use that to authenticate and access workspace from your pipeline code. the downside is that you'd bury the pwd in the code which is not secure.

[ArvidBaa]

@hning86 - I am able to access the workspace using Run.get_context().experiment.workspace, and from that I am able to access both models, images and (at this point only Acl) services. By your suggestion, do you mean that I might get more access, in particular to the AksWebservice objects?

[ArvidBaa]

@salilbapat Has there been any progress to the issue or at least on the timeframe?

[salilbapat]

Hey, sorry about delay in reply. We have this item on our backlog currently but it is not been scheduled. Given other priorities, this will not be taken up in next couple of months atleast.

[ArvidBaa]

@salilbapat Ok, then I will have to make do without that feature until then, but I really hope you take it up as soon as possible. You will have at least one happy user if you do!

Thanks for your help!

[j-martens]

please-close

[antti-saarela]

Any updates on this issue? IMO the issue being a clear stopper for any further and serious MLOps automation work shouldn't have been closed, but put on the backlog. I still haven't found any decent workaround for the issue. Any hints and comments are highly appreciated.
Thanks in advance.

[jonerei]

Still running into the same issue when trying to deploy a registered model from within a pipeline step. However the error is not limited to Aks computes. It shows up when obtaining a compute target within a pipeline step like so:
compute_target = ComputeTarget(ws, name=compute_target_name)
where

>>> compute_target.type
'ComputeInstance'

This is the error:

UserScriptException: UserScriptException:
    Message: Received bad response from Resource Provider:
Response Code: 401
Headers: {'Cache-Control': 'no-cache', 'Pragma': 'no-cache', 'Content-Type': 'application/json; charset=utf-8', 'Expires': '-1', 'WWW-Authenticate': 'Bearer authorization_uri="https://login.windows.net/78d2f605-610c-4d62-b4bc-e4eb2e3e4b0b", error="invalid_token", error_description="Could not find identity for access token."', 'x-ms-failure-cause': 'gateway', 'x-ms-request-id': 'e691b586-2642-4557-89d1-e6362391fcf7', 'x-ms-correlation-request-id': 'e691b586-2642-4557-89d1-e6362391fcf7', 'x-ms-routing-request-id': 'WESTEUROPE:20200622T135117Z:e691b586-2642-4557-89d1-e6362391fcf7', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 Jun 2020 13:51:17 GMT', 'Connection': 'close', 'Content-Length': '76'}
Content: b'{"error":{"code":"AuthenticationFailed","message":"Authentication failed."}}'

I am using the current azureml-sdk version : azureml-sdk==1.7.0.post1
Are there any workarounds for deploying a model straight from the AML pipeline?

[stawo]

@j-martens @hning86 , could you provide an update on this issue?
It's marked as closed, but people (including me) are still affected by it.