Azure-sdk-for-python: Error retrieving the environment definition

[areed145]

• Package Name: azureml
• Operating System: linux
• Python Version: 3.7

Describe the bug
Traceback (most recent call last):
File "pipeline.py", line 169, in
automl_run = experiment.submit(automl_config, show_output=False)
File "/azureml-envs/azureml_53944c83b0f490385cabad6315352750/lib/python3.6/site-packages/azureml/core/experiment.py", line 219, in submit
run = submit_func(config, self.workspace, self.name, **kwargs)
File "/azureml-envs/azureml_53944c83b0f490385cabad6315352750/lib/python3.6/site-packages/azureml/train/automl/automlconfig.py", line 99, in _automl_static_submit
show_output)
File "/azureml-envs/azureml_53944c83b0f490385cabad6315352750/lib/python3.6/site-packages/azureml/train/automl/automlconfig.py", line 247, in _start_execution
validate_non_prod_env_exists(experiment.workspace)
File "/azureml-envs/azureml_53944c83b0f490385cabad6315352750/lib/python3.6/site-packages/azureml/train/automl/_environment_utilities.py", line 290, in validate_non_prod_env_exists
env = Environment.get(ws, NON_PROD_ENVIRONMENTS[env_name])
File "/azureml-envs/azureml_53944c83b0f490385cabad6315352750/lib/python3.6/site-packages/azureml/core/environment.py", line 842, in get
environment_dict = environment_client._get_environment_definition(name=name, version=version)
File "/azureml-envs/azureml_53944c83b0f490385cabad6315352750/lib/python3.6/site-packages/azureml/_restclient/environment_client.py", line 46, in _get_environment_definition
raise Exception(message)
Exception: Error retrieving the environment definition. Code: 404
: {
"error": {
"code": "UserError",
"severity": null,
"message": "No definitions exist for environment AutoML-Non-Prod",
"messageFormat": null,
"messageParameters": null,
"referenceCode": null,
"detailsUri": null,
"target": null,
"details": [],
"innerError": {
"code": "NotFoundError",
"innerError": null
},
"debugInfo": null
},
"correlation": {
"operation": "a16e1388a17b884a9b08563ba8b825fe",
"request": "c444e47189fca044"
},
"environment": "westus",
"location": "westus",
"time": "2020-10-01T00:01:01.9738765+00:00",
"componentName": "environment-management"
}

To Reproduce
Steps to reproduce the behavior:

1. Run AML pipeline with with aml_config cloned on curated environment

Expected behavior
Expected to find environment and run Pipeline Script Step

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

[areed145]

what team should this go to?

[rtanase]

Hi @areed145! Can you please try upgrading azureml-train-automl-client package to 1.15.0.post1? The error should no longer occur.

[areed145]

@rtanase This is executed on AML remote compute. Should I add this version to the compute target environment, the compute where the pipeline is created, or both?

[rtanase]

@areed145, the version of the client package needs to be installed on the machines where the exception is being raised from (where the AutoML experiment is being submitted from). I don't know exactly what steps your pipeline contains, but I assume you have Pipeline Script Step where an AutoML experiment is started from. If that is the case, the environment that this step is running from, needs to have the updated that package.

[areed145]

@rtanase This worked, thanks for the help. Where is the best place to monitor for changes like this in the future to the azureml sdk?

[rtanase]

@areed145, glad it worked out and thanks for reporting the issue! Unfortunately, this particular issue we only discovered after the release went out to PyPi, so we had to release a patch after.

[kimix92]

please-close

[wynnlone]

Package Name: azureml
Operating System: Windows10
Python Version: 3.7

Describe the bug

Exception Traceback (most recent call last)
in
----> 1 remote_run=experiment.submit(automl_config, show_output=False)
2 remote_run

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/experiment.py in submit(self, config, tags, *kwargs)
217 submit_func = get_experiment_submit(config)
218 with self._log_context("submit config {}".format(config.__class__.__name__)):
--> 219 run = submit_func(config, self.workspace, self.name, *kwargs)
220 if tags is not None:
221 run.set_tags(tags)

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/train/automl/automlconfig.py in _automl_static_submit(automl_config_object, workspace, experiment_name, **kwargs)
97 compute_target,
98 parent_run_id,
---> 99 show_output)
100
101 automl_run.add_properties(global_tracking_info_registry.gather_all(settings.path))

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/train/automl/automlconfig.py in _start_execution(experiment, settings_obj, fit_params, run_config, compute_target, parent_run_id, show_output)
245 logger.info("Submitting remote.")
246 print("Running on remote.")
--> 247 validate_non_prod_env_exists(experiment.workspace)
248 automl_run = _default_execution(experiment, settings_obj, fit_params, False, show_output)
249

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/train/automl/_environment_utilities.py in validate_non_prod_env_exists(ws)
288 for env_name in NON_PROD_ENVIRONMENTS:
289 try:
--> 290 env = Environment.get(ws, NON_PROD_ENVIRONMENTS[env_name])
291 if not any([azureml.train.automl.VERSION in x for x in env.python.conda_dependencies.pip_packages]):
292 logger.warning(NON_PROD_MISMATCH_WARNING.format(azureml.automl.core.VERSION, env))

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/core/environment.py in get(workspace, name, version)
840 """
841 environment_client = EnvironmentClient(workspace.service_context)
--> 842 environment_dict = environment_client._get_environment_definition(name=name, version=version)
843 env = Environment._deserialize_and_add_to_object(environment_dict)
844

/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/_restclient/environment_client.py in _get_environment_definition(self, name, version)
44 message = "Error retrieving the environment definition. Code: {}n: {}".format(response.status_code,
45 response.text)
---> 46 raise Exception(message)
47
48 def _register_environment_definition(self, environment_dict):

Exception: Error retrieving the environment definition. Code: 404
: {
"error": {
"code": "UserError",
"severity": null,
"message": "No definitions exist for environment AutoML-Non-Prod",
"messageFormat": null,
"messageParameters": null,
"referenceCode": null,
"detailsUri": null,
"target": null,
"details": [],
"innerError": {
"code": "NotFoundError",
"innerError": null
},
"debugInfo": null
},
"correlation": {
"operation": "b10d8accf61b12409ca2aaca2abaa579",
"request": "581536d3312bca41"
},
"environment": "eastus",
"location": "eastus",
"time": "2020-10-08T14:14:04.5157895+00:00",
"componentName": "environment-management"
}

issue persists after upgrading
Requirement already satisfied: azureml-train-automl-client in c:programdataanaconda3libsite-packages (1.15.0.post1)

[rtanase]

@wynnlone , the stacktrace above is still pointing to the unpatched 1.15.0 package. Did you restart your kernel after upgrading the package? If you haven't done so, can you please try it?

[wynnlone]

Nope. Just tried, still not working.

[areed145]

@wynnlone I had to explicitly add the package to the remote compute environment conda-dependencies, ie clone a curated environment, add the patch package, then register the env under a new name

[ThiagoGrabe]

@wynnlone , the stacktrace above is still pointing to the unpatched 1.15.0 package. Did you restart your kernel after upgrading the package? If you haven't done so, can you please try it?

restarting the kernel worked! Thanks.

Steps:

1. !pip install --upgrade azureml-train-automl-client
2. Restart the kernel
3. Run/submit the experiment.

[wynnlone]

@wynnlone , the stacktrace above is still pointing to the unpatched 1.15.0 package. Did you restart your kernel after upgrading the package? If you haven't done so, can you please try it?

restarting the kernel worked! Thanks.

No, it didn't work for me. How did you do it?

[ThiagoGrabe]

@wynnlone , the stacktrace above is still pointing to the unpatched 1.15.0 package. Did you restart your kernel after upgrading the package? If you haven't done so, can you please try it?

restarting the kernel worked! Thanks.

No, it didn't work for me. How did you do it?

Steps:

1. !pip install --upgrade azureml-train-automl-client
2. Restart the kernel
3. Run/submit the experiment.

[rtanase]

@wynnlone, do you mind describing your scenario? Where are you trying to submit a remote experiment from? Are you using AzureML Pipelines by any chance?

[wynnlone]

@rtanase It worked. I used azure online notebooks instead, and typed !pip install --upgrade azureml-train-automl-client first.