Machinelearningnotebooks: Strange exception when submitting ScriptRun locally

[SamVanhoutte]

Hello all,

Just followed the very basic steps to try to execute a ScriptRun locally.
I uploaded my repo notebook here: https://gist.github.com/SamVanhoutte/58433f6fc8646dd7c2aa4674ef4f28f0 - you should be able to run it against any AzureML workspace following the executions.

The code I executed is very straight forward:

work_env =  Workspace.from_config()
exp = Experiment(workspace=work_env, name='Repo')

run_local = RunConfiguration()
run_local.environment.python.user_managed_dependencies = True

config = ScriptRunConfig(source_directory='.', script=script_file_name, run_config = run_local)
script_run = exp.submit(config)
script_run.wait_for_completion(show_output=True)

The error I'm getting is very strange and does not help me at all, as a user:

Entering context manager injector. Current time:2020-05-14T12:16:22.325243
Starting the daemon thread to refresh tokens in background for process with pid = 20466
Warning: Unable to import azureml.history. Output collection disabled.
Preparing to call script [ script_file.py ] with arguments: []
After variable expansion, calling script [ script_file.py ] with arguments: []



The experiment failed. Finalizing run...
Error occurred when setting run status to Finalizing: 'NoneType' object has no attribute 'get_process_name'
Traceback (most recent call last):
  File "azureml-setup/context_manager_injector.py", line 278, in <module>
    execute_with_context(options.inject, options.invocation)
  File "azureml-setup/context_manager_injector.py", line 191, in execute_with_context
    print("Error occurred when setting run status to Finalizing: {}".format(ex))
  File "/private/var/folders/s_/lzy93vds1hz0stvfdr565bdm0000gn/T/azureml_runs/Repo_1589451377_ed72aae1/azureml-setup/utility_context_managers.py", line 152, in __exit__
    end_time = time.time()
AttributeError: 'NoneType' object has no attribute 'time'

In general it seems there's a lot of samples that are outdated and it's quite hard to get some very straightforward things working.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 1d94ed89-5c02-627d-be4b-7be85ffeb876
• Version Independent ID: 581b19dc-544c-6a91-fa47-3d582f23a34b
• Content: azureml.core.workspace.Workspace class - Azure Machine Learning Python
• Content Source: AzureML-Docset/stable/docs-ref-autogen/azureml-core/azureml.core.workspace.Workspace.yml
• Service: machine-learning
• Sub-service: core
• GitHub Login: @j-martens
• Microsoft Alias: jmartens

[v-strudm-msft]

@SamVanhoutte Thanks for your feedback. We're routing this to the appropriate team to follow up.

[sumitkuml]

Hi,
Getting same error while trying to start training using custom docker.
anyone spot a workaround for this?

[SamVanhoutte]

I didn't get any of this to work locally, no. Really hoping for a solution here, @maxluk ?

[swanderz]

what version of the SDK is this on? We're seeing something similar with 1.7.0.post1, but the error goes away when we change show_output to be False

[SamVanhoutte]

I just retried (have everything wrapped in a custom library, that uses the above code) and it seems to be working without an issue for me now (and I still use the show_output=True)

[chengyu-liu-cs]

I am also having the same error. I have tried different versions including latest 1.12.0. But the problem remains.

I was running locally

[mx-iao]

Let me try to repro and route this to the product team.

[tailaiw]

Having the same issue too.

2020/11/07 03:35:35 logger.go:297: Attempt 1 of http call to http://10.0.0.5:16384/sendlogstoartifacts/info
2020/11/07 03:35:35 logger.go:297: Attempt 1 of http call to http://10.0.0.5:16384/sendlogstoartifacts/status
[2020-11-07T03:35:35.556096] Entering context manager injector.
[context_manager_injector.py] Command line Options: Namespace(inject=['ProjectPythonPath:context_managers.ProjectPythonPath', 'RunHistory:context_managers.RunHistory', 'TrackUserError:context_managers.TrackUserError'], invocation=['hello.py'])
Starting the daemon thread to refresh tokens in background for process with pid = 60
Warning: Unable to import azureml.history. Output collection disabled.
Current directory:  /mnt/batch/tasks/shared/LS_root/jobs/tailai-aml-workspace-0/azureml/day1-experiment-hello_1604718100_a13ea717/mounts/workspaceblobstore/azureml/day1-experiment-hello_1604718100_a13ea717
Preparing to call script [ hello.py ] with arguments: []
After variable expansion, calling script [ hello.py ] with arguments: []

Script type = None
Traceback (most recent call last):
  File "/mnt/batch/tasks/shared/LS_root/jobs/tailai-aml-workspace-0/azureml/day1-experiment-hello_1604718100_a13ea717/mounts/workspaceblobstore/azureml/day1-experiment-hello_1604718100_a13ea717/azureml-setup/context_manager_injector.py", line 329, in <module>
    execute_with_context(cm_objects, options.invocation)
  File "/mnt/batch/tasks/shared/LS_root/jobs/tailai-aml-workspace-0/azureml/day1-experiment-hello_1604718100_a13ea717/mounts/workspaceblobstore/azureml/day1-experiment-hello_1604718100_a13ea717/azureml-setup/context_manager_injector.py", line 234, in execute_with_context
    print("[{}] FinalizingInRunHistory is not called".format(datetime.datetime.utcnow().isoformat()))
  File "/mnt/batch/tasks/shared/LS_root/jobs/tailai-aml-workspace-0/azureml/day1-experiment-hello_1604718100_a13ea717/mounts/workspaceblobstore/azureml/day1-experiment-hello_1604718100_a13ea717/azureml-setup/utility_context_managers.py", line 155, in __exit__
    end_time = time.time()
AttributeError: 'NoneType' object has no attribute 'time'

[2020-11-07T03:35:41.642565] Finished context manager injector with Exception.
2020/11/07 03:35:41 logger.go:297: Failed to run the wrapper cmd with err: exit status 1
2020/11/07 03:35:41 logger.go:297: Attempt 1 of http call to http://10.0.0.5:16384/sendlogstoartifacts/status
2020/11/07 03:35:41 sysutils_linux.go:235: mpirun not found, trying job with default values: MPI publisher: open ; version: 
2020/11/07 03:35:41 logger.go:297: Process Exiting with Code:  1

[swanderz]

@tailaiw can you share the SDK version you're on and the control plane code (not the training script but how you submit the run)?

[rdzotz]

While troubleshooting this error I found it also occurs when running the tutorial register-model-deploy-local.ipynb