Machinelearningnotebooks: Bug in Estimator class when runconfig specifies "target: local"

[swanderz]

background

Using azureml SDK version 1.0.10

When specifying a local compute for an Estimator, the following Attribute error is thrown inside the _get_telemetry_values() method (lines 200-1) of azureml/train/estimator.py:
'RunConfiguration' object has no attribute 'type'

Commenting the lines out, the code runs without issue. Somehow I think that the _compute_target attribute is not being properly assigned when the .runconfig has type: local specified.

telemetry_values['computeTarget'] = self._compute_target if isinstance(self._compute_target, str) else \
    self._compute_target.type if self._compute_target else "amlcompute"

code to reproduce

import azureml
from azureml.core import Workspace, Run
from azureml.core.runconfig import RunConfiguration
from azureml.train.estimator import Estimator

# create local compute target
run_local = RunConfiguration()
run_local.environment.python.user_managed_dependencies = True

est_config_local = Estimator(
    source_directory = "./compute",
    use_docker = False,
    entry_script= "train.py",
    script_params = {
            "--data-folder": "C:\\temp\output",
            "--objective": "binary",
        },
    compute_target = run_local
)

from azureml.widgets import RunDetails
run = exp.submit(est_config_local, show_output=True)
run.wait_for_completion(show_output=False)
RunDetails(run).show()

[rastala]

Try this instead:

est_config_local = Estimator(
    source_directory = "./compute",
    use_docker = False,
    entry_script= "train.py",
    script_params = {
            "--data-folder": "C:\\temp\output",
            "--objective": "binary",
        },
    compute_target = "local",
    environment_definition=run_local.environment
)

[swanderz]

_quick tangent_: big thanks to everyone else on the team who's helped me so far: @j-martens, @SKrupa @archanpi @heatherbshapiro and everyone else.

I spent two hours just now trying to make a 'reproducible' example by putting a sample notebook, both runconfigs, dependencies.yml, control_log, and driver_log into this gist
https://gist.github.com/swanderz/c057f375b5b4800dc76bda6cee22cf17

@rastala I really have two asks (sorry for be so difficult!)

1. Run an estimator on AML compute using the runconfig as much as possible (like with PythonScriptStep or AutoMLConfig
2. Run an estimator on "local" compute where I can point to a specifically to a prebuilt env and not use docker. This is my dev dream-state because right now the script runs in 14 seconds, but takes 10 min to build a fresh env/container

AML Compute

I provide three examples, one that is working, one that isn't, and my attempt to get it working again.

Local compute

for the local compute, I'm completely lost and only have managed to get it to run when I comment out those lines of azureml/train/estimator.py.

When I try the code you shared it starts a job that never finishes that I have to cancel manually so there aren't any logs.

I'd like to know if I can specify a specific python environment to run in. in local.runconfig i'm specifying an env-specific python path which I thought would be enough, but it is ignored and instead:

• if use_docker = True, a new env will be built in ~\.azureml\envs, and
• if use_docker = False, the first env the SDK created in ~\.azureml\envs is used, in my case,

[hning86]

Looks like the above code is a bit buggy. here is what you should do:

# create local compute target
run_local = RunConfiguration()
run_local.environment.python.user_managed_dependencies = True
run_local.environment.python.interpreter_path = "/path/to/my/python/interpreter"
run_local.environment.docker.enabled = False

est_config_local = Estimator(
    source_directory = "./my_project_folder",
    entry_script= "train.py",
    script_params = {
            "--data-folder": "C:\\temp\output",
            "--objective": "binary",
        },
    compute_target = "local",
    environment_definition = run_local.environment
)

[swanderz]

Update Hallelujah, I've finally gotten both local and amlcompute working as I'd like now. With these setup, my team will definitely have a faster velocity on product development (and AML adoption)

For amlcompute:
I had userManagedDependencies incorrectly set to true.
For local:
I was referring to the run_config, run_local, instead of the string, "local",

last questions:
@hning86 . In the code provided above, is it the case that
condaDependenciesFile: aml_config/conda_dependencies.yml
is overriden by interpreterPath argument?

[hning86]

Correct. When you set user_managed_dependencies to True, we ignore the conda dependencies you provide, since the intent here is that you are pointing us to an existing Python environment with all your dependencies already installed. As a general rule: we will never install any packages into an existing Python environment you already have (because things could potentially get screwed up with such operation).