Mlflow: [BUG] MLflow Projects: can't copy environment variables to docker container environment

[TaikiInoue]

Willingness to contribute

The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

• [ ] Yes. I can contribute a fix for this bug independently.
• [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
• [x] No. I cannot contribute a bug fix at this time.

System information

• Have I written custom code (as opposed to using a stock example script provided in MLflow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu18.04
• MLflow installed from (source or binary): Binary (pip)
• MLflow version (run mlflow --version): 1.10.0
• Python version: 3.6.5
• npm version, if running the dev UI: N/A
• Exact command to reproduce: mlflow run --docker-args runtime=nvidia .

Describe the problem

mlflow run doesn't work when copying the environment variables containing semi-colon ; from the host to the docker container.

Code to reproduce issue

This is my MLproject file. The problem is AZURE_STORAGE_CONNECTION_STRING. It contains semi-colon ;. Don't worry, all the values (e.g. github repo and commit_hash) are dummies.

name: hoge

docker_env:
    image: hoge
    environment: ["MLFLOW_TRACKING_URI", "DATABRICKS_HOST", "DATABRICKS_TOKEN", "AZURE_STORAGE_CONNECTION_STRING"]

entry_points:
    main:
        parameters:
            data_repo: {type: str, default: "https://github.com/hoge/hoge.git"}
            data_commid_hash: {type: str, default: "974hj06p45;higghc8351"}
            code_repo: {type: str, default: "https://github.com/hoge/hoge.git"}
            code_commit_hash: {type: str, default: "hhjd8hasy8751d9gljlgej"}
        command: "sh entry_points.sh"

I run the command, and then I got the following error. These values (e.g. environment variables and RUN ID) are dummies as well.

mlflow run --docker-args runtime=nvidia .

2020/09/03 15:46:58 INFO mlflow.projects: === Building docker image ssad:129739b ===
2020/09/03 15:46:59 INFO mlflow.projects: === Created directory /tmp/tmpczukugoy for downloading remote URIs passed to arguments of type 'path' ===
2020/09/03 15:46:59 INFO mlflow.projects: === Running command 'docker run --rm --runtime nvidia -e MLFLOW_RUN_ID=1t3trg57dbfewbvfweb -e MLFLOW_TRACKING_URI=databricks -e MLFLOW_EXPERIMENT_ID=626262052256456 -e DATABRICKS_HOST=https://adb-997587567985267467.0.azuredatabricks.net -e DATABRICKS_TOKEN=gfvfbwgg4rgrvg14b3tbtrnbrt50a1232 -e AZURE_STORAGE_CONNECTION_STRING=DefaultEndpointsProtocol=https;AccountName=hoge;AccountKey=hgKGljLK+Uwg3vadvsdavsdAl1i7ERR2vsavds/tgAJMG9/IvavsamHGsavdaBERfdbd==;EndpointSuffix=core.windows.net hoge:12r35b sh entry_points.sh' in run with ID 'ehkg31435bdfglhkgg35j4fg4' === 
"docker run" requires at least 1 argument.
See 'docker run --help'.

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container
bash: ssad:129739b: command not found
2020/09/03 15:47:01 ERROR mlflow.cli: === Run (ID '1484357d5b3f42a4a512945941e7e656') failed ===

What component(s), interfaces, languages, and integrations does this bug affect?

Components

• [ ] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [ ] area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [x] area/projects: MLproject format, project running backends
• [ ] area/scoring: Local serving, model deployment tools, spark UDFs
• [ ] area/server-infra: MLflow server, JavaScript dev server
• [ ] area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• [ ] area/uiux: Front-end, user experience, JavaScript, plotting
• [x] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

Language

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients
• [ ] language/new: Proposals for new client languages

Integrations

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations
• [ ] integrations/databricks: Databricks integrations

[harupy]

@TaikiInoue Thanks for filing this. I have created a fixed version for this issue on my branch. Can you install it using the command below and see if it solves the issue?

pip install git+https://github.com/harupy/mlflow.git@fix-docker-env

[TaikiInoue]

@harupy Thank you for the quick response! Your modification solves my issue :tada:

[harupy]

@TaikiInoue Awesome! I'll file a PR so this error won't occur in the next version.