Mlflow: Broken mlruns after experiment deletion

[drorata]

System information

• No custom code
• macOS 10.13.6
• pip
• mlflow --version = 0.6.0
• Python version: 3.6
• Exact command to reproduce: mlflow experiments delete 1

Describe the problem

After deleting an experiment the UI breaks. An error is printed:

Exception: Yaml file '/my/project/dir/mlruns/.trash/meta.yaml' does not exist.

After manually deleting the directory .trash:

rm -rf mlruns/.trash

The UI functions as expected.

EDIT: However, after running another run, the problem pops up again.

[normster]

+1 getting the same issue on Ubuntu with no custom code, etc.

I copied the meta.yaml file from another experiment and the problem seemed to go away.

[andrewmchen]

Thanks for reporting the issue. I can report most of these issues and am working on a fix. The specific issue here is that we don't validate creating runs against deleted experiments.

[Hprasath]

@normster I am facing a similar issue as yours and I don't find meta.yaml under any experiment. Could you or @andrewmchen help me get around this? I haven't deleted the experimented manually and the folder structure for the current run and the older ones are identical.

[novakov-alexey]

I have the same issue:

  File "/home/an/miniconda3/lib/python3.7/site-packages/mlflow/store/tracking/file_store.py", line 360, in create_run
    experiment = self.get_experiment(experiment_id)
  File "/home/an/miniconda3/lib/python3.7/site-packages/mlflow/store/tracking/file_store.py", line 270, in get_experiment
    experiment = self._get_experiment(experiment_id)
  File "/home/an/miniconda3/lib/python3.7/site-packages/mlflow/store/tracking/file_store.py", line 247, in _get_experiment
    meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
  File "/home/an/miniconda3/lib/python3.7/site-packages/mlflow/utils/file_utils.py", line 159, in read_yaml
    raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
mlflow.exceptions.MissingConfigException: Yaml file <myproject>/mlruns/0/meta.yaml' does not exist

[bneijt]

I ran into the same issue with mlflow 1.8.0. I assumed mlflow ui would pick up my MLFLOW_TRACKING_URI from the environment, but it does not. Which means that my mlflow ui was looking for local metadata while my train.py was writing information to a local sqlite database because of having used export MLFLOW_TRACKING_URI=sqlite:///mlflow_tracking.sqlite before running my train.py.

Solution was to pass the alternative tracking uri as a commandline parameter to mlflow ui:
mlflow ui --backend-store-uri $MLFLOW_TRACKING_URI