Mlflow: Can not create new independent run when mlflow forces to use nested run

Created on 24 Jan 2019 · 2Comments · Source: mlflow/mlflow

I create a new experiment and re-run mlflow.start_run() in Python. The code as below

mlflow.set_tag('project', 'my_project' )
mlflow.set_experiment('my_project')

with mlflow.start_run():
    lgb_model = lgb.LGBMClassifier(**lgb_params)
    .... //

The error:

Run with UUID 541c15b5fd954e1988d1ab17bccf74a2 is already active. To start a nested run call start_run with nested=True.

I try to deactivate runs by using mlflow.end_run(), but it just deactivates 1 run, and triggers the others to become active. The error still occurs.

How to deal with the problem? I want to create a new independent run for the new experiment, and no need using a nested run here.
Any help would be appreciated.
Thank you.

System information

OS Platform and Distribution: Linux Ubuntu 16.04
MLflow installed from: pip install mlflow
MLflow version: mlflow, version 0.8.1
Python version: Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux

Source

chaupmcs

Most helpful comment

Hi @chaupmcs : It looks like you are using mlflow.set_tag to create a tag before starting a run. Tags are defined on runs (not experiments)--not we are considering adding those in the near future.

Given you script -- mlflow.set_tag will create a new run and activate it. One of the mechanism to fix this would be to move after start_run

mlflow.set_experiment('my_project')

with mlflow.start_run():
    lgb_model = lgb.LGBMClassifier(**lgb_params)
    mlflow.set_tag('project', 'my_project' )
    ...

cc: @smurching who debugged this issue

mparkhe on 24 Jan 2019

👍7

All 2 comments

Given you script -- mlflow.set_tag will create a new run and activate it. One of the mechanism to fix this would be to move after start_run

mlflow.set_experiment('my_project')

with mlflow.start_run():
    lgb_model = lgb.LGBMClassifier(**lgb_params)
    mlflow.set_tag('project', 'my_project' )
    ...

cc: @smurching who debugged this issue

mparkhe on 24 Jan 2019

👍7

I get the "Run with UUID is already active" error every time I run a seemingly innocuous read-only function get_artifact_uri() ("get*" functions are not supposed to alter anything...) - as a pure side effect of its being run. It breaks a python 3 code that worked previously for hundreds of runs and saved metrics for three different algos to a MariaDB database. The UI shows an empty table entry, and a new db record is being created for the current experiment.

Restarting python 3 kernel in the notebook (apart from commenting out the offending function call) is also required to get rid of the problem.

Every time I restart mlflow server (container) I get a different UUID in the error message:
Exception: Run with UUID 0c2471380c5949b4af7a18d0f1972f23 is already active. To start a nested run, call start_run with nested=True

Trying to kill the run with mlflow.end_run(status="KILLED") fails, and mlflow server reports the following error (keep scrolling:):

2019/10/24 14:53:34 ERROR mlflow.server: Exception on /api/2.0/mlflow/runs/update [POST]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 158, in make_managed_session
    yield session
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 466, in update_run_info
    run = run.to_mlflow_entity()
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/dbmodels/models.py", line 174, in to_mlflow_entity
    metrics=[m.to_mlflow_entity() for m in self.latest_metrics],
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 282, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 710, in get
    value = self.callable_(state, passive)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 730, in _load_for_state
    session, state, primary_key_identity, passive
  File "<string>", line 1, in <lambda>
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 872, in _emit_lazyload
    .with_post_criteria(set_default_params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 531, in all
    return list(self)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 429, in __iter__
    self.session._autoflush()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1588, in _autoflush
    util.raise_from_cause(e)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1577, in _autoflush
    self.flush()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2470, in flush
    self._flush(objects)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2608, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2568, in _flush
    flush_context.execute()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj
    update,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 996, in _emit_update_statements
    statement, multiparams
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(MySQLdb._exceptions.OperationalError) (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')
[SQL: UPDATE runs SET status=%s, end_time=%s WHERE runs.run_uuid = %s]
[parameters: ('KILLED', 1571928814925, 'f6ba2d9431994eaa81cf28db67277387')]
(Background on this error at: http://sqlalche.me/e/e3q8 )

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 102, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 238, in _update_run
    request_message.end_time)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 468, in update_run_info
    return run.info
  File "/opt/conda/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 165, in make_managed_session
    raise MlflowException(message=e, error_code=INTERNAL_ERROR)
mlflow.exceptions.MlflowException: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(MySQLdb._exceptions.OperationalError) (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')
[SQL: UPDATE runs SET status=%s, end_time=%s WHERE runs.run_uuid = %s]
[parameters: ('KILLED', 1571928814925, 'f6ba2d9431994eaa81cf28db67277387')]
(Background on this error at: http://sqlalche.me/e/e3q8 )

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 105, in wrapper
    response.set_data(e.serialize_as_json())
  File "/opt/conda/lib/python3.6/site-packages/mlflow/exceptions.py", line 50, in serialize_as_json
    return json.dumps(exception_dict)
  File "/opt/conda/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'OperationalError' is not JSON serializable
2019/10/24 14:53:37 ERROR mlflow.server: Exception on /api/2.0/mlflow/runs/update [POST]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 158, in make_managed_session
    yield session
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 466, in update_run_info
    run = run.to_mlflow_entity()
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/dbmodels/models.py", line 174, in to_mlflow_entity
    metrics=[m.to_mlflow_entity() for m in self.latest_metrics],
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 282, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 710, in get
    value = self.callable_(state, passive)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 730, in _load_for_state
    session, state, primary_key_identity, passive
  File "<string>", line 1, in <lambda>
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 872, in _emit_lazyload
    .with_post_criteria(set_default_params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 531, in all
    return list(self)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 429, in __iter__
    self.session._autoflush()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1588, in _autoflush
    util.raise_from_cause(e)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1577, in _autoflush
    self.flush()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2470, in flush
    self._flush(objects)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2608, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2568, in _flush
    flush_context.execute()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj
    update,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 996, in _emit_update_statements
    statement, multiparams
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(MySQLdb._exceptions.OperationalError) (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')
[SQL: UPDATE runs SET status=%s, end_time=%s WHERE runs.run_uuid = %s]
[parameters: ('KILLED', 1571928814925, 'f6ba2d9431994eaa81cf28db67277387')]
(Background on this error at: http://sqlalche.me/e/e3q8 )

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 102, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 238, in _update_run
    request_message.end_time)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 468, in update_run_info
    return run.info
  File "/opt/conda/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 165, in make_managed_session
    raise MlflowException(message=e, error_code=INTERNAL_ERROR)
mlflow.exceptions.MlflowException: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(MySQLdb._exceptions.OperationalError) (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')
[SQL: UPDATE runs SET status=%s, end_time=%s WHERE runs.run_uuid = %s]
[parameters: ('KILLED', 1571928814925, 'f6ba2d9431994eaa81cf28db67277387')]
(Background on this error at: http://sqlalche.me/e/e3q8 )

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 105, in wrapper
    response.set_data(e.serialize_as_json())
  File "/opt/conda/lib/python3.6/site-packages/mlflow/exceptions.py", line 50, in serialize_as_json
    return json.dumps(exception_dict)
  File "/opt/conda/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'OperationalError' is not JSON serializable
2019/10/24 14:53:41 ERROR mlflow.server: Exception on /api/2.0/mlflow/runs/update [POST]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 158, in make_managed_session
    yield session
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 466, in update_run_info
    run = run.to_mlflow_entity()
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/dbmodels/models.py", line 174, in to_mlflow_entity
    metrics=[m.to_mlflow_entity() for m in self.latest_metrics],
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 282, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 710, in get
    value = self.callable_(state, passive)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 730, in _load_for_state
    session, state, primary_key_identity, passive
  File "<string>", line 1, in <lambda>
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/strategies.py", line 872, in _emit_lazyload
    .with_post_criteria(set_default_params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 531, in all
    return list(self)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/ext/baked.py", line 429, in __iter__
    self.session._autoflush()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1588, in _autoflush
    util.raise_from_cause(e)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1577, in _autoflush
    self.flush()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2470, in flush
    self._flush(objects)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2608, in _flush
    transaction.rollback(_capture_exception=True)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 153, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2568, in _flush
    flush_context.execute()
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
    rec.execute(self)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
    uow,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj
    update,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 996, in _emit_update_statements
    statement, multiparams
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/opt/conda/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/opt/conda/lib/python3.6/site-packages/MySQLdb/connections.py", line 226, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(MySQLdb._exceptions.OperationalError) (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')
[SQL: UPDATE runs SET status=%s, end_time=%s WHERE runs.run_uuid = %s]
[parameters: ('KILLED', 1571928814925, 'f6ba2d9431994eaa81cf28db67277387')]
(Background on this error at: http://sqlalche.me/e/e3q8 )

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 102, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 238, in _update_run
    request_message.end_time)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 468, in update_run_info
    return run.info
  File "/opt/conda/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/store/sqlalchemy_store.py", line 165, in make_managed_session
    raise MlflowException(message=e, error_code=INTERNAL_ERROR)
mlflow.exceptions.MlflowException: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely)
(MySQLdb._exceptions.OperationalError) (4025, 'CONSTRAINT `status` failed for `mariadb`.`runs`')
[SQL: UPDATE runs SET status=%s, end_time=%s WHERE runs.run_uuid = %s]
[parameters: ('KILLED', 1571928814925, 'f6ba2d9431994eaa81cf28db67277387')]
(Background on this error at: http://sqlalche.me/e/e3q8 )

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/conda/lib/python3.6/site-packages/mlflow/server/handlers.py", line 105, in wrapper
    response.set_data(e.serialize_as_json())
  File "/opt/conda/lib/python3.6/site-packages/mlflow/exceptions.py", line 50, in serialize_as_json
    return json.dumps(exception_dict)
  File "/opt/conda/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/conda/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'OperationalError' is not JSON serializable