Mlflow: [FR] Event based notifications for model registry changes

[gtadamson]

Willingness to contribute

The MLflow Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature (either as an MLflow Plugin or an enhancement to the MLflow code base)?

• [ ] Yes. I can contribute this feature independently.
• [ ] Yes. I would be willing to contribute this feature with guidance from the MLflow community.
• [X] No. I cannot contribute this feature at this time.

Proposal Summary

Subscription service for model registry changes. To use MLflow to drive deployment, we're currently querying MLflow periodically to determine if new versions were promoted to a different stage. It would probably be cleaner and simpler to register with MLflow and listen to an event stream (pub sub model).

Motivation

• What is the use case for this feature? Automated model deployment. MLflow drives another system.
• Why is this use case valuable to support for MLflow users in general? As MLflow is used more, the ability for it to easily interface with a production environment would be useful. Event based notifications are cleaner than time based querying with diff calculations.
• Why is this use case valuable to support for your project(s) or organization? We currently query all registered models, filter them down to the ones we care about for each environment, and then compute what the diffs are from what is stored in the deployment environment.
• Why is it currently difficult to achieve this use case? I wouldn't classify it as difficult, just non-optimal and requiring more code to interface with MLflow.

What component(s), interfaces, languages, and integrations does this feature affect?

Components

• [ ] area/artifacts: Artifact stores and artifact logging
• [ ] area/build: Build and test infrastructure for MLflow
• [ ] area/docs: MLflow documentation pages
• [ ] area/examples: Example code
• [x] area/model-registry: Model Registry service, APIs, and the fluent client calls for
  Model Registry
• [ ] area/models: MLmodel format, model serialization/deserialization, flavors
• [ ] area/projects: MLproject format, project running backends
• [ ] area/scoring: Local serving, model deployment tools, spark UDFs
• [x] area/tracking: Tracking Service, tracking client APIs, autologging

Interfaces

• [ ] area/uiux: Front-end, user experience, JavaScript, plotting
• [ ] area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• [ ] area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• [ ] area/windows: Windows support

Languages

• [ ] language/r: R APIs and clients
• [ ] language/java: Java APIs and clients

Integrations

• [ ] integrations/azure: Azure and Azure ML integrations
• [ ] integrations/sagemaker: SageMaker integrations

Details

[dmatrix]

@gtadamson

Thanks for filling this enhancement. In general, it's a seems like good idea to have an event-based notification that triggers an action, such as execute custom code at the registered endpoint. Webhooks for actions come to mind. Is that what you had in mind?

[gtadamson]

@dmatrix That is fine. We use Kafka as our messaging bus but having an HTTP container that interfaces with MLflow and then publishes to a topic is easy enough. I'm certainly not an expert in this domain so any solution that is amenable to others is acceptable to me.

[echarso]

Hi also interested in this in the form of webhooks :)

Webhooks are more effective than implementing a service pulling information from MLflow about the status of a model and then triggering a proper deployment job.

According to this #2383 (https://github.com/mlflow/mlflow/issues/2383)
I thought that there is ongoing work on that .