Tfx: Changing launcher for InteractiveContext

[casassg]

I'm developing an internal launcher that contains some optimizations for internal usecases (and some instrumentation).

I've been able to successfuly make it run by changing the PipelineConfig(supported_launcher_classes=[MyLauncher]) when using the AirflowLauncher and BeamLauncher but it seems that it can't be changed for InteractiveContext.

I know InteractiveContext is an experimental feature, but looking at the codebase it seems that there's a reference to enable changing the launcher class.

Wondering if there's any good way to do this at the moment (or plan to enable this in the near future) or if I should just reimplement InteractiveContext to overwrite the launcher.

[1025KB]

InteractiveContext is for notebook use so we hardcode InProcessComponentLauncher.

it's not configurable for now, but if you just want to try out your launcher, change the interactive_context locally?

[casassg]

How can I change that? The only way I see now is to vendor the run method in and change the launcher there as it's hardcoded.

[1025KB]

You probably need to update the InteractiveContext code, upload it to a place that can be accessed by Colab, pip install your own package to overwrite InteractiveContext, and use your own InteractiveContext in the Colab

Why do you need to change the launcher? if you just want to test the launcher, consider using local e2e example (with your own tfx package) which doesn't require InteractiveContext (for Colab only)

[casassg]

We have an internal launcher that tracks components executions/failures across the company. Allows us (developers) to understand our internal usage (modelers, other engineers) better and track performance of components.

Another use case that I was thinking was to add integrations with internal services (data registration services and/or internal dashboards).

I believe launchers are the best place for this type of customization (better than having to instrument component executors one by one?), and so far we are only missing enabling this in the InteractiveContext.

I will enable it by overwriting the run method in an internal InteractiveContext.

[1025KB]

I see, I will keep this as a feature request for supporting different launcher in InteractiveContext