Tfx: Example to send beam jobs to cloud dataflow and training on Cloud ML Engine

[josekidengan]

Example shows the bare metal running of ml pipeline using airflow/kubeflow.
However there is no information regarding how to launch the beam jobs to cloud data-flow and training on ML engine to scale. Also how to serve model on google Cloud ML Engine.

[1025KB]

Hi, Jose, we are working on an example of data-flow and CLME, it will be released in Q2

[krazyhaas]

Hi Jose.

Take a look at https://github.com/tensorflow/tfx/blob/master/tfx/examples/chicago_taxi_pipeline/taxi_pipeline_kubeflow.py. You'll see it sets the beam runner to run on Dataflow via beam_pipeline_args, CMLE training via _ai_platform_training_args, and pushes to CMLE serving with _ai_platform_serving_args. While it uses the KFP runner, the same will work with Airflow. Is that what you're looking for?

[andrewlarimer]

Will setting the Trainer's executor_class to that AI Platform executor (and passing the correct AI Platform args) allow AI Platform to do hyperparameter tuning? If so, is there somewhere I can read more about the executor_class and the AI Platform executor? And/or does TFX provide other hyperparameter tuning support we should know about?

[zhitaoli]

@andrewlarimer Hyper parameter tuning definitely deserves its own thread to discuss, as the this one mainly tracks how to do normal training and data processing using dataflow.

To answer your question:

Assuming https://cloud.google.com/ml-engine/docs/tensorflow/using-hyperparameter-tuning is what you are looking for, I think the work happened so far can be used as a stepping stone, but we need more work to plumbing the additional command line arguments generated by the HParam tuning process, and also make sure it fits with our trainer executor.

If this is something useful for you, can you file a separate Github feature request? We'll discuss the best strategy to implement this.