Tfx: MNIST TFX Pipeline

[Iso-oss]

Hello!

First, I would like to thank you for your work. I am currently trying to create a TFX pipeline capable to train a deep learning model on MNIST data set. ImportExampleGen, StatisticGen and SchemaGen are pretty straight forward to setup with the documentation. My data are stored as tf records and images are put into tf records as bytes. I am now struggling to setup up a Transform components that will reshape images and apply some preprocessings (zero mean, unit standard deviation for instance). How can I use Tensorflow Transform to achieve this step ?

Thank you !

[ruoyu90]

Hi @Iso-oss , thanks for trying out TFX :)

In transform stage, we want to use tensor operations to transform input tensors and a graph representing the transformation will be created and passed down to downstream component. You can refer to TFT site for more information.

For you specific case, maybe tf.image API can help, especially this one.

[Iso-oss]

Thank you for your response ! With your answer I managed to transform images as I wanted. I just have one last issue about shapes. During the reshaping steps if I try to recover values for height and width from inputs with the following code :

height = _fill_in_missing(inputs["height"]) (same _fill_in_missing function from the taxi example)
height = tf.cast(height, tf.int32)
height = tf.reshape(height, (1,))

and the same from width, I get the following error :

ValueError : Tensor("map/TensorArrayStack/TensorArrayGatherV3:0", shape=(?, ?, ?, ?), dtype=float32) has invalid shape (?, ?, ?, ?) for FixedLenFeatures: apart from the batch dimension, all dimensions must have known size

Is there a workaround avoiding hardcoding or passing those by config file ?

Thank you.

[ruoyu90]

IIUC the question is about to get the shape of the input tensor instead of hardcoding the output shape when calling tf.reshape? You can always get the shape of the tensor by calling tf.shape which will give you the shape of the tensor in the format of List[int]. Note that the first element of the result is the batch size.

[Iso-oss]

Thank you, I managed to do the whole pipeline as I wanted. Great work, keep it up ! I just have one last issue with the ModelValidator and the Evaluator, I get the following error while running the full pipeline on Airflow :

[Current date] {__init__.py:1580} ERROR - An op outside of the function building code is being passed

a "Graph" tensor. It is possible to have Graph tensors

leak out of the function building context by including a

tf.init_scope in your function building code.

For example, the following function will fail:

 @tf.function

 def has_init_scope():

   my_constant = tf.constant(1.)

   with tf.init_scope():

     added = my_constant * 2

The graph tensor has name: ParseExample/ParseExample:1 [while running 'EvalCurrentModel/ExtractAndEvaluate/Predict/Predict']

Traceback (most recent call last):

 File "my_tfx_env/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 4229, in identity

   name, _ctx._post_execution_callbacks, input)

tensorflow.python.eager.core._FallbackException: This function does not handle the case of the path where all inputs are not already EagerTensors.

My TFX version is the current Master, my Tensorflow (nightly) version is 1.15.0-dev20190714 and my Python version is 3.7.2.

Thank you for your time !

[ruoyu90]

@Iso-oss It seems TF is complaining about mixing graph mode and eager mode. I assume the trainer runs fine? Could you attach the trainer code so that we have better context?

@mdreves could you shed some light on this?

[mdreves]

Can you try this with TF version 1.14.

[Iso-oss]

The pipeline works now :). I mixed up labels keys. Thank you for your help and keep on with TFX :)

[zhitaoli]

Hi @Iso-oss, good to hear that things eventually work. Are you willing to publish your example pipeline, either in this repo (through a PR) or in your own? We are looking to expand example coverage of TFX and some of our engineers are looking at creating an image based example, but we are definitely supportive if anyone in the community wants to contribute.

Thanks in advance!

[chanshah]

The pipeline works now :). I mixed up labels keys. Thank you for your help and keep on with TFX :)

Hi @Iso-oss I'm glad the pipeline is working for you now. Would you be willing to publish / contribute the sample for this pipeline for more developers to benefit from this?

[gilbblig]

Hi @Iso-oss, good to hear that things eventually work. Are you willing to publish your example pipeline, either in this repo (through a PR) or in your own? We are looking to expand example coverage of TFX and some of our engineers are looking at creating an image based example, but we are definitely supportive if anyone in the community wants to contribute.

Thanks in advance!

Hi @zhitaoli , I'm also looking to apply a pipeline based on TFX and Kubeflow for image classification. Since I just started by replicating a csv-based pipeline, I wonder if there is already some code you'd be willing to share.

[valeriano-manassero]

@gilbblig Maybe this can be useful. I just started a collection of on premise Kubeflow TFX pipelines:
https://github.com/valeriano-manassero/tfx-kubeflow-pipelines

There's also a Cifar-10 dataset pipeline.