Tfx: Addding Support for Image Dataset

[Mageswaran1989]

Hi,
We are looking for an pipeline example around Mnist or Cifar10 or Coco data sets.
If you could guide us on how to implement one also will do, we would be happy to support.

Thanks.

@thak123 FYI

[Mageswaran1989]

TFRecords + TFDV @ https://github.com/aaronlelevier/apache_beam_examples/blob/3afa0e411676c22734e7a2189602eda611938597/notebooks/MNIST-all-data-and-partition-fn.ipynb

[1025KB]

Hi,

We are creating a import example gen for directly import external tfrecord files (with tf example in it) and create train and eval split for downstream component

a custom component is also doable, we are working on creating an example for people to follow

[thak123]

@1025KB I am eagerly waiting for the example. The preprocessing_fn reads the content of the tfrecord files as sparse tensors. I am unable to convert them to fixedlen features and reshape the image data.

[1025KB]

preprocessing_fn is for Transform, so you are able to custom ExampleGen to convert your input source to tf.Examples?

[1025KB]

if you are talking about custom ExampleGen for reading data file format that we don't support yet, check this

you can also do your own conversion before TFX, and use ImportExampleGen to directly import the tf.Examples, check this

if your problem is Transform related, let me know and I can find you a Transform expert

[ruoyu90]

@thak123 There are some pending work in the Schemagen to enable our support for fixedlen features. We will keep this thread updated.

[thak123]

preprocessing_fn is for Transform, so you are able to custom ExampleGen to convert your input source to tf.Examples?

Thanks for showing me the right direction. I am able to train the model now.