Even though the main idea behind TFX is to enable production setting ML use cases, it would be helpful if people can use "available datasets" to test out the components and experiment with different structures.
Therefore, I think it's useful to provide tfds support in TFX ExampleGen components. Since currently ExampelGen only works with tfrecord and csv, the tf.dataset type output by tfds is not natively supported.
The easy way to provide support is to have a library that converts tf.dataset to csv and CsvExampleGen.
But there might be potential benefits to doing this natively as a stand along tfdsExampleGen. So it's worth discussing what's the best way forward.
tommywei110
All 5 comments
@1025KB Can you discuss this with @tommywei110 and @davidzats-eng ?
zhitaoli
on 20 Jun 2020
convert to csv might lose some type information. a custom DatasetExampleGen that takes dataset name and convert to tf.Example should be fine
1025KB
on 20 Jun 2020
@tommywei110,
Can you please confirm if @1025KB's comment has answered your question. Thanks!
rmothukuru
on 22 Jun 2020
Are there plans for this feature? Would a PR be welcome that would provide this functionality?
lxkain
on 2 Apr 2021
Cross-referencing:
https://github.com/tensorflow/datasets/issues/2082
lxkain
on 19 Apr 2021
Related issues
casassg
路
5Comments
tordsae
路
4Comments
rcrowe-google
路
3Comments
jeongukjae
路
3Comments
Malonl
路
6Comments
Most helpful comment
convert to csv might lose some type information. a custom DatasetExampleGen that takes dataset name and convert to tf.Example should be fine