Tfx: Shuffling in ExampleGen should be optional

[Mar-ai]

The Docs mention that ExampleGen "shuffles the dataset for ML best practice".
However, if the use case is a time series problem using sliding windows, shuffling before splitting in train and eval set is counterproductive as I'd need a coherent training set.

To accomplish this for now (as I understand it) one would have to create an entire custom ExampleGen by modifying base_example_gen_executor and remove 'Shuffle' >> beam.transforms.Reshuffle().

It would be great if this wasn't necessary and the shuffling in ExampleGen could be switched off directly when calling
example_gen = CsvExampleGen(input=examples) e.g. by using shuffle=False.

[codesue]

This would be a great feature to have! My colleague @casassg created an issue for this for a different use case https://github.com/tensorflow/tfx/issues/1907. We only resolved the issue because it was no longer relevant to what we were working on.

[casassg]

I believe the idea is that TFX will support SequenceExample for timeseries based datasets. May be wrong. It may be nice to get the shuffle=False option on ExampleGen until there's a good supported path for this.

[1025KB]

Hi, Mar-ai,

in ExampleGen, even we don't shuffle, there are no ordering guarantees. The splitting method uses the hash value of the record by default.

There are several options,

1. use feature base split method
2. pre-generated the train and eval split, and import them (shuffle is within split)
3. custom the ExampleGen with a customized split method

[Mar-ai]

We approached the data import for TimeSeries now like 1025KB's second suggestion, and split the data in /train and /eval before feeding it into the TFX-Pipeline. If I understood correctly, no shuffling is applied when the data is imported like that @1025KB?

I'd like to keep this issue open as feature request as other users are also interested in it :)
Thank you!

[1025KB]

The shuffle is within the split, it won't shuffle across splits.

if you provide one input split, we will split it into train and eval using the hash value of the record (or a feature if specified)
if you provide pre-generated train and eval split, the output train and eval is mapped to the input train and eval.

but for both, we will reshuffle the data within split before materialization (this reshuffle doesn't cause any issue as PCollection doesn't guarantee orders anyway)

[Mar-ai]

Apologies for being slow to comprehend here - I'm not very familiar with Apache Beam.
Is ExampleGen as is then at all a good approach to a time series problem, where I need the dataset to be in order?
So what would be needed is something like sequence examples as @casassg said above, or you mentioned here?

[1025KB]

Depending on the order of tf.Examples might be doable, e.g., A custom ExampleGen that outputs data in order and a dataset that read data in writing order. But I feel it would be better not to rely on the order between tf.Examples (the current TFX doesn't guarantee that).

SequenceExample is one way, or you can use the list in tf.Example to represent the order

[rezarokni]

One option would be to create the windows before the data is landed and consumed by ExampleGen. For example create your sliding window such that the Example has shape [timesteps,features]. You can then make use of tf.transform for a reshape. Similar to what is being done in this branch : https://github.com/GoogleCloudPlatform/dataflow-sample-applications/blob/TS-12-TFX-Upgrade-0.23.0/timeseries-streaming/timeseries-python-applications/MLPipeline/timeseries/encoder_decoder/encoder_decoder_preprocessing.py The downside to this approach is that the 'maximum' timestep available is fixed, as the window is already created before ExampleGen.

[ntakouris]

Creating the windows beforehand is a very very bad practise, which also inflates the size of the dataset by a huge margin. We can just use tf.data.dataset for that. Look for more info on the above issue that got mentioned.

A similar way to avoid unnecessary data duplication was used in the materialize=False parameter of the Transform component.

[rezarokni]

It is for sure a restrictive practice, but sometimes maybe necessary, it depends on the system as a whole.

Some more notes, that can help define the approach..
Apache Beam, does not guarantee order as mentioned by @1025KB ExampleGen will read the files in random order ( many threads can be reading in parallel). So even if the files that the examples are stored in have file names that would list in sequential order the read would not be sequential. And if the files are splittable, like uncompressed cvs files then a singles files read can be done by many threads as well.

Its potentially possible to write a custom ExampleGens,

• One version could use a Beam pipeline to read all the data , use the elements timestamps and then create sliding windows from that data. With the window parameters sent in to the ExampleGen at runtime.

• another option , which I have not had time to explore yet, would be to land fixed length sequences ( not sliding ) and then use Beam to process the metadata and create a processing Map with start end offsets for all the sequences windows then through GBK left-right combos to create the sliding windows from the fixed length sequences. SequenceExample would be a good candidate for storage here as the context could be used to provide the metadata needed for the first phase of the pipeline. But this is more complex than the first option and may not actually gain much in terms of processing time.

Another consideration... At its core one component will need to create a ordered sequence at some point. In a streaming prediction usecase, where the inference is being done in real time from a streaming source, the inference system needs to create the [timestep, feature] shape anyway, so in that case having that same system also output its values direct to a bucket ready for ExampleGen can make sense as the processing is being done already. However as pointed out the down side there is the amount of storage used does increase significantly, essentially by the length of the sliding window * the offset for the slide. A mitigation for this that is not valid for every use case is to downsample the data before adding to the sliding window. For example creation of fixed window First/Last/Max/Min objects which are then used within the sliding window to give objects of shape [[First/Last/Max/Min],[First/Last/Max/Min]...].

I hope to be able to explore the SequenceExample option with a custom examplegen in Dec.

[easadler]

This might not be the intended purpose of the component, but being able to create examples for our scoring sets is pretty useful for batch inference. Reshuffling is really expensive, but so is not being able to take advantage of all the data source connectors in TFX! If there is a way to get around it, it would be much appreciated. Maybe a custom component is the way to go?

[1025KB]

For custom examplegen, we recommend to still use the base_example_gen_executor, as it handles span/version/split and make sure the output is compatible with downstream components. Reshuffling is within split for each training instance (tf.example), the order of training instance shouldn't matter as Transform and Trainer component won't guarantee the read order too.

[1025KB]

If the order of tf.example is important, it means one tf.example doesn't map to a single training data instance.
For example, you need a group of tf.examples to do sessionization and generate one tf.examples as a single training data instance.

In that case, I actually recommend to do sessionization before tfx, sth like

raw log -> log processing pipeline (group raw log and do sessionization) -> raw tf.example (each represent a session, and it's single training data instance) -> TFX with ImportExampleGen

basically it would be good for each tf.example to represent one single training instance for Trainer's model.fit()

[vaskozl]

As mentioned in https://github.com/tensorflow/transform/issues/143 and https://github.com/tensorflow/tfx/issues/1907, reshuffle doesn't work well with DirectRunner and causes OOMing.

I've been patching out shuffling in every release and doing it in the DB query. This is yet another reason to give the option not to shuffle, at-least until we are able to shuffle with DirectRunner.

Other than being useful for those not on GCP, it would be useful to for IPython notebooks.