Tfx: tfdv.GenerateStatistics's output

[xiaochuan-du]

As example suggest,

  with beam.Pipeline(runner=...) as p:
    _ = (p
         | 'ReadData' >> beam.io.ReadFromTFRecord(data_location)
         | 'DecodeData' >> beam.Map(TFExampleDecoder().decode)
         | 'GenerateStatistics' >> GenerateStatistics()
         | 'WriteStatsOutput' >> beam.io.WriteToTFRecord(
             output_path, shard_name_template='',
             coder=beam.coders.ProtoCoder(
                 statistics_pb2.DatasetFeatureStatisticsList)))

Beam is a parallel processing framework, thus the output of GenerateStatistics should be more than one file that records statistics for different data chunks.
Do we need to merge these statistics before validating the test dataset? For example, col1 in data_chunk0 train dataset's legal range is from 0 to 5, col1in data_chunk1 legal range is from 1 to 6, thus in gross, col1 valid data range should be from 0 to 6. Is there a buildin function to complete the task?

[xiaochuan-du]

According to infer_schema's doc ,

statistics: A DatasetFeatureStatisticsList protocol buffer. Schema inference is currently only supported for lists with a single DatasetFeatureStatistics proto.

[caveness]

[xiaochuan-du]