Tfx: Schema and statistics in Transform TFX Pipeline Component

[wsuchy]

The main entry point in Transform component (preprocessing_fn) should also provide computed Stats and Schema next to the inputs. In some scenarios users might want to benefit from the statistics e.g. to eliminate unnecessary features.

[gowthamkpr]

@wsuchy Can you please also try posting this issue in the tensorflow/transform as this feature also is primarily targeted towards transform component. Thanks!

[wsuchy]

I've created related issue: https://github.com/tensorflow/transform/issues/136

[zoyahav]

Could you please expand on what this feature request is asking for?
Stats and transformed schema are computed by the Transform component, are you asking for easier access?

[wsuchy]

@zoyahav Indeed. Currently in the TFX Transform Pipeline Component the processing_fn has a signature of

processing_fn(inputs:Dict[tf.Tensor]) -> Dict[tf.Tensor]

and I am proposing to change it as follows:

processing_fn(inputs:Dict[tf.Tensor], metadata:DatasetMetadata, stats:DatasetFeatureStatisticsList ) -> Dict[tf.Tensor]

or even better, as a higher order function / factory:

processing_fn_factory(metadata:DatasetMetadata, stats:DatasetFeatureStatisticsList ) 
-> Callable[Dict[tf.Tensor], Dict[tf.Tensor]]

So now my processing function could depend on the stats and schema.
I know I could reach for it Stats and Schema manually, yet it requires me to talk to the SQLite database and keeping track of paths, etc.

[JoshLipschultz]

I'd like to re-up this request—having easier access to the metadata and schema from within the Transform callback would be tremendously useful. Some feature transformations could depend on the inferred schema. Is there some other workaround?