Tfx: Issue in TFX Transform component after upgrade it from 0.22 -> 0.24.1

[sakinaljana]

Hello folks,

Recently i'm trying to upgrade the TFX version to the latest (v0.24.1) from the old one (v0.22). I didn't change anything in the pipeline code, just pure library upgrade. After running the pipeline, found following issue inside Transform component:

Any idea why this happened? seems Transform component want to iterate the item but not found, hence throw StopIteration. It works well with v0.22. Thanks in advance!

[sakinaljana]

After downgrade it to v0.23.1, it works. So i suspect there might be some bugs inside Transform component for v0.24.1

[zoyahav]

Is there a colab notebook that we can use to repro this issue?
(I tried running this colab but it completed successfully - https://www.tensorflow.org/tfx/tutorials/tfx/components_keras)

[sakinaljana]

@zoyahav you can try to repro this colab. This is not my actual code, but the code quite similar like this one.

[hanneshapke]

Hi @zoyahav ,

I see the same issue as @sakinaljana reported, but not in all pipelines.

Pipeline with a failing TFT component:
https://colab.research.google.com/gist/hanneshapke/805ae7616dc240b0face3832156d746c/tfx_pipeline_for_bert_preprocessing_wo_tf-example.ipynb
Pipeline with a working TFT component: https://colab.research.google.com/gist/hanneshapke/a70897306c231dc55b93440b934ae4c7/interactive_pipeline_errors_with_tfx_0-24-1.ipynb

I doubt that this is an issue, but just in case: The working example ingests the data with the CSVExampleGen, while the failing pipeline ingests from TFRecords. In fact, all my failing pipelines ingest from TFRecords, but this seems more correlation than causation. Can't think why the TFT would fail because of it.

[mshearer0]

The failing example I have ingests from BigQuery.

[zoyahav]

Thanks everyone, I've confirmed on the first colab in https://github.com/tensorflow/tfx/issues/2668#issuecomment-712288375 that disabling cache prevents this issue and Transform succeeds (as a temporary workaround):

transform = Transform(
    examples=example_gen.outputs['examples'],
    schema=schema_gen.outputs['schema'],
    module_file=os.path.abspath("transform.py"),
    disable_analyzer_cache=True)

I'll look for the root cause and hopefully make a patch release with a fix.

[mshearer0]

Thanks @zoyahav - that works for me.

[zoyahav]

Btw, the issue should be fixed on latest tfx nightly with https://github.com/tensorflow/tfx/commit/e6c3d0560f2ec8304982abd4747c337c20113ad7 merged. See:
https://github.com/tensorflow/tfx/blob/master/docs/guide/index.md#nightly-packages

[mshearer0]

@zoyahav - I'm still getting the same behaviour - was the fix included in last night's build?

[zoyahav]

I've verified that the provided repro notebook in https://github.com/tensorflow/tfx/issues/2668#issuecomment-712288375 with the latest nightly tfx version (0.25.0.dev20201028) is now working.

@mshearer0 could you please verify which version of TFX you're running with and confirm that you're seeing the same error as before?
Could you provide a simple repro notebook if so? (it doesn't need to read from BigQuery, a static small dataset is fine)

[mshearer0]

@zoyahav - I’m running tfx 0.24.1 with the example from: https://github.com/GoogleCloudPlatform/ml-design-patterns/blob/master/06_reproducibility/storm_reports/similar_reports.ipynb

Nb: see https://github.com/GoogleCloudPlatform/ml-design-patterns/issues/17

And fix for type error in last comment at:

https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4573

[zoyahav]

As mentioned tfx 0.24.1 still has the issue present, only on the nightly version is the issue fixed.

The latest nightly can be install by running pip install -i https://pypi-nightly.tensorflow.org/simple tfx.

[mshearer0]

@zoyahav - yes, I’ve pip installed the nightly update on 0.24.1 base

[hanneshapke]

@zoyahav I have tested one pipeline with the tfx-nightly release (0.25.0.dev20201028) and it works on my side.
Thanks to the TFX team for providing nightly releases!

@mshearer0 After installing the tfx nightly version, you should see a version number like 0.25.0.dev ... You can check the version number with

import tfx
print(tfx.__version__)

If you see a version number like 0.24.1 or lower, pip might not update/uninstall your previous tfx version. Maybe you have to force an update with -U:

pip install -Ui https://pypi-nightly.tensorflow.org/simple tfx

[mshearer0]

Thanks @hanneshapke - Pip wasn't updating properly and i hadn't spotted it, forcing with -U gives 0.25.0.dev20201028.

@zoyahav - works now, thank you.

[zoyahav]

Thank you both for verifying. I believe we can close this issue once 0.25 is released.

[sakinaljana]

I think we can close this since already solved in 0.25

[tensorflow-butler[bot]]

Are you satisfied with the resolution of your issue?
Yes
No

[hanneshapke]

The implementation works for me on version 0.25. Thank you all for the updates!