Tfx: TFX 0.24 fails on/after trainer step in interactive run

[lyschoening]

The typical TFX pipeline fails with a kernel restart after the trainer is run for the first time.

For example, when running the following colab, the kernel restarts if the trainer step is re-run.
https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/components_keras.ipynb#scrollTo=429-vvCWibO0

The runtime logs list this error:

Oct 5, 2020, 4:05:08 PM | WARNING | WARNING:root:kernel 31895490-d985-4a77-9518-021f4be051b0 restarted
-- | -- | --
Oct 5, 2020, 4:05:08 PM | INFO | KernelRestarter: restarting kernel (1/5), keep random ports
Oct 5, 2020, 4:05:07 PM | WARNING | 2020-10-05 14:05:07.377834: F ml_metadata/metadata_store/query_config_executor.cc:299] Unexpected oneof:

[gowthamkpr]

@lyschoening Please downgrade to tfx 0.22 and let me know if the issue still persists. Thanks!

[casassg]

Any story moving forward? This is blocking upgrade.

[lyschoening]

@gowthamkpr This issue does not occur in TFX 0.22, only in TFX 0.24.

I ran the above Colab substituting !pip install tfx for !pip install tfx==0.22.1 apache-beam==2.21.0 and that works without issues. (The TFX 0.22 install is broken in a different way, as it depends on apache-beam[gcp]>=2.21,<3, yet is incompatible with 2.24.0.)

[gowthamkpr]

@Iyschoening This is a know issue and will be fixed in the next release. Thanks!

[casassg]

@gowthamkpr do we have any ETA for that release?

[charlesccychen]

CC: @hughmiao

[calvinleungyk]

Hi @gowthamkpr @charlesccychen @hughmiao do we have any updates on the ETA? This blocks upgrade if we can't run training reliably. Thanks

[charlesccychen]

This will be fixed in https://github.com/tensorflow/tfx/pull/2652. We will need a patch release to resolve this issue.

CC: @zhitaoli @dhruvesh09

[tensorflow-butler[bot]]

Are you satisfied with the resolution of your issue?
Yes
No

[tensorflow-butler[bot]]

Are you satisfied with the resolution of your issue?
Yes
No

[charlesccychen]

Reopening pending patch release.

CC: @zhitaoli @dhruvesh09

[dhruvesh09]

TFX 0.24.1 is now available. Please feel free to test it and report any issues you find.
CC: @zhitaoli @charlesccychen

[casassg]

It works indeed! Ty for the fast turnaround!

[tensorflow-butler[bot]]

Are you satisfied with the resolution of your issue?
Yes
No