We are seeing issues with Trainer/Transform when used from InteractiveContext due to not reimporting the module (starting on 0.26). Any chance we can backport https://github.com/tensorflow/tfx/commit/1b2052b1635739e3e5a97245e16e210f3cc91068 to 0.26 (we are blocked on upgrade for a bit)
CC @rcrowe-google
@dhruvesh09 to see if this can be cherry-picked.
+1 since this seems a bug which worth backporting.
Thank you @casassg for reporting this.
Will update once the 0.26 patch fix is available
We are using InteractiveContext a lot for initial POC's.
@casassg Can you explain which issues you have? Just to prevent we are running into the same issues.
Thanks a lot
@SaschaHeyer When using Trainer or Transformer components in TFX in a notebook, the module files for each were not being reloaded correctly. So for example, I had a bug on my training code in my notebook, I fixed it, but Trainer would still be using the previous version of the module instead of reloading the new one. I had to restart the kernel on my notebook in order for the fix to be picked up (which is less than ideal as it means you have to reexecute again different steps).
We confirm the same behavior.
Thanks for reporting @casassg
@SaschaHeyer @casassg , please open an issue for the bug mentioned by @casassg in his latest comment.This will help in tracking and streamlining the triage process.Thanks.
@casassg
TFX 0.26.3 patch fix is available. Can you please validate it from your end and let us know if you face an issue.
Thanks
@dhruvesh09 it does work! Ty! Validated on Collab (ran Trainer, modified train.py and re-run)
Most helpful comment
@dhruvesh09 it does work! Ty! Validated on Collab (ran Trainer, modified train.py and re-run)