Tfx: Cannot use Dict ExecutionParameter at `tfx>=0.28`

[jeongukjae]

System information

• Have I specified the code to reproduce the issue(Yes/No): Yes
• Environment in which the code is executed: Linux, MacOS, Interactive Notebook
• TensorFlow version (you are using): 2.4.1
• TFX Version: tfx>=0.28
• Python version: 3.7, 3.8

Describe the current behavior

In tfx>=0.28, when using ExecutionParameter with Dict type, I got the error like ValueError: Unexpected type <class 'dict'>.

Describe the expected behavior

If I run the same code in tfx==0.27, it works.

Standalone code to reproduce the issue

• tfx 0.28: https://colab.research.google.com/drive/1B22-pdZpxiRbZNaeT3Xxa3qCU-_P0B8M?usp=sharing
• tfx 0.27: https://colab.research.google.com/drive/1Np8Tl8eOfX5uo1Zxq7y9X51z-2qsSA25?usp=sharing

Other info / logs

https://github.com/tensorflow/tfx/blob/68637c1faa9c3d7a27e43d87ae85c1639ef1a32a/tfx/types/component_spec.py#L302-L307

As the above comments, it looks like using ExecutionParameter with Dict type should behave normally.

[chongkong]

Hi Jeonguk. There is a one strict assumption we have for component spec: all exec_properties should be serialized. Serialized exec_properties can be accessed from Executor.Do interface, and you have to manually deserialize your value, unless it is a primitive type. Allowed value types for exec_properties are only int, float, and str.

So being able to use Dict until 0.27 is rather a mistake, but this is not your fault as some components in TFX used to have Dict type exec_properties as well. We realized the mistake last year, and migrated existing Dict type exec_properties to a str type, and then added a serialization/deserialization steps. For example in Trainer there was custom_config exec_properties whose type was originally Dict[str, Any], but later changed to str in a2e040e2e3a8b62ceb2d9d0170188ff263d895c3. We recommend you to do the similar.

[jeongukjae]

Hi Jeonguk. There is a one strict assumption we have for component spec: all exec_properties should be serialized. Serialized exec_properties can be accessed from Executor.Do interface, and you have to manually deserialize your value, unless it is a primitive type. Allowed value types for exec_properties are only int, float, and str.

So being able to use Dict until 0.27 is rather a mistake, but this is not your fault as some components in TFX used to have Dict type exec_properties as well. We realized the mistake last year, and migrated existing Dict type exec_properties to a str type, and then added a serialization/deserialization steps. For example in Trainer there was custom_config exec_properties whose type was originally Dict[str, Any], but later changed to str in a2e040e. We recommend you to do the similar.

@chongkong Thanks. Then, are there any plans to support auto serialization/deserialization steps for exec_properties's value types that consist of python's primitive types and default collection types(e.g.,Dict[str, str], or List[str])? If those value types are type-checked well and automatically serialized/deserialized, I think it can reduce duplicated codes.

[chongkong]

It would definitely help, but I can't tell we will as we should also consider the backward compatibility and API consistency. Let me try to bring this issue to the internal discussions.