Hub: Multilingual-USE-3 for text_embedding_column

[tangwiki]

I am trying to use https://tfhub.dev/google/universal-sentence-encoder-multilingual/3 as a pre-trained text embedding and use hub.text_embedding_column for TF-Estimators. I have

tensorflow==2.1.0
tensorflow-estimator==2.1.0
tensorflow-hub==0.7.0
tensorflow-text==2.1.1 

I tried several options below, but none works for me so far. Could you please let me know a right direction to pursue?

1. If I use

                embedding_feature_column = hub.text_embedding_column(
                     key='text',
                    module_spec="https://tfhub.dev/google/universal-sentence-encoder-multilingual/3",
                    trainable=False
                )

I get
RuntimeError: Missing implementation that supports: loader(*('/usr/modeling/lib/tfhub/26c892ffbc8d7b032f5a95f316e2841ed4f1608c',), **{}), where /usr/modeling/lib/tfhub is the directory that the savedModel is downloaded. Under the directory, the sizes of the file and sub-directories are

4.0K    assets
7.9M    saved_model.pb
980M    variables

I tried one of the previous solutions of deleting the directory and re-running, the same error occurred. Also, the same error occurred if I use
module_spec=hub.load_module_spec("https://tfhub.dev/google/universal-sentence-encoder-multilingual/3")

1. If I use

my_module_spec=hub.create_module_spec_from_saved_model(tfhub_cache_path+"/26c892ffbc8d7b032f5a95f316e2841ed4f1608c/"),
embedding_feature_column = hub.text_embedding_column(
                    key='text',
                    module_spec= my_module_spec,
                    trainable=False
                )

I get the error of
File "/usr/local/lib/python3.6/dist-packages/tensorflow_hub/feature_column.py", line 108, in text_embedding_column key=key, module_spec_path=module_spec, trainable=trainable) File "/usr/local/lib/python3.6/dist-packages/tensorflow_hub/feature_column.py", line 158, in __init__ self.module_spec = module.as_module_spec(self.module_spec_path) File "/usr/local/lib/python3.6/dist-packages/tensorflow_hub/module.py", line 38, in as_module_spec raise ValueError("Unknown module spec type: %r" % type(spec)) ValueError: Unknown module spec type: <class 'tuple'>

1. If I use

    def build_module_fn(saved_model_path=tfhub_cache_path+"/26c892ffbc8d7b032f5a95f316e2841ed4f1608c"):
        def module_fn():
            text_input = tf.compat.v1.placeholder(dtype=tf.string, shape=[None])
            embed_layer = hub.KerasLayer(
                saved_model_path,
                input_shape=[],  # Expects a tensor of shape [batch_size] as input.
                dtype=tf.string,  # Expects a tf.string input tensor
                signature="serving_default",
                output_key="outputs")
            embeddings = embed_layer(text_input)
            inputs = dict(default=text_input)
            hub.add_signature(inputs=inputs, outputs=embeddings)
        return module_fn

and

my_module_spec = hub.create_module_spec(build_module_fn())
 embedding_feature_column = hub.text_embedding_column(
                     key='text',
                    module_spec=my_module_spec,
                    trainable=False
                )

I get the following error of Failed precondition: Table not initialized.
2020-02-10 01:22:45.464583: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at lookup_table_op.cc:809 : Failed precondition: Table not initialized. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call return fn(*args) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn target_list, run_metadata) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.FailedPreconditionError: {{function_node __inference_signature_wrapper_1819}} {{function_node __inference_signature_wrapper_1819}} [_Derived_]{{function_node __inference_pruned_1634}} {{function_node __inference_pruned_1634}} Error while reading resource variable module_1/EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_21 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/module_1/EncoderDNN/DNN/ResidualHidden_3/dense/kernel/part_21/N10tensorflow3VarE does not exist. [[{{node EncoderDNN/DNN/ResidualHidden_3/dense/kernel/ConcatPartitions/concat/ReadVariableOp_21}}]] [[StatefulPartitionedCall]] [[dnn/input_from_feature_columns/input_layer/utt_hub_module_embedding_1/module_1_apply_default/keras_layer/StatefulPartitionedCall]]

[tangwiki]

@vbardiovskyg @j-grimstad @gowthamkpr any update on this issue? Is there any timeline for the fix? Thanks.

[vbardiovskyg]

Hi, with the tensorflow-hub=0.8.0 release a new implementation of feature columns (https://github.com/tensorflow/hub/blob/master/tensorflow_hub/feature_column_v2.py#L57) is available that applies TF2.0 Saved Models.

This is dependent on changes in TensorFlow core, so until TensorFlow makes a new release (most likely tensorflow==2.2.0), it will need tf-nightly to work.

[tangwiki]

@vbardiovskyg Thanks so much for addressing the issue. When will be the release dates for tensorflow-hub=0.8.0 and tensorflow==2.2.0, respectively?

[vbardiovskyg]

Hi @tangwiki, the tensorflow-hub==0.8.0 is already out.

I've just checked the tensorflow package release cycle, so let me take back the previous statement. It seems actually unlikely that the dependent feature will get into 2.2.0. This means that until 2.3.0 is out, the only way to use this will be via tf-nightly. I will update this issue and https://github.com/tensorflow/hub/issues/420 when the feature is in TF stable.

[andresusanopinto]

Tensorflow 2.3 has now a release candidate and I confirmed that it allows hub.text_embedding_column_v2 to be used.

I am closing this now, though there is likely some more days until "pip install tensorflow==2.3" can be used. Though immediatly one can use "pip install tensorflow==2.3.0rc2".