Tfx: TypeError: __init__() got an unexpected keyword argument 'dtype'

[tordsae]

Am running a native keras pipeline similar to the one in https://github.com/tensorflow/tfx/blob/master/tfx/examples/iris/iris_utils_native_keras.py. When I run the model I get the the error TypeError: __init__() got an unexpected keyword argument 'dtype' during the evaluation component.

The evaluation component is configured as:

    eval_config = tfma.EvalConfig(
        model_specs=[tfma.ModelSpec(label_key=features.LABEL_KEY)],
        slicing_specs=[tfma.SlicingSpec()],
        metrics_specs=[
            tfma.MetricsSpec(
                thresholds={
                    "mse": tfma.MetricThreshold(
                        value_threshold=tfma.GenericValueThreshold(
                            upper_bound={'value': 10000}
                        ),
                        change_threshold=tfma.GenericChangeThreshold(
                            direction=tfma.MetricDirection.LOWER_IS_BETTER,
                            absolute={'value': -1e-2}
                        )
                    )
                }
            )
        ],
    )

    model_analyzer = Evaluator(
        examples=example_gen.outputs['examples'],
        model=trainer.outputs['model'],
        eval_config=eval_config
    )

[1025KB]

Hi,

Is "mse" metric calculated in the model?
change_threshold is not necessary since you don't have a baseline model.

And just make sure you env can finish iris keras pipeline without problems, right?
Could you provide the detailed log?

[mdreves]

Can you provide some details on the model setup (particularly the values passed to model.compile).

[tordsae]

"mse" is calculated in the model.
Code for model:

def _build_keras_model(act_func="relu"):
    l = tf.keras.layers
    opt = tf.keras.optimizers
    inputs = [tf.keras.Input(shape=(1,), name=features.transformed_name(f)) for f in features.DATA_VALUES]
    if len(inputs) == 1:
        input_layer = inputs[0]
    else:
        input_layer = l.concatenate(inputs)
    d1 = l.Dense(16, activation='relu')(input_layer)
    d2 = l.Dense(8, activation=act_func)(d1)
    d3 = l.Dense(2, activation=act_func)(d2)
    d4 = l.Dense(8, activation=act_func)(d3)
    d5 = l.Dense(16, activation=act_func)(d4)
    output = l.Dense(len(inputs))(d5)
    model = tf.keras.Model(inputs=inputs, outputs=output)
    model.compile(
        loss='mse',
        optimizer=opt.Adam(lr=0.001),
        metrics=[tf.keras.metrics.MeanSquaredError(name="mse")])
    model.summary(print_fn=absl.logging.info)
    return model

During the training step an error also occurs:
2020-03-28 17:55:50.681184: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled

Log for the evaluator step:

2020-03-28 17:56:13.703917: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-28 17:56:13.704073: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-28 17:56:13.704095: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
INFO:absl:Using setup_file '/tfx-src/setup.py' to capture TFX dependencies
INFO:absl:Running driver for Evaluator
INFO:absl:MetadataStore with gRPC connection initialized
INFO:absl:Adding KFP pod name anomaly-test-c8dp5-3047051545 to execution
INFO:absl:Adding KFP pod name anomaly-test-c8dp5-3047051545 to execution
INFO:absl:Running executor for Evaluator
INFO:absl:Nonempty beam arg setup_file already includes dependency
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
  label_key: "y"
}
slicing_specs {
}
metrics_specs {
  thresholds {
    key: "mse"
    value {
      value_threshold {
        upper_bound {
          value: 10000.0
        }
      }
    }
  }
}
INFO:absl:Using gs://mlflow_testing/tfx_pipeline_output/anomaly_test/Trainer/model/1423/serving_model_dir as  model.
2020-03-28 17:56:20.284146: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-03-28 17:56:20.284195: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-03-28 17:56:20.284232: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (anomaly-test-c8dp5-3047051545): /proc/driver/nvidia/version does not exist
2020-03-28 17:56:20.284537: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-03-28 17:56:20.292162: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2020-03-28 17:56:20.292549: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7975d40 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-03-28 17:56:20.292681: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
INFO:absl:Evaluating model.
INFO:absl:Using 1 process(es) for Beam pipeline execution.
INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO:root:Setting socket default timeout to 60 seconds.
INFO:root:socket default timeout is 60.0 seconds.
INFO:root:Starting the size estimation of the input
INFO:oauth2client.transport:Attempting refresh to obtain initial access_token
INFO:root:Finished listing 2 files in 0.09192585945129395 seconds.
Traceback (most recent call last):
  File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 378, in <module>
    main()
  File "/tfx-src/tfx/orchestration/kubeflow/container_entrypoint.py", line 371, in main
    execution_info = launcher.launch()
  File "/tfx-src/tfx/orchestration/launcher/base_component_launcher.py", line 205, in launch
    execution_decision.exec_properties)
  File "/tfx-src/tfx/orchestration/launcher/in_process_component_launcher.py", line 67, in _run_executor
    executor.Do(input_dict, output_dict, exec_properties)
  File "/tfx-src/tfx/components/evaluator/executor.py", line 185, in Do
    slice_spec=slice_spec))
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pvalue.py", line 113, in __or__
    return self.pipeline.apply(ptransform, self)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 470, in apply
    label or transform.label)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 480, in apply
    return self.apply(transform, pvalueish)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 515, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 175, in apply
    return m(transform, input, options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 181, in apply_PTransform
    return transform.expand(input)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 829, in expand
    return self._fn(pcoll, *args, **kwargs)
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_model_analysis/api/model_eval_lib.py", line 934, in ExtractEvaluateAndWriteResults
    | 'WriteResults' >> WriteResults(writers=writers))
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pvalue.py", line 113, in __or__
    return self.pipeline.apply(ptransform, self)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 470, in apply
    label or transform.label)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 480, in apply
    return self.apply(transform, pvalueish)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 515, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 175, in apply
    return m(transform, input, options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 181, in apply_PTransform
    return transform.expand(input)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 829, in expand
    return self._fn(pcoll, *args, **kwargs)
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_model_analysis/api/model_eval_lib.py", line 676, in ExtractAndEvaluate
    update(evaluation, extracts | v.stage_name >> v.ptransform)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pvalue.py", line 113, in __or__
    return self.pipeline.apply(ptransform, self)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 470, in apply
    label or transform.label)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 480, in apply
    return self.apply(transform, pvalueish)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 515, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 175, in apply
    return m(transform, input, options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 181, in apply_PTransform
    return transform.expand(input)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 829, in expand
    return self._fn(pcoll, *args, **kwargs)
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py", line 661, in _EvaluateMetricsAndPlots
    plots_key=plots_key))
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pvalue.py", line 113, in __or__
    return self.pipeline.apply(ptransform, self)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 470, in apply
    label or transform.label)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 480, in apply
    return self.apply(transform, pvalueish)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/pipeline.py", line 515, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 175, in apply
    return m(transform, input, options)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 181, in apply_PTransform
    return transform.expand(input)
  File "/opt/venv/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 829, in expand
    return self._fn(pcoll, *args, **kwargs)
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py", line 516, in _ComputeMetricsAndPlots
    metric_specs.to_computations(metrics_specs, eval_config=eval_config)))
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_specs.py", line 486, in to_computations
    _deserialize_tf_metric(m, tf_metric_classes))
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_specs.py", line 707, in _deserialize_tf_metric
    return tf.keras.metrics.deserialize({'class_name': cls_name, 'config': cfg})
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py", line 3056, in deserialize
    printable_module_name='metric function')
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/utils/generic_utils.py", line 305, in deserialize_keras_object
    return cls.from_config(cls_config)
  File "/opt/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/losses.py", line 140, in from_config
    return cls(**config)
TypeError: __init__() got an unexpected keyword argument 'dtype'

[mdreves]

Found issue and will have a fix shortly. The root of the problem is that the loss that was added (MSE) and the metric that was added (MSE) both use the same classname. As a workaround you could can remove the metric since it is a duplicate of the loss calculation anyway (note the loss is also returned in the keras model.metrics output under the name 'loss' so you would need to use 'loss' for the threshold).