Tfx: ModelValidator does not work with regression model

[Malonl]

I have created my own custom pipeline, using the taxi pipeline as basis to branch out from. So this pipeline contains everything from CsvExampleGen to the Pusher.
I am trying to build a DNNLinearCombinedRegressor model. This all works fine until the ModelValidator component. It tries to compare the accuracy of the blessed model with the current model, however since we are running a regressor model, there is no accuracy metric and so the component fails on that point and with that the rest of the pipeline.

This is the error message that pops up here:

[2019-05-22 10:01:52,613] {logging_mixin.py:95} INFO - [2019-05-22 10:01:52,613] {executor.py:51} INFO - TF Logging: current metric: ((), {u'label/mean': {'doubleValue': 5.047114372253418}, u'average_loss': {'doubleValue': 335.8445129394531}, u'post_export_metrics/example_count': {'doubleValue': 43299.0}, u'prediction/mean': {'doubleValue': 5.458231449127197}})
[2019-05-22 10:01:52,613] {models.py:1788} ERROR - 'accuracy'
Traceback (most recent call last):
File "/home/username/taxi_pipeline/local/lib/python2.7/site-packages/airflow/models.py", line 1657, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/username/taxi_pipeline/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 103, in execute
return_value = self.execute_callable()
File "/home/username/taxi_pipeline/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 108, in execute_callable
return self.python_callable(self.op_args, *self.op_kwargs)
File "/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/orchestration/airflow/airflow_adapter.py", line 150, in python_exec
executor.Do(self._input_dict, self._output_dict, self._exec_properties)
File "/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/components/model_validator/executor.py", line 164, in Do
blessed_model_dir=blessed_model_dir)
File "/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/components/model_validator/executor.py", line 108, in _generate_blessing_result
blessed_model_eval_result)):
File "/home/username/taxi_pipeline/local/lib/python2.7/site-packages/tfx/components/model_validator/executor.py", line 55, in _compare_eval_result
if (current_metric[1]['accuracy']['doubleValue'] <
KeyError: 'accuracy'

The code where this fails is in tfx/components/model_validator/executor.py in the _compare_eval_result definition:

  # TODO(jyzhao): customized validation support.
  def _compare_eval_result(self, current_model_eval_result,
                           blessed_model_eval_result):
    """Compare accuracy of all metrics and return true if current is better or equal."""
    for current_metric, blessed_metric in zip(
        current_model_eval_result.slicing_metrics,
        blessed_model_eval_result.slicing_metrics):
      # slicing_metric is a tuple, index 0 is slice, index 1 is its value.
      tf.logging.info("TF Logging: current metric: " + str(current_metric))
      tf.logging.info("TF Logging: current metric: " + str(blessed_metric))
      if current_metric[0] != blessed_metric[0]:
        raise RuntimeError('EvalResult not match {} vs {}.'.format(
            current_metric[0], blessed_metric[0]))
      if (current_metric[1]['accuracy']['doubleValue'] <
          blessed_metric[1]['accuracy']['doubleValue']):
        tf.logging.info(
            'Current model accuracy is worse than blessed model: {}'.format(
                current_metric[0]))
        return False
    return True

There seems to be no switch to check for accuracy when using a classification model and something else, like average_loss, when using a regressor model.
I did a workaround by just replacing accuracy with average_loss, but this is hardcoded in the component code, so not very desirable.

Is there something that I am missing here, and is there actually a switch or a place to add custom configuration for the model validator? Or is this something that is indeed not implemented?

I am running in Python 2.7 and TFX 0.13.0.

Thanks!

[Efaq]

Also experiencing that. I wonder if there is some possible configuration to be passed to the validator. There is a (not so well) documented input called "eval_spec" in a documentation page but this doesn't seem to be present in the actual code.

documentation page: https://www.tensorflow.org/tfx/guide/modelval

[1025KB]

Hi, currently it's not supported, will keep this as a feature request to support customized model validation

for now, you can change the _pass_threshold() and _compare_eval_result() to do customization

[Efaq]

@1025KB just to check: the "eval_spec" argument to ModelValidator is in the documentation anyways (https://www.tensorflow.org/tfx/guide/modelval).

Is this documentation wrong (the argument don't exist), or is it outdated (it existed in a previous version but not anymore), or is it referring to some other functionality not related to this issue?

[zhitaoli]

@Efaq sorry the documentation was incorrect. I'll work with @rcrowe-google to fix it.

We definitely plan to open up more configurations for model validator in short term future.

[Malonl]

Ok, thanks for all the answers.

We will keep our workaround for now.

[Efaq]

@zhitaoli Ok, thank you for these updates!