Tfx: RuntimeError: Cannot find matching execution with pipeline name chicago_taxi_beam

[ed-alertedh]

I'm trying to get the Chicago taxi example running on top of spark using a docker-compose testing environment. It gets as far as running the Evaluator on the trained model when I get the error below. I have tried with both sqlite metadata store and mysql and get the same error. I made sure that all of the containers are running beam 2.21.0 on python 3.7 and have pinned all these packages at the version combinations you have tested:

    - tfx==0.22.0
    - apache_beam[gcp]==2.21.0
    - ml-metadata==0.22.0
    - pyarrow==0.16.0
    - tensorflow==2.2.0
    - tensorflow-data-validation==0.22.0
    - tensorflow-metadata==0.22.0
    - tensorflow-model-analysis==0.22.1
    - tensorflow-transform==0.22.0
    - tfx-bsl==0.22.0
    - google-api-core==1.16.0

I'm happy to collect more info to help narrow down the problem, apologies if this issue is more applicable to ml-metadata.

Traceback (most recent call last):  File "export_chicago_taxi_beam.py", line 293, in <module>    BeamDagRunner().run(tfx_pipeline)    absl.logging.info('Component %s is scheduled.', component_id)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/pipeline.py", line 524, in __exit__    self.run().wait_until_finish()  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/pipeline.py", line 497, in run    self._options).run(False)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/pipeline.py", line 510, in run    return self.runner.run_pipeline(self, self._options)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 130, in run_pipeline    return runner.run_pipeline(pipeline, options)
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 179, in run_pipeline    pipeline.to_runner_api(default_environment=self._default_environment))
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 189, in run_via_runner_api    return self.run_stages(stage_context, stages)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 335, in run_stages    bundle_context_manager,
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 545, in _run_stage    expected_timer_output)
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1051, in process_bundle
    for result, split_result in executor.map(execute, part_inputs):  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
    yield fs.pop().result()
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/utils/thread_pool_executor.py", line 44, in run
    self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1048, in execute
    part_map, expected_outputs, fired_timers, expected_output_timers)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 947, in process_bundle
    result_future = self._worker_handler.control_conn.push(process_bundle_req)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 349, in push
    response = self.worker.do_instruction(request)
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 471, in do_instruction
    getattr(request, request_type), request.instruction_id)
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 506, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 972, in process_bundle
    element.data)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py", line 218, in process_encoded    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 330, in apache_beam.runners.worker.operations.Operation.output  File "apache_beam/runners/worker/operations.py", line 332, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 195, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 670, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 671, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 963, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1045, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback    raise exc.with_traceback(traceback)  File "apache_beam/runners/common.py", line 961, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 726, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 814, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/tfx/orchestration/beam/beam_dag_runner.py", line 84, in process
    self._run_component()
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/tfx/orchestration/beam/beam_dag_runner.py", line 88, in _run_component
    self._component_launcher.launch()
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 197, in launch
    self._exec_properties)
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/tfx/orchestration/launcher/base_component_launcher.py", line 166, in _run_driver
    component_info=self._component_info)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/tfx/components/base/base_driver.py", line 239, in pre_execution    driver_args, pipeline_info)
  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/tfx/components/base/base_driver.py", line 150, in resolve_input_artifacts    producer_component_id=input_channel.producer_component_id)  File "/home/ed/miniconda3/envs/terra15_tfx/lib/python3.7/site-packages/tfx/orchestration/metadata.py", line 920, in search_artifacts    producer_component_id))
RuntimeError: Cannot find matching execution with pipeline name chicago_taxi_beam,run id 2020-06-26T13:53:31.958980 and component id None [while running 'Run[Evaluator]']

[drzl386]

Hey there, did we have any solution for this error? I tried the new version 0.22.1, also had this error.

[AlexandrePieroux]

Any update on this ? I'm facing the same issue. I'm also in the same configuration and get the same error:

RuntimeError: Cannot find matching execution with pipeline name chicago_taxi_airflow,run id manual__2020-08-05T17:38:04.451224+00:00 and component id None

[AlexandrePieroux]

Ok I've resolved my error. There is a missing component in the pipeline: model_resolver.

So the corresponding cell in the notebook should be:

components = [ example_gen, statistics_gen, schema_gen, example_validator, transform, trainer, model_resolver, evaluator, pusher ]

[dbustosp]

I had the same issue and the answer that @AlexandrePieroux provided is correct. This issue is triggered when a missing component in your Pipeline. Make sure you defined all the Components you are using in your pipeline. Will be great if you provide more information how your pipeline is defined.

[tensorflow-butler[bot]]

Are you satisfied with the resolution of your issue?
Yes
No