Core: Tensorflow errors on 0.114 with TF2

[droopanu]

The problem

Environment

• Home Assistant Core release with the issue: 0.114
• Last working Home Assistant Core release (if known): 0.113
• Operating environment (OS/Container/Supervised/Core): Core
• Integration causing this issue: Tensorflow
• Link to integration documentation on our website:

Problem-relevant configuration.yaml

• platform: tensorflow
  scan_interval: 10
  source:
  
  
  
  • entity_id: camera.reo2_image
    
    file_out:
  
  
  • "/data/tf/{{ camera_entity.split('.')[1] }}_latest.jpg"
  
  
  • "/data/tf/{{ camera_entity.split('.')[1] }}_{{ now().strftime('%Y%m%d_%H%M%S') }}.jpg"
    
    model:
    
    graph: /home/homeassistant/.homeassistant/tensorflow/models/efficientdet_d3_coco17_tpu-32/
  
  

Traceback/Error logs

2020-08-13 17:22:01 ERROR (MainThread) [homeassistant.helpers.entity] Update for image_processing.tensorflow_reo2_image fails
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity.py", line 263, in async_update_ha_state
    await self.async_device_update()
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/helpers/entity.py", line 454, in async_device_update
    await self.async_update()  # type: ignore
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/components/image_processing/__init__.py", line 132, in async_update
    await self.async_process_image(image.content)
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/components/image_processing/__init__.py", line 112, in async_process_image
    return await self.hass.async_add_job(self.process_image, image)
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/components/tensorflow/image_processing.py", line 361, in process_image
    detections = model(input_tensor)
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 611, in _call
    return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.AlreadyExistsError:  Resource __per_step_0/Postprocessor/Decode/ArithmeticOptimizer/AddOpsRewrite_add_1/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
         [[{{node Postprocessor/Decode/ArithmeticOptimizer/AddOpsRewrite_add_1/tmp_var}}]] [Op:__inference_detect_fn_90986]

Function call stack:
detect_fn

Additional information

[droopanu]

I have noticed that i do not get those errors with a different model:

efficientdet_d0_coco17_tpu-32

But the objects are still not being detected.

I also noted these errors on startup:

Logger: homeassistant
Source: tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py:163
First occurred: 17:30:23 (1 occurrences)
Last logged: 17:30:23

Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/components/tensorflow/image_processing.py", line 156, in tensorflow_hass_start
    model_config=pipeline_configs["model"], is_training=False
  File "/home/homeassistant/.homeassistant/tensorflow/object_detection/builders/model_builder.py", line 1062, in build
    add_summaries)
  File "/home/homeassistant/.homeassistant/tensorflow/object_detection/builders/model_builder.py", line 375, in _build_ssd_model
    is_training=is_training)
  File "/home/homeassistant/.homeassistant/tensorflow/object_detection/builders/model_builder.py", line 350, in _build_ssd_feature_extractor
    return feature_extractor_class(**kwargs)
  File "/home/homeassistant/.homeassistant/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 312, in __init__
    name=name)
  File "/home/homeassistant/.homeassistant/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 163, in __init__
    overrides={'rescale_input': False})
  File "/srv/homeassistant/lib/python3.7/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 494, in from_name
    model = cls(config=config, overrides=overrides)
  File "/srv/homeassistant/lib/python3.7/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 452, in __init__
    output = efficientnet(image_input, self.config)
  File "/srv/homeassistant/lib/python3.7/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 388, in efficientnet
    x = mb_conv_block(x, block, config, block_prefix)
  File "/srv/homeassistant/lib/python3.7/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 294, in mb_conv_block
    name=prefix + 'se_expand')
  File "/srv/homeassistant/lib/python3.7/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 193, in conv2d_block
    x = conv2d(**init_kwargs)(inputs)
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 960, in __call__
    self._set_inputs(cast_inputs, outputs)
  File "/usr/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/srv/homeassistant/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 418, in inner_cm
    for fn in self._scope_exit_callbacks:
TypeError: 'NoneType' object is not iterable

[probot-home-assistant[bot]]

tensorflow documentation
tensorflow source
_{^{(message by IssueLinks)}}

[dshokouhi]

Did you make sure to re-run the steps before pulling in the new models? This integration did have a breaking change in 0.114. Also double check the permissions for all of the tensorflow stuff as well.

[droopanu]

Yes, I did all those steps again, and I have checked all the permissions.
I also noticed that when using "efficientdet_d3_coco17_tpu-32" I get lots of errors and then hass crashes after a few minutes.
Using "efficientdet_d0_coco17_tpu-32" it does not crash, but it also does not recognize anything that it used to.

[electron64]

Hello, i had the same issue,

• Just rollback to previous version ( i hope you did a backup before updating).
• Remove the lines for Tensorflow in your "configuration.yaml" file.
• Remove the folder Tensorflow in your "config" directory.
• Restart HA.
• Update to the last version.
• follow the documentation

Et voila!
It should run properly.

[benroscoe]

I also had the same issue of timing but only resolved it by moving to the "d0" model. After that point it doesn't time out and does detect objects correctly.
Honestly I'm not sure what level of resolution I had on my object detection before, as it was usually just one option for a model.

[dshokouhi]

I get the same error as https://github.com/home-assistant/core/issues/38822#issuecomment-673291829 however tensorflow is working for me. Make sure you test out each model for accuracy and time

[CMDR-Sloma]

TF2 is unusable with 0.114.

1. Either I get this error all the time with zero detection:

Error doing job: Future exception was never retrieved

with:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/tensorflow/image_processing.py", line 155, in tensorflow_hass_start
detection_model = model_builder.build(
File "/config/tensorflow/object_detection/builders/model_builder.py", line 1015, in build
return build_func(getattr(model_config, meta_architecture), is_training,
File "/config/tensorflow/object_detection/builders/model_builder.py", line 372, in _build_ssd_model
feature_extractor = _build_ssd_feature_extractor(
File "/config/tensorflow/object_detection/builders/model_builder.py", line 350, in _build_ssd_feature_extractor
return feature_extractor_class(kwargs)
File "/config/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 554, in __init__
super(SSDEfficientNetB3BiFPNKerasFeatureExtractor, self).__init__(
File "/config/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 161, in __init__
efficientnet_base = efficientnet_model.EfficientNet.from_name(
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 494, in from_name
model = cls(config=config, overrides=overrides)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 452, in __init__
output = efficientnet(image_input, self.config)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 388, in efficientnet
x = mb_conv_block(x, block, config, block_prefix)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 256, in mb_conv_block
x = conv2d_block(x,
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 197, in conv2d_block
x = batch_norm(axis=bn_axis,
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in __call__
outputs = call_fn(cast_inputs, *args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/layers/normalization.py", line 741, in call
outputs = self._fused_batch_norm(inputs, training=training)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/layers/normalization.py", line 632, in _fused_batch_norm
self.add_update(mean_update)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1459, in add_update
updates = [process_update(x) for x in updates]
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1459, in
updates = [process_update(x) for x in updates]
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1455, in process_update
reachable = tf_utils.get_reachable_from_inputs(relevant_inputs, [update])
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 138, in get_reachable_from_inputs
outputs = x.consumers()
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 674, in consumers
return [
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 675, in
self.graph._get_operation_by_name_unsafe(name)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3663, in _get_operation_by_name_unsafe
return self._nodes_by_name[name]
KeyError: 'stack_1/block_0/se_reduce_activation/Sigmoid'

Or

`Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/tensorflow/image_processing.py", line 155, in tensorflow_hass_start
detection_model = model_builder.build(
File "/config/tensorflow/object_detection/builders/model_builder.py", line 1015, in build
return build_func(getattr(model_config, meta_architecture), is_training,
File "/config/tensorflow/object_detection/builders/model_builder.py", line 372, in _build_ssd_model
feature_extractor = _build_ssd_feature_extractor(
File "/config/tensorflow/object_detection/builders/model_builder.py", line 350, in _build_ssd_feature_extractor
return feature_extractor_class(kwargs)
File "/config/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 294, in __init__
super(SSDEfficientNetB0BiFPNKerasFeatureExtractor, self).__init__(
File "/config/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 161, in __init__
efficientnet_base = efficientnet_model.EfficientNet.from_name(
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 494, in from_name
model = cls(config=config, overrides=overrides)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 452, in __init__
output = efficientnet(image_input, self.config)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 364, in efficientnet
x = conv2d_block(x,
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 197, in conv2d_block
x = batch_norm(axis=bn_axis,
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 922, in __call__
outputs = call_fn(cast_inputs, *args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/layers/normalization.py", line 741, in call
outputs = self._fused_batch_norm(inputs, training=training)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/layers/normalization.py", line 632, in _fused_batch_norm
self.add_update(mean_update)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1459, in add_update
updates = [process_update(x) for x in updates]
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1459, in
updates = [process_update(x) for x in updates]
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1455, in process_update
reachable = tf_utils.get_reachable_from_inputs(relevant_inputs, [update])
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 138, in get_reachable_from_inputs
outputs = x.consumers()
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 674, in consumers
return [
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 675, in
self.graph._get_operation_by_name_unsafe(name)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3663, in _get_operation_by_name_unsafe
return self._nodes_by_name[name]
KeyError: 'stem_activation/mul'

Or

Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/tensorflow/image_processing.py", line 155, in tensorflow_hass_start
detection_model = model_builder.build(
File "/config/tensorflow/object_detection/builders/model_builder.py", line 1015, in build
return build_func(getattr(model_config, meta_architecture), is_training,
File "/config/tensorflow/object_detection/builders/model_builder.py", line 372, in _build_ssd_model
feature_extractor = _build_ssd_feature_extractor(
File "/config/tensorflow/object_detection/builders/model_builder.py", line 350, in _build_ssd_feature_extractor
return feature_extractor_class(kwargs)
File "/config/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 294, in __init__
super(SSDEfficientNetB0BiFPNKerasFeatureExtractor, self).__init__(
File "/config/tensorflow/object_detection/models/ssd_efficientnet_bifpn_feature_extractor.py", line 161, in __init__
efficientnet_base = efficientnet_model.EfficientNet.from_name(
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 494, in from_name
model = cls(config=config, overrides=overrides)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 452, in __init__
output = efficientnet(image_input, self.config)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 388, in efficientnet
x = mb_conv_block(x, block, config, block_prefix)
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 256, in mb_conv_block
x = conv2d_block(x,
File "/usr/local/lib/python3.8/site-packages/official/vision/image_classification/efficientnet/efficientnet_model.py", line 193, in conv2d_block
x = conv2d(*init_kwargs)(inputs)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 960, in __call__
self._set_inputs(cast_inputs, outputs)
File "/usr/local/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "/usr/local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py", line 418, in inner_cm
for fn in self._scope_exit_callbacks:
TypeError: 'NoneType' object is not iterable

1. Or if detection works it is very patchy, positive scans as rare as a few per day (even if there is a lot of activity in front of the cameras).

2. I have tried different efficientdet models already and d0 is the only one which works sometimes but not as good as before. With TF1.0 I didn't use efficientdet models but ''faster_rcnn_inception'' which had spot on detection efficiency. I can't use v2 of the above because Hassio crashes and system reboots as soon as the detection is triggered.

My Hassio runs on Intel Nuc through Proxmox and I followed steps below to get it working on 0.114:

Uninstall the addon, delete the deps and tensorflow folders in config, run the git command, and find a new model (I recommend one of the EfficientDet models).

[Mark612]

TF2 is unusable with 0.114.

My Hassio runs on Intel Nuc through Proxmox and I followed steps below to get it working on 0.114:

I'm also running on the NUC via docker, and having problems as well. HA continues to fault and restart.

[TestPilot23]

Has anyone had any luck with getting this working again?

[electron64]

Has anyone had any luck with getting this working again?

I also gave up on this integration since update to Tf2.
It does not detect efficiently..
I use Doods now and it works perfectly.

https://www.home-assistant.io/integrations/doods/