Core: Tensorflow not available in Hassio

[robmarkcole]

Tensorflow component, HA 0.82 on Hassio.

The docs state that tensorflow is included in Hassio, but it is not listed in the build logs nor requirements.txt. On following the setup process (here) and configuring the component, I do not get any cannot import tensorflow errors, but I get the following, indicating an import problem (as we expect):

Error while setting up platform tensorflow
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/homeassistant/helpers/entity_platform.py", line 128, in _async_setup_platform
    SLOW_SETUP_MAX_WAIT, loop=hass.loop)
  File "/usr/local/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/homeassistant/components/image_processing/tensorflow.py", line 127, in setup_platform
    detection_graph = tf.Graph()
AttributeError: module 'tensorflow' has no attribute 'Graph'

@hunterjm @dale3h @arsaboo

[frenck]

This is correct, we currently do not have the resources to actually add Tensorflow to Hass.io.
Secondly, it makes the Docker images for Hass.io grow tremendously.

We are currently looking into proper solutions for this.

Ref: home-assistant/architecture#103

[jmtatsch]

@pvizeli mentioned that tensorflow was removed and instead added to some kind of add-on.
I looked at official & and community addons and couldn't find anything.
Can someone please point me to that add-on or doesn't it exist anymore?

[OrangeTang16]

Is this still an issue you are experiencing? Can you please try upgrading to the latest version of Home Assistant (0.90) and report back if this is still a problem? Thanks!

[theastropath]

I still see the same error in 0.90

[bbrendon]

If you think you've installed tensorflow, you may have installed it outside of the home assistant python virtual environment which will cause this error.

[sameeraman]

I'm getting the same error. I'm running home assistant on docker. @bbrendon did you manage to get pass this error ?

[ivelin]

@frenck Have you considered including Tensorflow Light in hassio instead ? It’s a shrunk down TF for mobile and embedded devices. Still runs Pre-trained TF models.
https://www.tensorflow.org/lite/guide/get_started

I might have some spare cycles to help in this area if there is interest.

[pvizeli]

https://twitter.com/pvizeli/status/1141612000570740737

[ivelin]

OK, I'm back from summer break and will try to work on this. Has anyone started, yet?

Before the break, I contributed to Tensorflow to get a sense of the source code and project structures.

Tensorflow proper is about 100MB per hardware architecture and with all the active contributions in the community I expect this to grow. Comparatively, the Tensorflow Lite binary is about 300KB (!) clearly targeting mobile and embedded systems.

I just got HASS running fine on RPI4 using docker and quite a bit of manual steps from this helpful guide. It was a few days battle.

In the meanwhile hass.io Beta has been released for RPI4. Will try that next since this issue is specifically for TF Lite on hass.io.

Can someone please share a summary of the status and plan for TF Lite? Any dev guidelines and dev environment setup pointers would be appreciated to get things moving.

[robmarkcole]

HI @ivelin I think it is worth first evaluating whether performance on a pi will be satisfactory before investing too much effort. Generally we want inferences times of less than a second in order to be useful. If it takes 3-5 seconds to process an image I think people would be disappointed. As a baseline some benchmarking with mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite would be informative. In my side project I am using hardware acceleration to get decent performance with this model and it performs well for home automation. Cheers

[ivelin]

Great point, @robmarkcole . I will try that benchmark on my rpi4.

If you've already run this coco model on your rpis without Coral, please share your results.

Your coral-pi-rest-server project looks promising. Are you coordinating with @blakeblackshear who's working on Frigate or these are separate initiatives?

[robmarkcole]

I haven't run the model without hardware acceleration yet, that would require installing tensorflow on the pi :) Currently coral-pi-rest-server is a standalone project but I have limited its scope purposefully so that it could be a Hassio addon

[ivelin]

Interesting challenge. Google's official TF images for RPi3 are problematic. RPI4 is not officially supported yet. There are people who provide unofficial workarounds for TF 1.x, but its chasing a moving target. Google postponed fixes until TF 2.0 which is still in Beta. I don't expect users to have to deal with custom TF builds from source. I'll keep working on this a bit more to see if there is an elegant way out.

Once running, TF Lite inference is under 1sec for object detection on RPI4, but I'll have to cleanup and run more thorough tests. The bigger challenge is how to quickly and painlessly add TF Lite to a HASS environment when needed.

I can see why you are looking closer at Coral. Although Google's official install doesn't support RPI4 yet, it was a quick fix and took a few minutes to get the demos running.

[ivelin]

Update:
Here is what I get with TF Lite on RPI4:

Model name | Execution time | Model size
------------ | ------------- | -------------
mobilenet_v2_1.0_224_quant.tflite | 222ms | 3.6MB
coco_ssd_mobilenet_v1_1.0_quant_2018_06_29 | 353ms | 4.1MB
mobilenet_v1_1.0_224.tflite | 331ms | 16.9MB
mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite | Didn't find custom op for name 'edgetpu-custom-op' . Appears that its a model built for Coral that is not supported by the latest FOSS TF Lite package. | 6.9MB

I would like to get some feedback whether the above combination of performance and size are reasonable in order to consider including some them as default implementation of the HASS image processing component?

TF Lite itself installed is about 1.5M plus a model of 3-10MB in size, we are looking at approximately 15-20MB budget for 2 default models (maybe object detection and face recognition).

If the performance and binary size are acceptable, I think it could be helpful for new users to get some of the AI benefits out of the box. With more powerful alternatives available via add-ons as needed. I see quite a bit of discussion on the forums from folks trying to setup object and face detection on HASS.

Thoughts?

[robmarkcole]

The inference times look very good, certainly anything under 1s is acceptable. The models are small relative to full tensorflow, but I leave it to others to comment on if the size is acceptable to include in an image. I expect we will want to wait for tf2 official rather than chase a moving target or faff around with 1x.
In my experience people are interested in 3 classes for detections - people, faces and cars, so we will want to cover these. However faces are only useful if you can then recognise trained faces, and none of the models will do this out of the box, so maybe we add faces later.

[pvizeli]

It will be available with next release

[ivelin]

It will be available with next release

@pvizeli Could you please elaborate. What will be available with which next release?

@robmarkcole 's suggestion makes a lot of sense to me. For faces we'd have to include an easy way for users to add custom faces and train, which excludes TF Lite, since it only allows inference but not training. That leaves people and cars which can be probably addressed with one of the good pre-trained models for object detection focused on 5-10 categories/classes. Recommendations?

Since full TF (and other popular frameworks with regression capability) doesn't run well on RPI due to memory and computing constraints, it probably makes sense to look at trainable ML solutions that integrate with HASS as add-ons and remotely interface with a standalone GPU/TPU powered system.

[Priva28]

Does next release mean we will be able to use TF Lite on Hass.io? Does this mean 0.98 or the next 0.97.3? I have been trying, yesterday, to get Tensorflow working in the Hass.io Docker container in Ubuntu but have been unsuccessful. Do I give this up if it will be supported natively?

[Priva28]

Add packages version to Tensoflow for wheels packages

What does new wheels for Hass.io refer to?

Edit: looking at the changes in the code looks like TensorFlow is now a default requirement for the TF component. Hopefully this means it's preinstalled for Hass.io and we only need to add our model.

[robmarkcole]

@pvizeli is it possible to build wheels for tensorflow 2? If yes I am happy to update the tensorflow integration when this is ready.

CC @hunterjm

[pvizeli]

I have currently no time to work on it, but it's open for everyone: https://github.com/home-assistant/hassio-tensorflow

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.