Core: Error configuring Luftdaten.info Sensor ID via Frontend

[hudecitydave]

Home Assistant release with the issue:
0.84.6

Last working Home Assistant release (if known):
0.83.x

Operating environment (Hass.io/Docker/Windows/etc.):
hass.io on Raspi3

Component/platform:
Luftdaten: https://www.home-assistant.io/components/luftdaten/

Description of problem:
I had some problems showing my Luftdaten.info sensors. After deleting them on the frontend I am not able to re-enter my values.
When I try to add an sensor I always get the message (German hass.io):

"Keine Kommunikation mit Lufdaten API möglich" (Luf-t-daten is misspelled here)

The message is shown VERY fast, so I think there is no "real" try to connect... In my opinion it is too fast for every "DNS or something" timeout...

The Logfiles says:
ERROR (MainThread) [luftdaten] Can not load data from luftdaten.info

I am sure my sensors and the Luftdaten API is working, I can see all data via Browser.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

YAML Config is disabled - using frontend only!

Traceback (if applicable):

Additional information:
I know about problem #19591 and I have the same problem. With both issues Luftdaten in unusable for me in 0.84.6 now.

[fabaff]

Adding, deleting and re-adding a sensor works fine on dev.

"Keine Kommunikation mit Lufdaten API möglich" (Luf-t-daten is misspelled here)

Is a typo in the translation which was fixed.

[jensihnow]

Same issue here running on 0.84.6. Once I noticed a bunch of Luftdaten entries under _Integrations_. I deleted them and removed the config, restarted and unable to add the Luftdaten sensor. If configured using configuration.yaml it just throws errors like this one:

2018-12-29 17:21:29 ERROR (MainThread) [luftdaten] Can not load data from luftdaten.info

[jensihnow]

Found some _old_ entires in _config/.storage/core.entity_registry_

{
"config_entry_id": null,
"device_id": null,
"disabled_by": null,
"entity_id": "sensor.luftdaten_16193_p1",
"name": null,
"platform": "luftdaten",
"unique_id": "16193_P1"
},
{
"config_entry_id": null,
"device_id": null,
"disabled_by": null,
"entity_id": "sensor.luftdaten_16193_p2",
"name": null,
"platform": "luftdaten",
"unique_id": "16193_P2"
},
{
"config_entry_id": null,
"device_id": null,
"disabled_by": null,
"entity_id": "sensor.luftdaten_11217_p2",
"name": null,
"platform": "luftdaten",
"unique_id": "11217_P2"
},
{
"config_entry_id": null,
"device_id": null,
"disabled_by": null,
"entity_id": "sensor.luftdaten_11217_p1",
"name": null,
"platform": "luftdaten",
"unique_id": "11217_P1"
},

Removing and restarting didn't helped, still having the same issue: not able to configure or use it.

Tracing the call using the the browser I get this response:

{"data_schema": [{"name": "sensor_id", "required": true, "type": "string"}, {"default": false, "name": "show_on_map", "optional": true, "type": "boolean"}], "description_placeholders": null, "errors": {"sensor_id": "communication_error"}, "flow_id": "2eae859c40af4cd69490e213118feca6", "handler": "luftdaten", "step_id": "user", "type": "form"}

[jensihnow]

One more piece after putting logging into _debug_:

2018-12-29 18:31:20 INFO (MainThread) [homeassistant.components.http.view] Serving /api/config/config_entries/flow/2450d237a5434b6fae0f118e151574cc to 192.168.28.104 (auth: True)
2018-12-29 18:31:20 DEBUG (MainThread) [luftdaten] Response from luftdaten.info: 403
2018-12-29 18:31:20 ERROR (MainThread) [luftdaten] Can not load data from luftdaten.info

Now the question is why do we get a 403... whereas _curl https://api.luftdaten.info/v1/sensor/11217/_ returns full json.

[hudecitydave]

While testing I now saw the translation of the config-header is away, too.

I tried to see the communication to luftdaten.info, but wireshark was quiet. I can see every try to connect via curl or browser, but nothing while config with frontend.

Update:
It is not translated:
strings.json:
"title": "Define Luftdaten",

[jensihnow]

Hi,

are you running HA using hass.io (this is what I do)?
I tried configuration on a fresh Docker Container on my local computer an it works like @fabaff pointed out. But on my live system using HassOS it doesn’t or it is related to existing data/configuration.

J.

[hudecitydave]

@jensihnow yes, it is hass.io, based on a Raspi3. For me everything went fine, until I got problems to see values from the sensors in lovelace after upgrading to 0.84.6. (I randomly got erros "entity not available..."). While testing, I deleted the (UI configured) sensor via UI. After that now there is no way to re-add sonsors.

A very strange effect is: I have copied the luftdaten component dir from 0.83 to my custom_components, and the .py to the custom_components/sensors dir - but even with this custom-rollback the error appears.
(Checked it: I am fine with a "new, blank" test instance, too)

[hudecitydave]

@fabaff can you tell me, where the URL of the luftdaten api is stored? I tried to find it in the code, but failed (sorry).

For test, I activated an old yaml config, and put an old 0.82.x version of luftdaten.py to the custom_components - But also here I get the message

2018-12-30 18:23:32 ERROR (MainThread) [custom_components.sensor.luftdaten] Unable to retrieve data from luftdaten.info

maby something kills the url (but why?) ?

[mattsches]

@hudecitydave @jensihnow I've also started a fresh Docker container on my NAS running 0.84.6, but still get ERROR (MainThread) [luftdaten] Can not load data from luftdaten.info … but I can request the data using curl on my computer.

Update: I've started a Docker container using the dev image (0.85.0.dev0), and now everything is working as expected. So it seems to be an issue with 0.84.6 that should be resolved in the next version.

[mattsches]

@fabaff can you tell me, where the URL of the luftdaten api is stored? I tried to find it in the code, but failed (sorry).

It's in the external library that is used by the component: https://github.com/fabaff/python-luftdaten/blob/master/luftdaten/__init__.py#L15

[hudecitydave]

@mattsches Ah, thanks! Now I understand the whole chain. I think I can wait for 0.85. I cannot find a hook to this error for the moment, and with a fix in 0.85 there is no need to go deeper. :-)

For me this was another chance to get deeper into the ha universe... thanks, great community!

[jvanderneutstulen]

It looks like the webserver is blocking specific HA version strings (0.84.3 through 0.84.6)

$ telnet api.luftdaten.info 80
Trying 81.169.180.11...
Connected to api.luftdaten.info.
Escape character is '^]'.
GET / HTTP/1.0
User-Agent: HomeAssistant/0.84.6
Host: api.luftdaten.info

HTTP/1.1 403 Forbidden

$ telnet api.luftdaten.info 80`
Trying 81.169.180.11...
Connected to api.luftdaten.info.
Escape character is '^]'.
GET / HTTP/1.0
User-Agent: HomeAssistant/0.84.7
Host: api.luftdaten.info

HTTP/1.1 302 Found

[jensihnow]

Interesting catch and I just sent an email to Jan from the luftdaten.info project to see if there are any specific reasons for this.

[hudecitydave]

@jensihnow @jvanderneutstulen After all this updates, I think I reopen this issue to see where we are endig up.

[UniXSouL]

Interesting catch and I just sent an email to Jan from the luftdaten.info project to see if there are any specific reasons for this.

Was there any response from their side ? Are there any visible work-arounds ?

[jensihnow]

Was there any response from their side ?

Not yet, continue to wait and will follow up if they don't get back in 7 days.

Are there any visible work-arounds ?

Question is if they are just going to block the next version of HA as they see high number of requests for this User-Agent. They actually should block by IP and user-agent, unless there is a clear issue on the implementation.

[jensihnow]

One more...

Home Assistant is actually documented as a PlugIn on the luftdaten.info page, see https://luftdaten.info/faq/#toggle-id-8

[jvanderneutstulen]

I got the following response :

die neueren Versionen von HomeAssistant fragen über das Luftdaten Plugin zum Teil den gleichen Sensor bis zu 20 Mal in einer Sekunde ab.
So basically newer versions of HA request hammer their with servers with too many requests (up to 20 per second).

As test/workaround I changed the luftdaten api url in the library and pass it through a proxy which alters the User-Agent. In the last couple of days I see a normal request rate at 1 request every 10 minutes, which is the default for the integration. (running HA 0.84.6)

[jensihnow]

These are NOT individual offenders. Its a known bug oft the Luftdaten plugin used with newer versions oft HomeAssistant. The plugin is creating new sensor instances on every new start of the HomeAssistant server.
So its an offensive software version we are blocking.

@fabaff You are aware of the bug and/or fix?

[UniXSouL]

Today I've upgraded to the latest version of Home-assistant 0.85 on my hass.io instance. The Luftdaten integration now seems to work correctly after the upgrade. Looking forward for the Lovelace UI to become standard in 0.86, it is awesome :)

[hudecitydave]

Same here. Works as charme with 0.85

[DanielWe2]

It seems as they have blacklisted 0.85 too.. Probably caused by the issue in #18838 :(

The manual configured one worked so well for months :(

[khambrecht]

Can confirm this for 0.85.0:

$  curl -i  -H "User-Agent: HomeAssistant/0.85.0" https://api.luftdaten.info/v1/sensor/11217/
HTTP/1.1 403 Forbidden
Date: Wed, 16 Jan 2019 12:38:18 GMT
Server: Apache/2
Content-Length: 225
Content-Type: text/html; charset=iso-8859-1

However, 0.85.1 does work:

$ curl -i  -H "User-Agent: HomeAssistant/0.85.1" https://api.luftdaten.info/v1/sensor/11217/
HTTP/1.1 200 OK
Date: Wed, 16 Jan 2019 12:38:53 GMT
Server: Apache/2
Vary: Accept-Encoding
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
Transfer-Encoding: chunked
Content-Type: application/json

But I would expect it's just a matter of time to have this one also blacklisted unless the luftdaten component is fixed.

[hudecitydave]

Not good - see also this topic: https://github.com/home-assistant/home-assistant/issues/19981