Core: Keyerror with Nederlandse Spoorwegen integration

[klaasnicolaas]

The problem

My sensors from the Dutch railways no longer want to load, I get an error message in my logs.

Environment

• Home Assistant release with the issue: 0.105
• Last working Home Assistant release (if known): 0.104.X
• Operating environment (Hass.io/Docker/Windows/etc.): Hassio
• Integration causing this issue: Nederlandse Spoorwegen
• Link to integration documentation on our website:
  https://www.home-assistant.io/integrations/nederlandse_spoorwegen/

Problem-relevant configuration.yaml

This is how my own yaml config looks like:

---
# Nederlandse Spoorwegen #
- platform: nederlandse_spoorwegen
  api_key: !secret ns_api_key
  routes:
    - name: Alkmaar - Leiden
      from: Amr
      to: Ledn
    - name: Leiden - Alkmaar
      from: Ledn
      to: Amr

Traceback/Error logs

The error that occurs in my log:

2020-02-09 13:02:12 ERROR (MainThread) [homeassistant.components.sensor] nederlandse_spoorwegen: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 461, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/nederlandse_spoorwegen/sensor.py", line 208, in update
    2,
  File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 932, in get_trips
    return self.parse_trips(raw_trips, requested_time)
  File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 843, in parse_trips
    newtrip = Trip(trip, requested_time)
  File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 606, in __init__
    trip_part = TripSubpart(part)
  File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 448, in __init__
    if part_dict['punctuality'] != 100.0:
KeyError: 'punctuality'

Additional information

[probot-home-assistant[bot]]

Hey there @YarmoM, mind taking a look at this issue as its been labeled with a integration (nederlandse_spoorwegen) you are listed as a codeowner for? Thanks!

[YarmoM]

Thanks for creating the issue. This is due to the new API made by NS that is seemingly a bit inconsistent. The issue is being discussed upstream here: https://github.com/aquatix/ns-api/issues/24 and should be solved relatively soon.

[MrUserY]

Just installed it an our ago. De integration worked for 5minutes correct, and then started shouting the following error (it's the same I see):

2020-02-09 17:55:13 ERROR (MainThread) [homeassistant.components.sensor] nederlandse_spoorwegen: Error on device update!
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_add_entity
await entity.async_device_update(warning=False)
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 461, in async_device_update
await self.hass.async_add_executor_job(self.update)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/usr/src/homeassistant/homeassistant/util/init.py", line 240, in wrapper
result = method(args, *kwargs)
File "/usr/src/homeassistant/homeassistant/components/nederlandse_spoorwegen/sensor.py", line 208, in update
2,
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 932, in get_trips
return self.parse_trips(raw_trips, requested_time)
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 843, in parse_trips
newtrip = Trip(trip, requested_time)
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 606, in init
trip_part = TripSubpart(part)
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 448, in init
if part_dict['punctuality'] != 100.0:
KeyError: 'punctuality

[fwestenberg]

Also an error here on the Stations list:

Error while setting up nederlandse_spoorwegen platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 159, in _async_setup_platform
await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
return fut.result()
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/nederlandse_spoorwegen/sensor.py", line 49, in setup_platform
stations = nsapi.get_stations()
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 951, in get_stations
return self.parse_stations(raw_stations)
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 939, in parse_stations
for station in obj['payload']:
KeyError: 'payload'

[neforce]

Got also this issue. NS integration worked for a few days when i started using HA,but since a month or so it stopped working;

`Logger: homeassistant.components.sensor
Source: components/nederlandse_spoorwegen/sensor.py:51
Integration: Sensor (documentation, issues)
First occurred: 13:34:54 (1 occurrences)
Last logged: 13:34:54

Error while setting up nederlandse_spoorwegen platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 179, in _async_setup_platform
await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
return fut.result()
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/nederlandse_spoorwegen/sensor.py", line 51, in setup_platform
stations = nsapi.get_stations()
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 951, in get_stations
return self.parse_stations(raw_stations)
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 937, in parse_stations
raise RequestParametersError('The request could not be handled by the API')
ns_api.RequestParametersError: The request could not be handled by the API`

[YarmoM]

As you can see in the error logs, this is a separate issue :) but let's solve it!

The error is quite generic ("Request could not be handled"), possibly hinting at there being an error in the configuration. Could you post your NS-specific configuration yaml (without API keys of course)?

[neforce]

As you can see in the error logs, this is a separate issue :) but let's solve it!

The error is quite generic ("Request could not be handled"), possibly hinting at there being an error in the configuration. Could you post your NS-specific configuration yaml (without API keys of course)?

Sure!

sensor:
  - platform: nederlandse_spoorwegen
    api_key: XXXXXXXXXXXXXXXXXXXXXXXXXXX
    routes:
      - name: Assen-Meppel
        from: Asn
        to: Mp

that's it actually :)

[YarmoM]

That looks good indeed. You said it stopped working about a month ago. That's around the time the NS made big API changes. Can you confirm you got the API key as described in the updated documentation?

To obtain an API key, create an account on the NS API-Portaal and obtain an API key for the Reisinformatie API which is part of the Ns-App product.

https://www.home-assistant.io/integrations/nederlandse_spoorwegen/

[neforce]

That looks good indeed. You said it stopped working about a month ago. That's around the time the NS made big API changes. Can you confirm you got the API key as described in the updated documentation?

To obtain an API key, create an account on the NS API-Portaal and obtain an API key for the Reisinformatie API which is part of the Ns-App product.

https://www.home-assistant.io/integrations/nederlandse_spoorwegen/

Yes, i folowed that updated version. I've got a API Key and i see in the history graph that the integration worked from 5 (when i started) til 9 february this year. With 1588 request to 'getTrips' and 12 to 'getAllStations'.

After 9 february it stopped working.

[YarmoM]

I am not sure what is happening here. I was able to take your configuration and get it working. The error is generated by the upstream library, not the integration itself. In fact, I made the PR implementing in (https://github.com/aquatix/ns-api/pull/26) and it was designed to be triggered when the NS responds without data, meaning the communication is fine but something in the request was incorrect. I checked your config and it is works. That only leaves the API key, but you say it's fine.

I can propose something a little unorthodox. NS gives two API keys and a very easy way to cycle them. I can generate a new one, test it on my system, then pass it to you via a secure channel, you test it and shortly after, I will regenerate a new key invalidating the one we exchanged.

[neforce]

I am not sure what is happening here. I was able to take your configuration and get it working. The error is generated by the upstream library, not the integration itself. In fact, I made the PR implementing in (aquatix/ns-api#26) and it was designed to be triggered when the NS responds without data, meaning the communication is fine but something in the request was incorrect. I checked your config and it is works. That only leaves the API key, but you say it's fine.

I can propose something a little unorthodox. NS gives two API keys and a very easy way to cycle them. I can generate a new one, test it on my system, then pass it to you via a secure channel, you test it and shortly after, I will regenerate a new key invalidating the one we exchanged.

Woah! According to your message i first tried regenerating a new api key, and... it works! Strange! But thanks for all!

[YarmoM]

Strange indeed but glad it worked out \o/

[fwestenberg]

Also an error here on the Stations list:

Error while setting up nederlandse_spoorwegen platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 159, in _async_setup_platform
await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
File "/usr/local/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
return fut.result()
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(self.args, *self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/nederlandse_spoorwegen/sensor.py", line 49, in setup_platform
stations = nsapi.get_stations()
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 951, in get_stations
return self.parse_stations(raw_stations)
File "/usr/local/lib/python3.7/site-packages/ns_api.py", line 939, in parse_stations
for station in obj['payload']:
KeyError: 'payload'

I just found out in my case I didn't subscribe for the right API product. Subscribing to the NS-App product fixed this for me, which is also clearly explained in the documentation.

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.