Core: Error saving html5_push_registrations.conf

[fsa317]

Make sure you are running the latest version of Home Assistant before reporting an issue.

You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:

Home Assistant release (hass --version):
0.58.0

Python release (python3 --version):
3.5.2

Component/platform:
HTML5 Push Notifications

Description of problem:
When I try to register my android device from chrome I see the following in the log:

2017-11-21 20:59:28 ERROR (MainThread) [homeassistant.util.json] Failed to serialize to JSON: /home/homeassistant/.homeassistant/html5_push_registrations.conf
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/util/json.py", line 38, in save_json
    data = json.dumps(config, sort_keys=True, indent=4)
  File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 200, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.5/json/encoder.py", line 429, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 436, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'pRNH7pFGQ18HbYmRK0motQ==' is not JSON serializable
2017-11-21 20:59:28 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/util/json.py", line 38, in save_json
    data = json.dumps(config, sort_keys=True, indent=4)
  File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.5/json/encoder.py", line 200, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.5/json/encoder.py", line 429, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 403, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.5/json/encoder.py", line 436, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'pRNH7pFGQ18HbYmRK0motQ==' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/aiohttp/web_protocol.py", line 416, in start
    resp = yield from self._request_handler(request)
  File "/srv/homeassistant/lib/python3.5/site-packages/aiohttp/web.py", line 323, in _handle
    resp = yield from handler(request)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/http/ban.py", line 58, in ban_middleware
    return (yield from handler(request))
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 209, in coro
    res = yield from res
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 209, in coro
    res = yield from res
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/http/__init__.py", line 430, in handle
    result = yield from result
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/notify/html5.py", line 176, in post
    if not save_json(self.json_path, self.registrations):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/util/json.py", line 45, in save_json
    raise HomeAssistantError(error)
homeassistant.exceptions.HomeAssistantError: b'pRNH7pFGQ18HbYmRK0motQ==' is not JSON serializable

In the html5 conf file I only see 1 entry from my desktop machine. The notifications to my phone work until I restart, then they stop working until I re-register. This seems to be related specifically to when using my android device to register.

Expected:
No errors, and html5_push_registrations.conf should have an entry for my mobile device.

Problem-relevant configuration.yaml entries and steps to reproduce:

1. Click the switch to turn on push notifications on my mobile device.
2. 
   
   
   3.
   

Traceback (if applicable):

Additional info:

[leranp]

I had a similar problem when my phone was unlink from the push message every couple of hours, i think it is related to the new google app updatea , after changing version the problem stopped

[fsa317]

After changing the version of what? HASS?

[leranp]

No, after changing the google app on Android

[fsa317]

I have the latest version and I get the error from the original post every time I try to subscribe.

[iago1460]

I got the same issue using hassio (v 0.58.1), I've registered two android phones successfully before getting this error when I was trying to subscribe from Chrome MacOS Version 62.0.3202.94 (Official Build) (64-bit)

Failed to serialize to JSON: /config/html5_push_registrations.conf
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/homeassistant/util/json.py", line 38, in save_json
    data = json.dumps(config, sort_keys=True, indent=4)
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/lib/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable

[fsa317]

Are you saying you got the error on the Android phones and things still worked? Or you didn't get the error on Android?

[iago1460]

Subscribing from Android was fine, the problem happened from desktop chrome, I've fixed my previous answer

[fsa317]

Mine was the opposite, my laptop's chrome was fine but Android throws that error. I think its some type of hash being generated and if you are unlucky you get one that causes the bug. Now, how do we get someone to fix it :-)

[fsa317]

I deleted the contents of html5.conf file and then re-registered the two devices in the opposite order and it worked.

[bratanon]

Still a problem.

Mine is also adding a new "unnamed device" with the exact same values as my phone that only has another name.

[parberge]

Had this problem. Removed html5_push_registrations.conf. No longer get the exception (is not JSON serializable) but something else is quite wrong; When I register one new device, I get atleast 3 new "unnamed device", with the exact same values (except for the name)

[dshokouhi]

One of my devices that was working fine had an issue today and I had to re-register it but I kept getting this error upon registering the device and had to make a few different attempts and restarts to get it to work. I wonder if it is related to this issue. I even had a case where the device was renamed.

[bergemalm]

If you have problem, does this solve it (without removing the file and start over)?

head -c -1 html5_push_registrations.conf > html5_push_registrations.conf_temp && mv html5_push_registrations.conf_temp html5_push_registrations.conf

(removes newline at end)
Restart ha.
Using for example vi editor adds a newline when setting a proper name for your device...

[parberge]

@bergemalm This is not solving the problem for me.

It's hard to debug, because I get strange behaviour;
Duplicate entries and sometimes stack traces.

For example I removed everything in html5_push_registrations.conf and restarted ha.
Got this in my home-assistant.log:

Traceback (most recent call last):
  File "/home/homeassistant/homeassistant/lib/python3.4/site-packages/homeassistant/util/json.py", line 19, in load_json
    return json.loads(fdesc.read())
  File "/usr/lib64/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

This is the content of the file:

{
    "unnamed device": {
        "browser": "chrome",
        "subscription": {
            "endpoint": "https://android.googleapis.com/gcm/send/xxx",
            "expirationTime": null,
            "keys": {
                "auth": "xxx",
                "p256dh": "xxx"
            }
        }
    }
}

[DoloresHA]

I'm getting similar errors, see below. I can send a notification to ALL enrolled devices, but if I send it to just one target, I get the error below. I noticed this because an automation that sends notifications to all devices worked, but another one that has a specific target didn't. I didn't get any error message, until I used the developer tools to try to send a notification.

One thing I've noticed- in the past I learned the hard way that device names were case dependent. My target name is something like "DoloresPhone".

Now that the services page looks different under developer tools (that is, now the service is notfiy.html5_doloresphone, when before it is notify.html5 and then I could choose a target in the next field), could this be causing issues, in particular with case-sensitive targets?

Error handling request
Traceback (most recent call last):
  File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/util/json.py", line 38, in save_json
    data = json.dumps(config, sort_keys=True, indent=4)
  File "/usr/local/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/local/lib/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/lib/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
  File "/usr/local/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable

[bergemalm]

No, you're right. It doesn't solve the problem. I can get it to work without exceptions for a while but each time you enter the configuration ui it adds another device which is a duplicate. After a while the exceptions return (which stops additional duplicate add-ons)...
Some refactoring is needed for this component.

[finnysamuel]

Same issue happens to me too.. Waiting for a fix

[thepapamaan]

Not sure if it is related to your solution @Chris-V but I did a little troubleshooting and found som issues in how the conf-file is handled:

• removed html5_push_registrations.conf
• reregistered 2 x android phones and 1 x W10 computer (Chrome) - success
• removing devices (by unregistering for push) seem to work in the browsers but config-file does not update properly - the first one i deregistered whas removed but not the next two.
• tried removing push again from 2 different devices, browser did not complain and shows the switch is off
• did a full refresh of the HA-frontend in Chrome on the devices - Push is still active and nothing is removed in the conf-file.
• html5_push_registrations.conf now has 8 registrations - "unamed device" to "unamed device_8"
• tried renaming the devices (from "unamed device" to a better identifier), saved file.
• opened config file again - changes remain
• registered a new device for push - the renames where reverted back to "unamned device.."

[Chris-V]

There seem to be more than one issue there!

Many scenarios where the file fails to update properly should be solved when #11437 is merged
The registrations getting duplicated as unnamed device is something else I noticed as well and should be taken care of in #11451

I have both fixes applied locally, so I'll do another round of testing to see if anything else feels wrong.

[Chris-V]

@thepapamaan

html5_push_registrations.conf is loaded at boot and the content kept in memory. Any changes such as renaming a device must be followed by a restart of home-assistant to be picked up. Some annoyance here: the frontend sends the registration object as soon as you visit the config panel -- thus overwriting any changes made to the file.

To fix things once and for all, we would need a way to uniquely identify the browser independently of the registration object or the "device name" in order really update past subscriptions. #10503 seems to explain that use case, so I suggest we move the discussion over there :)