Core: HTML5 push notifications - not received while phone is unused

[andriej]

Home Assistant release with the issue: 0.90

Last working Home Assistant release (if known): hard to say

Operating environment (Hass.io/Docker/Windows/etc.): Linux / venv / Python 3.7.2

Component/platform: html5 notify

Description of problem:
Since few versions (but it was hard to notice) HTML5 push notifications are not received by android phones while the phones are laying unused.
Push notifications work properly when testing with active screen or few minutes after it. If the screen is off (idle) for longer period, the notification never reaches the phone - not even when the phone is unlocked/active again.

On one of the groups we've managed to confirm problem on different phone brands, all with latest android 9 system. Also all of us run python 3.7.2.

Someone has also noticed that it may be because of some undocumented (in FCM Firebase) priority type, which defined, may change the behaviour and 'force' phone to show notification even when phone is not in use.

There's no issue with other notifications in meanwhile (communicators like whatsapp/messenger etc.)

Could person responsible for the html5 (@robbiet480 ?) could take a closer ook at this issue?

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant): not relevant

Traceback (if applicable): no error messages

Additional information:
What's worth nothing it's not the matter of phone power-save features - we've tried different settings including turning them off.

[bieniu]

Same issue here. OnePlus 6 (Android 9) with turned off all possible battery optimisation services. HA 0.89/0.90, Linux, venv, Python 3.7.2.

[robbiet480]

@andriej I'm not gonna be able to help out with this as I only use Chrome for HTML5 Push Notifications.

[pergolafabio]

yes , happens to me also occasionally, also oneplus 6T user here
when i turn on the phone, i see all notifications coming
i think the "chromebrowser" app is closed after a while , maybe?

[dshokouhi]

I think sometimes it may not be that your phone dozes but that googles servers are delayed. If it does have to do with doze I would be curious to see if the FCM push notification works while HTML5 does not. Personally I have seen myself and others at the same time see the same delays so it leads to me think it was a server issue, I had to stop using HTML5 as a result of that. The notifications would be delayed by like 5 minutes or more and that was not suitable for an alarm.

[asjmcguire]

Doze means the phone will only wake up periodically to download batches of notifications. However if the developers set the urgency of the message then it will normally be delivered immediately to the device causing the device to wake up. As seen here - https://firebase.google.com/docs/cloud-messaging/concept-options#setting-the-priority-of-a-message a webpush would have the priority defined in the header of the push, and I don't see anything setting the priority in the notify/html5.py code.

[andriej]

Seems like lack of priority may be the issue!

[robbiet480]

@andriej Can you submit a PR to update docs to note that?

[andriej]

Isn't it lack of priority declaration in the code that @asjmcguire meant?

[robbiet480]

Misread, sorry @andriej, yep you are correct. Mind submitting a PR to add priority to the code?

[andriej]

That's the moment I can't help - I don't know python. ;-)

[andriej]

@robbiet480 - what one of guys from polish HomeAssistant group described as resolution is working payload.
what he says is differentiation for VAPID and is crucial to get it working is not only 'high' android priority but also "priority" => 10.

Maybe someone (@bieniu ? :-)) could test it to be added too.

[bieniu]

I tested it. I added these lines to the component

payload['android'] = {'priority': 'high'}
payload['priority'] = 10

and unfortunately this didn't help.

[pszafer]

@bieniu so now go to Advanced settings->Battery manager->protected aps or in Sony case Battery -> Excludes from Power Saving and tick Chrome.
For me after 20 minutes, still getting notification with screen off.

[pergolafabio]

That's not possible on a OnePlus, some apps you can indeed exclude from battery saving, but not chrome

[pszafer]

@pergolafabio don't worry. It doesn't help. After 40 minutes phone is not getting notification at all...

Might be related: https://bugs.chromium.org/p/chromium/issues/detail?id=935931

[robbiet480]

If y'all come up with a new fix I can submit a PR with it if no one else is willing to.

[robbiet480]

Related forum thread

[pszafer]

I changed code to something like this:

            jwt_claims = {'exp': jwt_exp, 'nbf': timestamp,
                          'iat': timestamp, ATTR_TARGET: target,
                          ATTR_TAG: payload[ATTR_TAG]}
            #jwt_token = jwt.encode(jwt_claims, jwt_secret).decode('utf-8')
            #payload[ATTR_DATA][ATTR_JWT] = jwt_token
            android_headers = { 'priority': 'high', 'ttl':86400 }
            webpush_headers = {'headers': {'Urgency': 'high'}, 'ttl': 4500}
            info[ATTR_SUBSCRIPTION]['android'] = android_ttl
            info[ATTR_SUBSCRIPTION]['webpush'] = webpush_headers
            info[ATTR_SUBSCRIPTION]['priority'] = 10
            payload['android'] = android_headers
            payload['priority'] = 'high'
            payload['webpush'] = webpush_headers 
            if self._vapid_prv and self._vapid_claims:
                response = webpush(
                    info[ATTR_SUBSCRIPTION],
                    json.dumps(payload),
                    vapid_private_key=self._vapid_prv,
                    vapid_claims=self._vapid_claims,
                    ttl=86400
                )

Now I receive message when I wake up device, but still no wake...
Can somebody check if has similar behaviour with such code?

[bieniu]

@pszafer I tested your fix and the effect is the same. The notification comes after phone wake. This isn't ideal but it's a huge improvement.

[bieniu]

I found this about high priority notifications: "High priority messages generally should result in user interaction with your app or its notifications. If FCM detects a pattern in which they don't, your messages may be de-prioritized. Android P introduced app standby buckets which limit the number of FCM high priority messages you can send to your app that don't result in the user using your app or viewing a notification. If, in response to a high priority message, a notification is displayed in a way that is visible to the user, then your app standby bucket quota will not be consumed by that message."
https://firebase.google.com/docs/cloud-messaging/concept-options
Maybe we have bad habits (dismissing notifications from HA) and that's why high-priority notifications not wake our phones.

[pszafer]

@bieniu let's try to narrow down issue.
https://gist.github.com/pszafer/f8ad3782ef3d48c98471bd6cf1632cf0

I think that we need ttl in webpush method and in webpush headers... It's hard to investigate on one phone.

About second. I always click on notification and get back click event.
Our receiver of messages is Chrome / web browser, so I think html5 push notification to chrome never gonna land in prioritzed bucket.
I think this is our bug: https://bugs.chromium.org/p/chromium/issues/detail?id=777106

[bieniu]

@pszafer On this version also, I receive notification after phone wake up. If I send two notifications when the phone is sleep then I receive only the first ones after wake up.

[Konstigt]

Yikes.. I have both this problem but also that notifications aren't sent at all after "a while" because "UnauthorizedRegistration". That started after switching from GCM to VAPID.

https://github.com/home-assistant/home-assistant/issues/22314

[asjmcguire]

@bieniu let's try to narrow down issue.
https://gist.github.com/pszafer/f8ad3782ef3d48c98471bd6cf1632cf0

I think that we need ttl in webpush method and in webpush headers... It's hard to investigate on one phone.

About second. I always click on notification and get back click event.
Our receiver of messages is Chrome / web browser, so I think html5 push notification to chrome never gonna land in prioritzed bucket.
I think this is our bug: https://bugs.chromium.org/p/chromium/issues/detail?id=777106

I'd be inclined to agree that is the problem, having read the entire bug post - it's clear that expecting this to be fixed anytime soon is a lost cause. It can't hurt Home Assistant to ensure the priority metadata is set - but it looks like setting Chrome to be excluded from battery optimising techniques is the only solution for the moment.

[bieniu]

@asjmcguire It's not always possible to exclude Chrome from battery optimization. On OnePlus phones Chrome is a system application and You can't exclude system applications. You can exclude Chrome Beta but it doesn't help. I checked this today. With Beta notification when the phone sleeps isn't delivered. The next problem is that if several notification are sent when the phone sleeps, only the first ones will be delivered after phone awake. The rest of the notifications will be lost.

[pszafer]

@bieniu about several notification you have to modify tag to not replace previous notification. Isn't it working for you?

[bieniu]

@pszafer I know that. I send notifications without tag or with different tags and always only first notification is delivered after phone awake. Of course, this problem doesn't occur if the phone has a screen turned on.

[bieniu]

@pszafer I tested this once again. And the result is interesting. I sent 5 notifications from HA while the phone was sleeping. Notifications had such data:

{"title":"title 1","message":"message 1","data":{"tag":"tag-1"}}
{"title":"title 2","message":"message 2","data":{"tag":"tag-2"}}
{"title":"title 3","message":"message 3","data":{"tag":"tag-3"}}
{"title":"title 4","message":"message 4","data":{"tag":"tag-4"}}
{"title":"title 5","message":"message 5","data":{"tag":"tag-5"}}

When I woke up the phone, I received notifications number 1, 3 and 5. 2 and 4 were lost.

[ajfriesen]

Just stumbled over this thread.
I am running a Oneplus6 and have no problems with the html5 notifications.

But could it be related that your phone is pretty harsh in regards of sleep mode?

https://dontkillmyapp.com/

[fleshinachair]

I think that's exactly the problem unfortunately. No amount of changing the
settings will rectify this on my Honour Play phone.
I have the same issue with hangouts too.

On Thu, 28 Mar 2019, 20:45 Andrej Friesen, [e-mail] wrote:

Just stumbled over this thread.
I am running a Oneplus6 and have no problems with the html5 notifications.

But could it be related that your phone is pretty harsh in regards of
sleep mode?

https://dontkillmyapp.com/

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/home-assistant/issues/22249#issuecomment-477763940,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AeIQT8jLOjIF9NayZ-8uGLubeflwwU4-ks5vbSnSgaJpZM4cBbcU
.

[andriej]

@ajfriesen what exactly did you set on oneplus not to have those problems? I'm on 6, I think I've followed the instruction and issue is still here...

[pergolafabio]

i have also oneplus 6t, and also the same issues here
seems chrome is an app, that you cant disable for battery optimization

[ajfriesen]

I never had the problems with my html5 push notifications and therefore did not try to optimize something.

I was searching for something different in the issues, stumbled on this one and just wanted you to know about that site, since it helped me to understand my problem:

I had the problem with my owntracks app which was killed by Android and never did update my location.

I did 2 things:

1. Deactivate battery optimization
2. Pin the app in the recent app window so that you can not close it via a swipe.

This works okay so far.
These tips should be on the site too but may be different depending on which device you use.

I even have to do that with my alarm clock!
Otherwise it will not ring in the morning.
Isn't that ridiculous?

[bieniu]

@ajfriesen Can you do a test with sending 5 notifications when your phone is asleep (with different titles, text and tags)? I only receive odd notifications.

@pszafer Your version of the component works for me much better than the original. Are you going to add a PR with your change?

[pszafer]

@bieniu I'm still testing. I have 2 HA sites, one with "new" code and one with old.
Last week both sites were pretty reliable with notification for me and I don't really know if it is because I stopped swipping away notification or what...

[andriej]

For me - swiping is not the case, I'm on the current HA native code and even regarding deactivating battery optimization and locking the app - still few minutes after phone screen is off, notifications are either not reaching at all or there's one and no further arrives.

Also they don't arrive any further time on, just get lost in the middle...

[pszafer]

@andriej have you checked my gist? if it is the same for you?

[ajfriesen]

I am using some notification like everybody I think.
You can see my automation for the notification here:
https://github.com/ajfriesen/home_assistant_configuration/blob/master/automations.yaml

@bieniu I get all my notifications

@andriej Nothing more then you can read up in here: https://dontkillmyapp.com/ about the oneplus in regards to owntracks. For home assistant notifcations I did not do anything on the phone other than accept them via the web interface.

[bieniu]

@ajfriesen Are you sure you are using the html5 component? Your notify service name in automation is notify.chrome_push_oneplus6. For the html5 component, the service name is notify.html5_<device name>

[ajfriesen]

I am using this: https://www.home-assistant.io/components/notify.html5/

Here is my setup for the notify stuff:

https://github.com/ajfriesen/home_assistant_configuration/blob/master/configuration.yaml#L17

You can name that service whatever you want.

Maciej Bieniek [e-mail] schrieb am Fr., 29. März 2019,
18:05:

@ajfriesen https://github.com/ajfriesen Are you sure you are using the
html5 component? Your notify service name in automation is
notify.chrome_push_oneplus6. For the html5 component, the service name is notify.html5_ name>

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/home-assistant/issues/22249#issuecomment-478075484,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGgvv-Yj_sAEFvdjEKbhCzwfuc1dgJNhks5vbkfZgaJpZM4cBbcU
.

[bieniu]

@ajfriesen OK, I forgot about name in configuration ;)

[andriej]

Some info found online:

★ Note: Starting January 2019, App Restrictions (in Battery Setting) will include restrictions on FCM messages. You can find out if your app is in the restricted state with the isBackgroundRestricted API. Once your app is in the restricted state, no FCM messages will be delivered to the app at all. This will apply to both high and normal priority FCM messages and when app is in either foreground or background.

source, near the bottom.

In linked further materials there's also info about how frequent app is used makes the difference of high-priority messages allowed anyway.

[andriej]

Here is list of (lots) of event's going back to FCM that can be debugged - they show also:
events that can be interesting to debug:

MESSAGE_DELAYED_DOZE: the message delivery to the device was being delayed because the device is in doze mode;
QUOTA_EXCEEDED: the request to send a message was rejected due to insufficient quota;

if I only know how to make it work on my account ;-)

Also: relatively new blog post for developers (february 2019) about life of a message FCM

and

Android Restrictions: If the web application is running on a browser on Android, all the restrictions defined under 'Android Factors' will apply to Web as well.

[bieniu]

@pszafer Why did you change jwt_exp? My tests show that it doesn't change anything. The ttl is important because pywebpush defaults use ttl = 0 which means "send now and forget".

ttl: The Time To Live in seconds for this message if the recipient is not online. (Defaults to "0", which discards the message immediately if the recipient is unavailable.)

When the phone is sleeping, the message isn't delivered and the FCM server forgets about it. Also ttl for phones is necessary.

[pszafer]

I commented out jwt_exp in first version because I couldn't find any docs that it is needed in fcm. IMO it's not needed but I'm not sure. In latest version I came back to jwt because it doesn't break things either.

TTL shouldn't be necessary as critical notifications should have TTL 0 (phone call) but for chrome it doesn't always work...

[bieniu]

In my opinion, TTL should be used. For example, the user has switched off the phone and during this time gets critical notification from the Home Assistant (alarm, movement, gas leak, smoke detection, etc.). After turning on the phone, the user should get a notification and not get it with TTL = 0.

[andriej]

In my opinion TTL=0 would work if all notifications would get to the phone and wake it up. Currently because of this parameter those are 'dying' on the way, while phone is in doze.

[robbiet480]

Relevant

[andkit]

In my opinion TTL=0 would work if all notifications would get to the phone and wake it up. Currently because of this parameter those are 'dying' on the way, while phone is in doze.

I rather agree with @bieniu, ttl=0 is not only an issue for Doze. There's also spotty mobile coverage, airplane mode and rumor has it that some people even turn their phone off occasionally, or forget to charge it ;)

Therefore I think making ttl configurable per notification might be the best solution, since not all notifications stay relevant equally long.

[eyager1]

To rule out Chrome issues, I have tried testing HTML5 push notifications in Firefox for Android. Without the TTL being set neither push notifications in Chrome, nor in Firefox work when the phone (Moto 5G plus) is in doze. I can confirm that push notifications don't work when the phone is in doze with Brave either (but its chromium based). It doesn't matter if the priority flag is set is to high. With the TTL value set to a different value (I used 36000) Firefox will display the push notification when the phone wakes from doze, but Chrome has still dropped the push notification.

So I think there are two separate bugs causing the problem.... One in Chrome which we can't fix. The other is in home assistant, with the TTL being set to zero, which is expiring the notification when the phone is in doze.

[andkit]

If it helps: I have used patched my instance with a ttl of 3600s and the priorities mentioned above.
On my phone I use home assistant in "webapk" mode or whatever it's called (add to Homescreen)
I generally use Chrome canary (also set as webview implementation), so excluded both assistant and chrome canary from battery optimization.

With these settings my html5 notifications work fine

[pergolafabio]

On OnePlus phones, we can't exclude chrome from battery optimalization

[bieniu]

@pergolafabio You can exclude Chrome Canary from battery optimization but can't stable build.

[pergolafabio]

Aha, didn't know what canary was , thnx for info

[eyager1]

Extended testing has revealed that eventually the firefox browser started dropping and completely ignoring push notifications while the phone is in doze. It doesn't matter if battery optomization for firefox is disabled. So I suspect there there is indeed an algorithm at work within android deciding that the behavior of home assistant's push notifications is too spammy, too potentially battery wasting, and not meeting their user interaction metrics and thus the expected behavior is being silently modified in the name of optimizing my android user experience.

[thomasgermain]

Just tested the fix in https://github.com/home-assistant/home-assistant/pull/22988.

I created an html5 custom component containing changes of the PR
I tested with my oneplus 5 (android 9) with "webapk" (add to homescreen). It works !

I tested like this:

• phone in use -> working
• phone just went to sleep -> working (notification is displayed "instantly" and it's waking up my phone)
• phone in doze (>5 minutes) -> working (notification is displayed with 1~5 seconds delay and it's waking up my phone).

Also tested with my wife's phone (samsung A5 2017, android 8) which was already working, everything works fine too.

Thanks for the fix !

[pszafer]

@thomasgermain so it's far better working for you than for me :)
My phone never wakes, but notification is always received after wake up (Sony XZ1 /Android 9).

[thomasgermain]

@pszafer sorry for being too optimistic :(

I forgot to mention that the webapk is not battery optimized, but I guess is something you already tried.

[eyager1]

edit Test environment is a MotoG5 Plus (Android 8.1.0) and Chrome Canary, with home assistant as a webapk with battery optomization disabled. After patching the correct file FCM push notifications now are received reliability upon waking at the very least.

[robbiet480]

Again relevant to this conversation is the previously mentioned website https://dontkillmyapp.com/. You should check the site for details about what extra measures, if any, your manufacturer takes to conserve battery.

[0BuRner]

For those who have time to inspect deeper what's going on, BatteryHistorian is pretty useful : https://developer.android.com/studio/profile/battery-historian

[e2m32]

Just tested the fix in #22988.

I created an html5 custom component containing changes of the PR
I tested with my oneplus 5 (android 9) with "webapk" (add to homescreen). It works !

I tested like this:

• phone in use -> working
• phone just went to sleep -> working (notification is displayed "instantly" and it's waking up my phone)
• phone in doze (>5 minutes) -> working (notification is displayed with 1~5 seconds delay and it's waking up my phone).

Also tested with my wife's phone (samsung A5 2017, android 8) which was already working, everything works fine too.

Thanks for the fix !

22988 Did the trick! Thanks! I just copied @pszafer' s files from here and put them in <config>/custom_components/html5. After a reboot, the notifications are coming in again 🎉🎉.

I did do a code edit to make all html5 messages 'high priority'. This way there is no delay in the message. Most of my messages are from my alarm, so I need them to come in as soon as possible.

Line 84:
DEFAULT_PRIORITY = 'high'

It looks like this might be able to be configured from config.yaml or even for each automation, but I didn't want to take the time to find out. This way I know they are all set to high priority.

I am on a Pixel 2 XL running Android 9 with battery optimization off on Chrome and the webapk. I tried Chrome Canary at the same time and there was no difference in responsiveness, but I do really like the app-like feel that canary offers with the webapk. I'm going to keep that :-)

[nickm324]

I did the same thing and it prevented HA lovelace from being accessible. I could get to the samba shares so it appeared that HA was loading and running. But until I removed the HTML5 folder from custom_components folder it was not letting the UI come up. Did I miss a step?

[jurgenweber]

Looks like it is all related; https://github.com/home-assistant/home-assistant/issues/23613

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[pergolafabio]

hi, whats the actual status on this? is it possible now to have the messages instantly ?

[thomasgermain]

Hi,

you will have to set priority case by case. This is working for me. I have some prior notifications that are waking up my phone and other non-prior notifications I will only see when unlocking my phone.

See https://www.home-assistant.io/integrations/html5/#ttl-and-priority

[pergolafabio]

yeah, i know, setup like this now, but not reliable

  - service: notify.html5
    data_template:
      message: "Er staat iemand aan de deur!"
      title: "Deurbel"
      data:
        tag: alert
        image: !secret snapshot_buiten_deurbel
        vibrate:
          - 300
          - 100
          - 400
        renotify: 1 
        ttl: 86400
        priority: high

[fleshinachair]

I'm hoping this will all be resolved the when the Android app is released?

On Tue, 8 Oct 2019, 14:03 pergolafabio, [e-mail] wrote:

yeah, i know, setup like this now, but not reliable

• service: notify.html5
  data_template:
  message: "Er staat iemand aan de deur!"
  title: "Deurbel"
  data:
  tag: alert
  image: !secret snapshot_buiten_deurbel
  vibrate:
  - 300
  - 100
  - 400
  renotify: 1
  ttl: 86400
  priority: high

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/home-assistant/issues/22249?email_source=notifications&email_token=AHRBAT255NHHWKNYU2W4LSTQNSALNA5CNFSM4HAFW4KKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAUCPQA#issuecomment-539502528,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHRBATYTQ7YPXWX3ZP2RP7LQNSALNANCNFSM4HAFW4KA
.

[andriej]

Hopefully. Right now most of the messages arrive to the phone, except they are not waking it up.

[pergolafabio]

yeah, same behaviour here, when i wake up the phone, i receive the message
so not usefull for a doorbell automation :-)

i started today with the "pushover" integration, seems the work verry good and verry easy
for me an alternative untill html5 is working again