Fenix: [Bug] Unable to send Leanplum messages in (Migrated) Fennec Nightly app, API Key improperly configured

[lwa-moz]

Steps to reproduce

Stage and send notification in Leanplum to a registered user of the Firefox Nightly (aka org.mozilla.fennec_aurora) app.

Expected behavior

Notification is sent.

Actual behavior

Notification is not sent.

Device information

• Android device: Google Pixel 3
• Fenix version: Nightly 200301 18:00
• Android OS Version: 10

According to Leanplum support:
"There are no Android Notification Channels defined for the app, so the notification was dropped.
Please, define a channel as described here:
https://docs.leanplum.com/reference#section-android-notification-channels"

It is critical that we define notification channels in the Leanplum SDK so that we can distribute notifications to our users.

[lwa-moz]

This bug is potentially related to this issue: https://github.com/mozilla-mobile/fenix/issues/8745

CC'ing @vesta0, @jonalmeida, @cbonacuse for visibility.

[jonalmeida]

"There are no Android Notification Channels defined for the app, so the notification was dropped.
Please, define a channel as described here:
https://docs.leanplum.com/reference#section-android-notification-channels"

This makes sense since we only setup notification channels for Firefox Preview (org.mozilla.fenix) afaik.

[lwa-moz]

@jonalmeida sounds like we can confirm the source of the issue.

Please let me know if you need help with the channel titles.

[jonalmeida]

@lwa-moz I've set this up for the org.mozilla.fennec_aurora with "Firefox news" as the notification channel name.

[liuche]

@lwa-moz is this working now, or did this fix not resolve the problem?

[cbonacuse]

@liuche & @lwa-moz - I tested this but did not receive a notification. Below is Leanplum Support's response. Can you look into this?

I have inspected the logs for this specific message and user and here is the error that is generated:

Status:ERROR||NotificationId:b054aaf0-5807-4202-a42c-4355c8432cc5|Users:43222ab4-acbd-405b-9464-9cc1ccb2bde7|SuppressionReason:GCM_MISMATCHED_SENDER_ID|ThrottlingAdjustment:0|AppId:5094741967896576|Attempt:0|Stage:SEND_RESULT|MessageId:889575665

In most cases, this error appears when there is a Google API key mismatch. The Google API key that is provided in Leanplum "Setting" section is different than the key from the Firebase console. Please provide this information to your developers at your earliest convenience and ask them to verify that the following key for the "Firefox Android Nightly" app is the same as the key from the Firebase console.

https://www.screencast.com/t/bnyDTzXhxP

[lwa-moz]

@st3fan looks like there may be a mismatch between the "Google Play API key" listed in the Leanplum dashboard and what's listed on our end.

I'm a little perplexed at Leanplum support mentioning a "Firebase console", and can follow up for more information if we don't believe there's a mismatch between the API Key listed in the Leanplum console and our code.

[AndiAJ]

Hi, I've tried sending push messages after performing a migration from Fennec to Nightly Build #2015729651 from 3/20 using the following devices:
• Google Pixel 3a (Android 10) - Device ID 811d1464-7f36-48d5-b630-43e6c7a323b8
• Huawei Mate 20 Lite (Android 8.1.0) - Device ID 5e7630e2-4902-42d8-96db-6829caf4a467

In both cases, no messages were properly sent.
► Leanplum confirmation

► Google Pixel 3a notifications section

► Huawei Mate 20 Lite notifications section

I'll remove the QA needed until further notice.

[lwa-moz]

Updating this issue with feedback from Leanplum support:
"I have inspected the logs for this specific message and user and here is the error that is generated:

Status:ERROR||NotificationId:b054aaf0-5807-4202-a42c-4355c8432cc5|Users:43222ab4-acbd-405b-9464-9cc1ccb2bde7|SuppressionReason:GCM_MISMATCHED_SENDER_ID|ThrottlingAdjustment:0|AppId:5094741967896576|Attempt:0|Stage:SEND_RESULT|MessageId:889575665

In most cases, this error appears when there is a Google API key mismatch. The Google API key that is provided in Leanplum "Setting" section is different than the key from the Firebase console. Please provide this information to your developers at your earliest convenience and ask them to verify that the following key for the "Firefox Android Nightly" app is the same as the key from the Firebase console.

https://www.screencast.com/t/bnyDTzXhxP

Please update the case once the API key is verified."

@liuche it appears there may be some mismatch in the API key that's being used in the Leanplum SDK integration in Firefox Nightly.

I suspect that the API key was not updated when the Nightly update was made. My guess is that it is probably still set to the Firefox Preview API Key.

[st3fan]

@jonalmeida @JohanLorenzo is it possible that we use the Fenix Nightly key here instead of the Fennec Nightly key? I wonder if those keys are bound to an app identifier, and if we use the same GCM key for both org.mozilla.fennec_aurora and org.mozilla.fenix.nightly then that may not work?

[jonalmeida]

I don't have access to the FCM key from the Firebase console, but what I can see on Leanplum seems like the client API key was entered into the Leanplum console instead of the server API key.

I might as well look into this. :)

[JohanLorenzo]

Hey there!

I checked the Fennec nightly leanplum secret stored at [1]. It's not the same we use on Fenix nightly[2]. When built on mozilla-esr68, Fennec nightly used [3], which contains exactly the same secret as [1].

In order words, Fennec Nightly uses the right key.

[1] https://firefox-ci-tc.services.mozilla.com/secrets/project%2Fmobile%2Ffenix%2Ffennec-nightly
[2] https://firefox-ci-tc.services.mozilla.com/secrets/project%2Fmobile%2Ffenix%2Fnightly
[3] https://firefox-ci-tc.services.mozilla.com/secrets/project%2Freleng%2Fgecko%2Fbuild%2Flevel-3%2Fleanplum-sdk-nightly.token

[jonalmeida]

@JohanLorenzo I think the issue here is on the Leanplum console that has the wrong API key to communicate with FCM.

I've updated those keys to the correct one on the console for 'Firefox Android Nightly' and 'Firefox Android Beta'. I don't see a project for a "Firefox Android" meant for production, but I guess we ask @lwa-moz about setting that up when we get there?

@AndiAJ could this please be re-tested once again? Thank you in advance!

[cbonacuse]

@jonalmeida & @AndiAJ - we are still seeing this issue in Nightly where no notifications are being received and tried sending tests in Beta yesterday as well to no avail. Below is the feedback from Leanplum support on the issue of not receiving notifications in Beta. Is it possible to have someone look into this ASAP? We are currently blocked on sending a critical time-sensitive push notification to Beta users on Friday 4/3 (before the upgrade to Fenix).

cc: @st3fan

"Checking the logs for the provided message I can see that the following error is recorded:

|Status:ERROR|QuietHoursAdjustment:0|NotificationId:52b0259e-465a-4f36-a6fe-f4f11bdab3e3| Users:2e025405-a82b-4bb7-9624-8164dc21b469 |SuppressionReason:GCM_MISMATCHED_SENDER_ID|791872|Attempt:0|NotificationType:GCM_PUSH|Stage:SEND_RESULT|MessageId:5231885831503872|

The error is the same as the one we have discussed around a week ago in case 32546. Here is what I have provided as information in the other case.

In most cases, this error appears when there is a Google API key mismatch. The Google API key that is provided in Leanplum "Setting" section is different than the key from the Firebase console. Please provide this information to your developers at your earliest convenience and ask them to verify that the following key for the "Firefox Android Nightly" app is the same as the key from the Firebase console.

https://www.screencast.com/t/bnyDTzXhxP

The previous time the issue was related to the "Firefox Android Nightly" application. The issue for this one seems to be the same. Can you please let me know whether you have forwarded this information to your developers and whether you managed to fix the problem for the "Firefox Android Nightly" application. If the issue is fixed for the Nightly application the same approach should be used for the Beta application too."

[st3fan]

The quotes messafe from LP Support was from Yesterday 3/31 at 5:30 PM ET.

I just checked the LP Dashboard and we do have the correct Google API Key configured for both Fennec Nightly and Fennec Beta. (Which includes not-migrated and migrated users)

I'll catch up with people today to understand this problem better.

[AndiAJ]

Hi, I've re-tested this matter after performing a migration to Fenix Nightly Build #2015731955 from 4/1 and the push notifications was properly sent on the following devices:
• Google Pixel 3a (Android 10) - Device ID 2480ca10-6a6c-48bc-8e9e-2618c9814364
• Huawei Mate 20 Lite (Android 8.1.0) - Device ID 1e29fd10-0594-45f9-964b-a15a6ccfe8ff

► Screenshots

@st3fan & @jonalmeida - Should I test this on other android versions as well?
Does this matter need to be checked for Beta migration ?

[st3fan]

@AndiAJ would mind also testing the same on Fennec Beta. Both before and after migration?

[AndiAJ]

Hi @st3fan , I've re-check this matter as it follows:

1. Migrated Fenix Nightly

✔️ Google Pixel 3a, Android OS 10 - eb935ca3-d538-410f-89d6-424b1877efd9

✔️ HUAWEI SNE-LX1, Android OS 9 - cf079d2a-ada4-47d2-be08-512af26dbedd

✔️ OnePlus ONEPLUS A3000, Android OS 6.0.1 - 4c32a70a-861a-4034-8c88-7a7337093c4b

✔️ Samsung SM-G930F, Android OS 7.0 - 186a1b84-fab6-460f-8009-aaa699e7d7a3

✔️ Samsung SM-T285, Android OS 5.1.1 - f0714b3c-096f-4284-9f07-3623a7765f63

❌ LGE Nexus 4, Android OS 5.1.1 -1e00ed58-0403-44a0-99f2-9d86396eaa78 - ❗️ No push token got generated

2. Firefox Beta

✔️ Google Pixel 3a, Android OS 10 - 02e3a402-9afc-4ef5-b8ac-497c0cf8c9fe

✔️ HUAWEI SNE-LX1, Android OS 9 - adcc46a8-8cee-4625-a7e9-5566272ebfee

✔️ OnePlus ONEPLUS A3000, Android OS 6.0.1 - 9590df5c-d3ae-4b73-87b2-1bdbe8958235

✔️ Samsung SM-G930F, Android OS 7.0 - 97bc2403-e43e-422b-9a37-5ae6ee550496

✔️ Samsung SM-T285, Android OS 5.1.1 db595d7f-3f1d-4102-8fd7-38a946116cb0

❌ LGE Nexus 4, Android OS 5.1.1 - 6e87aeac-0152-4514-8876-6a59acbdd141 - ❗️ No push token got generated

3. Perfomed a migration to Fenix Beta 75.0.0-beta.3 from 4/1

❌ Google Pixel 3a, Android OS 10 - 02e3a402-9afc-4ef5-b8ac-497c0cf8c9fe
❌ HUAWEI SNE-LX1, Android OS 9 - adcc46a8-8cee-4625-a7e9-5566272ebfee
❌ OnePlus ONEPLUS A3000, Android OS 6.0.1 - 9590df5c-d3ae-4b73-87b2-1bdbe8958235
❌ Samsung SM-G930F, Android OS 7.0 - 97bc2403-e43e-422b-9a37-5ae6ee550496
❌ LGE Nexus 4, Android OS 5.1.1 6e87aeac-0152-4514-8876-6a59acbdd141 ✅ The push token got generated
❌ Samsung SM-T285, Android OS 5.1.1 db595d7f-3f1d-4102-8fd7-38a946116cb0

4. Directly installed the migration build Fenix Beta 75.0.0-beta.3 from 4/1

❌ Google Pixel 3a, Android OS 10 - a94e80f3-536d-402b-aac0-67dfd71b53ce
❌ HUAWEI SNE-LX1, Android OS 9 - c403a75f-ed1d-4d3a-aac4-3f18c96afc2e
❌ OnePlus ONEPLUS A3000, Android OS 6.0.1 - 2b687871-f682-428a-8050-814c2014bd9b

Please review and share your thoughts. ☺️

[jonalmeida]

Thanks @AndiAJ , that sounds like the results we expected - 3 and 4 is expected to fail with the current Leanplum dashboard setup that has only the fennec tokens setup.

[jonalmeida]

Closing this as fixed

cc: @st3fan @lwa-moz