Firebase-admin-node: Firebase Messaging returning "Invalid registration token" when sending silent APNs notification

[gdieguez]

[REQUIRED] Step 2: Describe your environment

• Operating System version: MacOS Big Sur
• Firebase SDK version: 9.6.0 (Firebase Admin version)
• Firebase Product: messaging
• Node.js version: 12
• NPM version: 6.14.12

[REQUIRED] Step 3: Describe the problem

FCM started responding with INVALID_ARGUMENT with a description of Invalid registration token to any calls for sending APNs silent notifications. We've verified that the tokens are actually valid. This happens in the node client, and also when calling directly the HTTP v1 send endpoint.

We started noticing this behavior roughly on 2021-04-08 14:30:00 ET, which suspiciously coincides with a deploy of our production functions. Note that this deploy didn't include any updates to the relevant codepaths nor libraries involved.

We also found someone with the very same problem on Stack Overflow: https://stackoverflow.com/questions/67051941/fcm-throwing-invalid-argument-for-ios-notifications

Steps to reproduce:

Call the HTTP v1 send endpoint:

POST https://fcm.googleapis.com/v1/{parent=projects/*}/messages:send

with the following payload:

{
  "validate_only": false,
  "message": {
    "token": <FCM TOKEN>,
    "apns": {
        "payload": {
            "aps": {
              "content-available": 1
            }
        }
    }
  }
}

with a valid FCM token and you should receive the following response:

{
    "error": {
        "code": 400,
        "message": "Request contains an invalid argument.",
        "status": "INVALID_ARGUMENT",
        "details": [
            {
                "@type": "type.googleapis.com/google.firebase.fcm.v1.FcmError",
                "errorCode": "INVALID_ARGUMENT"
            },
            {
                "@type": "type.googleapis.com/google.rpc.BadRequest",
                "fieldViolations": [
                    {
                        "field": "message.token",
                        "description": "Invalid registration token"
                    }
                ]
            }
        ]
    }
}

But if you replace content-available with badge, the notification is sent correctly:

{
    "name": "projects/<PROJECT_ID>/messages/<MESSAGE_ID>"
}

We've also tried sending content_available and contentAvailable as keys, and true as value, to no avail.

[gdieguez]

@hiranya911 please take a look when you can

[hiranya911]

This has been forwarded to the FCM backend team. They are looking into it. We'll post an update when new information is available.

[frenberg]

@hiranya911 Sweet, we're seeing this too. Is there somewhere we can follow the progress over at the FCM backend team?

[ssprang]

Hi! We are using the java version of this library and we're not getting any exceptions but our APNs silent notifications stopped working at about the same time on 2021-04-08. @hiranya911 it sounds like a backend release of FCM caused this, would it be possible to rollback that change? APNs silent notifications not working is affecting our end users.

[chong-shao]

FCM backend team has identified the issue, expect to have it fixed within a week. I'll post more updates on timelines, etc.

[gdieguez]

Thanks for the update. For all of you who are waiting for this to be fixed, we were able to send silent notifications using the legacy API. I left some details on the stack overflow post I linked before.

[ssprang]

FCM backend team has identified the issue, expect to have it fixed within a week.

That is great! But a week is a very long time.. Especially since it's been broken for almost a week already. @chong-shao or @hiranya911 could you confirm if this issue is affecting all FCM APNs silent notifications (including those of us using the java version of this library)? Can't the FCM backend team rollback the release that caused the issue to begin with?

[viktorvoltaire]

If this is a FCM backend problem, shouldn't there be any mention of it at https://status.firebase.google.com/ ?

[gdieguez]

They actually put a message on the status website a couple hours ago but now it's gone, so I tested and it seems to be fixed. @hiranya911 @chong-shao can you confirm?

[chong-shao]

Latest update: FCM backend performed a fast rollback today. The backend should be at the good version without this issue now.

[viktorvoltaire]

Latest update: FCM backend performed a fast rollback today. The backend should be at the good version without this issue now.

@chong-shao
Even though it was never confirmed that this affected the firebase-admin-java library, we can say that after the rollback it is also functioning again. Whats scary is that we saw no errors on our end, suddenly silent notifications just stopped occurring in our app and the graph for sending messages went down to 0 at https://console.firebase.google.com/u/0/project/PROJECT/notification/reporting but we saw 0 increase in errors towards the API

I hope this could be forwarded to the team that it would be nice if we at least got a status error in the library when such an important feature stopps working :)

Cheers for the rollback!

[salkovich]

We still have the problem here in our project. Reports show that not a single push from background came to client.
We also tried examples in your docs: concept options via OAuth2 playground. Same thing happens.

If we try via legacy system, pushes are working as they should.

[Matoz]

Works for us again. Thanks for the rollback

[chong-shao]

@salkovich Hello, did you observe "invalid_argument_error"/"invalid registration token" error from the send response? The issue you described might be a separate one. It's recommended to file a support ticket https://firebase.google.com/support and provide your: project id, FCM token (if any), sample request/response so our support team can investigate. Thanks!

[ashrafsabrym]

I'm still getting this error on my Xamarin iOS app. I've upgraded the FirebaseAdmin Nuget package to the latest version 2.1.0 on my ASP.Net backend but still get the issue
Does this have to do with the Xamarin.Firebase.iOS.CloudMessaging Nuget package on the app?