Amplify-js: Pinpoint: Exceeded maximum endpoint per user count: 10

[joebernard]

Describe the bug
After upgrading to "@aws-amplify/analytics": "^4.0.0" push notifications have stopped working. I can no longer update endpoints. This was originally solved in #5423 but seems to have reappeared recently. Possibly related to the merge in #7245 .

To Reproduce
Install the app on a device more than 10 times.

Expected behavior
Amplify should clear old endpoints as mentioned in the docs.

Environment

System:
    OS: macOS 10.15.7
    CPU: (12) x64 Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
    Memory: 208.17 MB / 32.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 14.2.0 - ~/.nvm/versions/node/v14.2.0/bin/node
    Yarn: 1.22.5 - /usr/local/bin/yarn
    npm: 6.14.8 - ~/.nvm/versions/node/v14.2.0/bin/npm
    Watchman: 4.9.0 - /usr/local/bin/watchman
  Browsers:
    Chrome: 86.0.4240.198
    Firefox: 82.0.3
    Safari: 14.0
  npmPackages:
    @apollo/client: ^3.2.7 => 3.2.7 
    @aws-amplify/analytics: ^4.0.0 => 4.0.0 
    @aws-amplify/auth: ^3.4.12 => 3.4.12 
    @aws-amplify/cache: ^3.1.37 => 3.1.37 
    @aws-amplify/core: ^3.8.4 => 3.8.4 
    @aws-amplify/storage: ^3.3.12 => 3.3.12 
    @babel/core: ^7.8.4 => 7.11.6 
    @babel/runtime: ^7.8.4 => 7.9.2 
    @bugsnag/react-native: ^7.5.2 => 7.5.2 
    @expo/react-native-action-sheet: ^3.8.0 => 3.8.0 
    @react-native-community/art: ^1.2.0 => 1.2.0 
    @react-native-community/async-storage: ^1.12.1 => 1.12.1 
    @react-native-community/eslint-config: ^1.1.0 => 1.1.0 
    @react-native-community/masked-view: ^0.1.10 => 0.1.10 
    @react-native-community/netinfo: ^5.9.7 => 5.9.7 
    @react-native-community/picker: ^1.8.1 => 1.8.1 
    @react-native-community/push-notification-ios: 1.7.3 => 1.7.3 
    @types/react: 16.9.56 => 16.9.56 
    amazon-cognito-identity-js: ^4.5.5 => 4.5.5 
    apollo-link-retry: ^2.2.16 => 2.2.16 
    appcenter: 3.1.2 => 3.1.2 
    appcenter-analytics: 3.1.2 => 3.1.2 
    appcenter-crashes: 3.1.2 => 3.1.2 
    aws-appsync-auth-link: ^3.0.2 => 3.0.2 
    aws-appsync-subscription-link: ^3.0.3 => 3.0.3 
    babel-jest: ^25.1.0 => 25.5.1 
    eslint: ^6.5.1 => 6.8.0 
    exponential-backoff: ^3.1.0 => 3.1.0 
    graphql: 15.4.0 => 15.4.0 
    jest: ^25.1.0 => 25.5.4 
    lodash.debounce: ^4.0.8 => 4.0.8 
    lodash.throttle: ^4.1.1 => 4.1.1 
    metro-react-native-babel-preset: ^0.59.0 => 0.59.0 
    moment: ^2.29.1 => 2.29.1 
    moment-timezone: ^0.5.32 => 0.5.32 
    prop-types: ^15.7.2 => 15.7.2 
    react: 16.13.1 => 16.13.1 
    react-dom: ^16.12.0 => 16.13.1 
    react-native: 0.63.3 => 0.63.3 
    react-native-animatable: ^1.3.3 => 1.3.3 
    react-native-camera: 3.40.0 => 3.40.0 
    react-native-code-push: ^6.4.0 => 6.4.0 
    react-native-config: 1.4.0 => 1.4.0 
    react-native-country-picker-modal: ^2.0.0 => 2.0.0 
    react-native-device-info: ^7.1.0 => 7.1.0 
    react-native-fast-image: ^8.3.4 => 8.3.4 
    react-native-fs: ^2.16.6 => 2.16.6 
    react-native-gesture-handler: ^1.8.0 => 1.8.0 
    react-native-get-random-values: ^1.5.0 => 1.5.0 
    react-native-haptic-feedback: ^1.11.0 => 1.11.0 
    react-native-orientation-locker: ^1.2.0 => 1.2.0 
    react-native-permissions: ^2.2.2 => 2.2.2 
    react-native-progress: ^4.1.2 => 4.1.2 
    react-native-reanimated: ^1.13.2 => 1.13.2 
    react-native-root-toast: ^3.2.1 => 3.2.1 
    react-native-safe-area-context: ^3.1.9 => 3.1.9 
    react-native-safe-area-view: ^2.0.0 => 2.0.0 
    react-native-screens: ^2.15.0 => 2.15.0 
    react-native-share: 4.1.0 => 4.1.0 
    react-native-svg: ^12.1.0 => 12.1.0 => 0.3.4 
    react-native-tab-view: ^2.15.2 => 2.15.2 
    react-native-vector-icons: 7.1.0 => 7.1.0 
    react-native-video: ^5.1.0-alpha8 => 5.1.0-alpha8 
    react-native-webview: ^10.10.2 => 10.10.2 
    react-navigation: ^4.4.3 => 4.4.3 
    react-navigation-props-mapper: ^1.0.0 => 1.0.4 
    react-navigation-stack: ^2.10.1 => 2.10.1 
    react-navigation-tabs: ^2.10.1 => 2.10.1 
    react-test-renderer: 16.13.1 => 16.13.1 
    uuid: ^8.3.1 => 8.3.1 
  npmGlobalPackages:
    @aws-amplify/cli: 4.32.1
    appcenter-cli: 2.7.2
    gatsby-cli: 2.14.0
    lerna: 3.22.1
    npm: 6.14.8
    serverless: 2.11.1
    typescript: 3.9.6

Additional context
May be related to #7245

[jpignata]

Thanks @joebernard! Are you seeing this locally in development during test or in production use?

[joebernard]

@jpignata I've only been able to test in development so far but the previous time this happened it was affecting production too. This only started breaking again after upgrading to 4.0.0. I tried rolling back but then started experiencing a different issue (#6145). I can't seem to get around that one either as it seems to persist through re-installs so I'm currently blocked.

[v1pz3n]

Also here

[v1pz3n]

Amplify updated to 3.3.10 and it worked

[joebernard]

Interesting. Maybe this is only broken in the modular packages. Unfortunately I cannot easily convert.

[njdullea]

I am not using modular packages, was on amplify 3.3.9 and encountered this issue. Updating to 3.3.10 did not resolve the issue.

Update: reverting to 3.3.8 resolved it for now.

[joebernard]

Can we get an update on how to proceed now that we are stuck with this artificial endpoint limit? Are we expected to manually manage this limit within Pinpoint somehow? What was the rationale behind removing this functionality? This has become a huge blocker for my team.

[jpignata]

Hi @joebernard - we're working on this with Pinpoint and will have more to share soon. Do you have anytime to chat tomorrow on discord perhaps? I'd like to learn more about the block and see if we can help. Happy Thanksgiving!

[joebernard]

@jpignata That would be helpful. I can meet anytime between 9:00 AM and 2:00 PM ET Nov 27. Happy Thanksgiving to you as well.

[sammartinez]

@joebernard Are you ok if we resolve this issue? Just wanted to validate if there were any other issues you were facing after our conversation last week. Please let us know

[joebernard]

Hi @sammartinez, thanks for asking. I would consider my case temporarily mitigated by rolling back, but the underlying issue re-introduced in @aws-amplify/analytics": "4.0.0" is still a problem. We uninstall / re-install apps so often that we quickly hit that 10 endpoint limit. It could also affect our production users though it would be more of an edge case. With no way for us to manage endpoints, this issue effectively breaks push notifications.

To me, this is a showstopper that precludes us from upgrading to 4.0.0 and beyond. We're locked into specific versions and might not be able to benefit from future features or security patches. I feel we either need some mechanism to manually manage endpoints, automated management of endpoints in Amplify or (preferably) in Pinpoint, or other guidance that would allow us to continue staying current with Amplify packages while avoiding this issue.

[v1pz3n]

I thought it worked but it was only in the web version.
For some reason, my react native on android and ios has the pinpoint error.

[dylan-westbury]

Hi @sammartinez, thanks for asking. I would consider my case temporarily mitigated by rolling back, but the underlying issue re-introduced in @aws-amplify/analytics": "4.0.0" is still a problem. We uninstall / re-install apps so often that we quickly hit that 10 endpoint limit. It could also affect our production users though it would be more of an edge case. With no way for us to manage endpoints, this issue effectively breaks push notifications.

I believe new endpoints not only occur when you uninstall apps, but just by clearing local storage.

For us, our big concern is that iOS users will stop receiving push notifications as we can't update the endpoint with the address with the new token.

[blacksheep--]

This is now the 5. incarnation of the very same topic. It seems this problem keeps popping up every other release. I guess it may be due to the fragile nature of the underlying code, where the returned message is string-checked to be starting with 'Exceeded maximum endpoint per user count' in AWSPinpointProvider.ts#461:

!String(message).startsWith('Exceeded maximum endpoint per user count')

A slight change of the message kills it, which already happened in the past.

Also OP seems absolutely right with his assumption that PR #7245 removed the clean-up of unused endpoints which worked perfectly before. How is this supposed to be handled?

[paddlefish]

Here's a manual workaround you can do using the CLI for each affected user:

step 1. Use cognito to search for the affected users sub

in this example i found user 719f397f-c6a6-4598-b603-7cc4d9d17911
also use pinpoint console to look up your analytics application id.

step 2. see how many endpoints are in pinpoint

aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189
{
"EndpointsResponse": {
"Item": [
{
"ApplicationId": "daafcc04059a444aa9363e484113c189",
"Attributes": {},
"CohortId
...

count them .. here i'm using jq and wc to count

➜ aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189| jq '.EndpointsResponse.Item[]|.Id' | wc
10 10 390

yup there are 10. so this user can't sign in due to this 10 endpoint restriction

step 3. Clear the oldest endpoints

3a. find the oldest ones. I scanned the list printed in step 2 and had manually found the newest endpoint creation date ('2020-12-18T19:45:48.212Z') to put in this expression:

 aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189| jq '.EndpointsResponse.Item[]|select(.CreationDate | contains("2020-12-18T19:45:48.212Z") | not)|.Id'

"3aa4dc90-2501-11eb-a16e-99d12931a92e"
"e5cd9240-276f-11eb-af3c-8130a9b73a99"
"3f66e740-2dae-11eb-aae4-153b561a5e3b"
"35b34700-2513-11eb-a296-0558d678e564"
"7ddf5e70-29cc-11eb-bdbf-671aec7e1f1f"
"94427fe0-3038-11eb-bf01-5b5367143dd5"
"2cbe3c00-3296-11eb-a617-23713ff6a6ee"
"50a2f3a0-357e-11eb-a551-958abb691880"
"3928cc60-3964-11eb-9bc3-bdabe9b761f6"

3b. use xargs to delete them

aws --profile YourProfile pinpoint get-user-endpoints --user-id 719f397f-c6a6-4598-b603-7cc4d9d17911 --application-id daafcc04059a444aa9363e484113c189| jq '.EndpointsResponse.Item[]|select(.CreationDate | contains("2020-12-18T19:45:48.212Z") | not)|.Id' -r
| xargs -n 1 aws --profile YourProfile pinpoint delete-endpoint --application-id daafcc04059a444aa9363e484113c189 --endpoint-id
{
"EndpointResponse": {
/* nine responses */

[dylan-westbury]

Receiving amplify security notification in AWS console but can't update due to this issue, we must stay on previous version otherwise user will eventually hit 10 endpoints and will no longer receive push notifications.

[joebernard]

@sammartinez Has there been any progress on this "exceeded maximum endpoint per user" issue"?

@dylan-westbury Which version of aws-amplify did you find avoids this issue? I'm using modules now but want to try that package instead.

[dylan-westbury]

Hi @joebernard

We upgraded aws-amplify and noticed this issue, so we downgraded back to the last working version we had within the app, which was:

"aws-amplify": "3.3.8",

We no longer received the "exceeded maximum endpoint per user" once we downgraded.