Microsoft-graph-docs: My system stopped receiving notifications

[raimundasm]

Hello,
Subscriptions for certain inboxes were created long ago, and my system successfully received notifications about received emails,
here's the list of my subscriptions
https://graph.microsoft.com/v1.0/subscriptions/
{
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#subscriptions",
"value": [
{
"id": "ef4d00ca-cf0c-4da6-bded-1ab47295e3f5",
"resource": "users/622eee14-9413-49e3-ac53-10148eba09be/messages",
"applicationId": "281e7aa8-7ef8-4a02-9195-83ffb149d95c",
"changeType": "created",
"clientState": null,
"notificationUrl": "https://...",
"expirationDateTime": "2019-12-03T23:00:03.0238504Z",
"creatorId": "2ca67f35-32ba-4ef5-99ab-6686a93e0007"
},
{
"id": "6f056383-df34-47b5-aaf2-dce3fca1c511",
"resource": "users/3ccbd0c8-f11e-4bac-99df-27b78be74aaa/messages",
"applicationId": "281e7aa8-7ef8-4a02-9195-83ffb149d95c",
"changeType": "created",
"clientState": null,
"notificationUrl": "https://...",
"expirationDateTime": "2019-12-03T23:00:01.4332477Z",
"creatorId": "2ca67f35-32ba-4ef5-99ab-6686a93e0007"
}
]
}

but starting today (2019-12-02) or few days before system stopped receiving notifications.
Then I've created subscription with my test account
subscription was created successfully,
calling https://graph.microsoft.com/v1.0/subscriptions/, I've received response
{
"@odata.context": "https://graph.microsoft.com/v1.0/$metadata#subscriptions",
"value": [
{
"id": "d74be81b-97cd-41f7-8151-98ba7dd7450a",
"resource": "users/ebaeb8ab-44c1-40e1-8725-635759c035db/messages",
"applicationId": "991ead1f-14b0-4934-a471-c749edcbb352",
"changeType": "created",
"clientState": null,
"notificationUrl": "https://63efa1ea.ngrok.io/api/Core/MailNotification/Listen",
"expirationDateTime": "2019-12-04T09:29:19.4158476Z",
"creatorId": "fc01f0e2-2b2c-4dc1-a18c-75529fb75953"
}
]
}

I've tried sending email to mailbox, mentioned above, but still I receive no notifications.
Does something changed to Ms Graph? Is there a way to troubleshoot this problem, can I see some logs in azure portal or somewhere else, did MS Graph tried to send me a notification. Or it just suddenly stopped working?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: bc1ec710-fff6-0690-7eb3-16b9773145ef
• Version Independent ID: c4a14ee1-f979-8409-4481-45272b35d965
• Content: Set up notifications for changes in user data - Microsoft Graph
• Content Source: concepts/webhooks.md
• Product: non-product-specific
• Technology: microsoft-graph
• GitHub Login: @piotrci
• Microsoft Alias: MSGraphDocsVteam

[ana2019-swiss]

Hello,

Today our WebHook slso stopped receiving notifications for calendar events. Everything worked fine for months now without problem. Subscriptions are live still but no notification received.

Any hrlp appreciated

[DenitsaGencheva]

The same thing here! We had a similar issue in May covert in here: https://github.com/microsoftgraph/microsoft-graph-docs/issues/4644
So the push notifications fail from time to time, we have a backup, but still, a lot of users are complaining about the delay.

Please, someone, take a look into this!

[raimundasm]

The same thing here! We had a similar issue in May covert in here: #4644
So the push notifications fail from time to time, we have a backup, but still, a lot of users are complaining about the delay.

Please, someone, take a look into this!

It's not the same issue, in #4644 problem was that .../subscriptions/ endpoint returns empty list although subscriptions were created and could be retrieved one by one, via .../subscriptions/{id}.
Current problem is that subscription is successfully created, but I receive no notifications.

[cv711]

We have been experiencing the same issue, no notifications for our email subscriptions, for the last 12 hours today. This incident also happened for 5 hours last Friday. Besides fixing it urgently could we add some kind of status page to be able to confirm it faster next time?

[ckuklaImpact]

We have been dealing with this issue as well. Not exactly sure when it started. We thought it was just today, but it may have been going on for a while now based off of some of these other reports.

[fangzhoulu-hiretual]

Same thing here. As we are heavily relying on the webhook feature, please post some status for fixing, please!

[mfulan]

Same for our client intranet using webhooks. Please post here if you find out details from MS Support

[ocdsoft]

We're experiencing the same issue.

[sidxyz]

I have a subscription on any changes in the user resource and even I am not receiving any notifications for the same. I tried deleting my existing subscriptions and creating new ones but still the same case in the past 24 hours.

[JordanW7]

@piotrci This outage issue is reasonably urgent and effecting services for a number of people, is it possible to please receive at least an acknowledgment of the issue?

[raimundasm]

1 day passed and system started to receive notifications again. I did nothing to my code. It just started to work again.

It is sad that there is no response from Microsoft.

[JordanW7]

@raimundasm please leave this open. This is still affecting us.

[raimundasm]

Does someone know if there is some health status about Ms Graph like this?

[sidxyz]

@raimundasm : I checked on my systems, we are still not receiving any notifications.
I even tried deleting my old subscription and made a new subscription to test.

This was working fine a couple of days ago. Request you to kindly re-open this issue.
Thanks.

[sidxyz]

@piotrci : This issue caused major embarrassment to my team today who were demoing an integration to one of our clients.
We had to spend some time explaining this to them, it was quite difficult considering they belong from a non technical background.

Could you please provide some status updates? or even an ETA so that we can communicate further ?

[cv711]

@raimundasm can you please reopen the issue, this hasn't been resolved for us yet.

[afkirzak]

The same issue here. I'm not able to receive notifications even with existing and new subscriptions.

[DenitsaGencheva]

Currently, we receive some notifications, but with great delay.
For example - the message was received 13:21 UTC, but the notification was sent at 15:48 UTC.

[rsweetland]

We were also impacted. This issue was made worse by the fact it is a silent failure. Webhooks just stopped arriving.

I made a suggestion on #1726 to improve status monitoring (which really should be re-opened), and left fruitless fruitless trail of debug attempts on Stack Overflow in case it helps anyone else isolate another issue like this.

[mccainz]

Same issue, no notifications at all or ridiculous delays of 6-10 hours for those that do make it to us. Most are still just dropped. Outage seemed to start around 13:00 CST on Dec 30th.

[anoopnair]

Identical situation here too - have not received any notifications at all since Friday / Saturday with no changes made to the application.

Interestingly, it may be that some O365 domains in Europe are sending webhook notifications, but surely are not seeing any for users in North America.

[JordanW7]

I've received a response from Microsoft Graph Team via Twitter, who have advised to fill out a Support Request on the Azure Portal Help+Support page:

I suggest that everyone still with problems raises a request on this page.

[levb]

Since I was in the middle of development, and since the failures were intermittent, I ended up wasting a full day chasing non-existing issues in my own code. Please acknowledge and fix the issue, or provide a workaround.

[JordanW7]

Issue has now resolved for us at 10pm GMT, pretty much exactly 48 hours after the issue started.

[anoopnair]

Webhook notifications are being received again now. However, it would be good to have a _Health Status_ page as suggested in https://github.com/microsoftgraph/microsoft-graph-docs/issues/1726 to indicate whenever there are issues.

[piotrci]

We identified a degradation of service that was affecting Microsoft Graph change notifications (aka webhooks), mostly for Outlook resources (messages, events, etc.). The issue lasted for around 48 hours. We are still investigating the details of the root cause - it was related to the internal queuing mechanism that change notifications use when publishing to your apps. Once fully known, we will take steps to make sure the system is more resilient to such problems in the future.

We apologize for any inconvenience caused to you.

[MalathiBa-Kore]

I am facing the same issue.What can i do to resolve?Please suggest me.Thanks in advance.

[levb]

I'd like to 👍 the recommendation for a Health Status page. We are currently developing an Office Calendar plugin for Mattermost, and would like to rely on the Event subscriptions to provide timely change notifications to the users.

The (recent?) SLA issues are understandable considering the tremendous growth of the Graph APIs, but the users and customer support must have the clarity on whether the underlying Graph functionality is actually healthy.

[sidxyz]

I am still not receiving any notifications.
I did try using the beta subscription api as well but no luck.

I am now changing my approach to polling each resource on a timed frequency(mostly going to rely on Delta query) to achieve the desired functionality.
If anybody comes to know anything about any update on health of Subscription/Webhook, please reply to this thread or email me at [email protected].

Thanks!