Microsoft-graph-docs: Throttling when batching

[svrooij]

Where can I find the throttling specifications for bulk changing calendars?

We created an multi-tenant application with client credentials that uses the batch endpoint to change user calendars. Until recently we could combine up to 15 changes in one batch and they would all complete successfully.

This 15-requests limit is discussed multiple times:

• https://github.com/microsoftgraph/microsoft-graph-docs/issues/4366
• https://stackoverflow.com/questions/56873802/microsoft-graph-api-batch-limit?noredirect=1#comment109103171_56873802

But recently we found that we can only combine up to 4 change requests when changing calendars.
All other requests in the batch fail with a 429 error (indicating we are throttled).

In this case it seems that the old (unsupported) Outlook rest API, was actually better, https://docs.microsoft.com/en-us/outlook/rest/compare-graph#batching

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 8df32e5f-8ccb-0c93-28d9-02392f9c7256
• Version Independent ID: 4aadd084-fb61-e89b-f63f-3b71622636b7
• Content: Microsoft Graph throttling guidance - Microsoft Graph
• Content Source: concepts/throttling.md
• Product: non-product-specific
• Technology: microsoft-graph
• GitHub Login: @baywet
• Microsoft Alias: MSGraphDocsVteam

[svrooij]

Hi Vincent (@baywet),
do you know if we can expect feedback on this documentation issue?
Thanks!

[baywet]

Hey Stephan,
I don't have a definitive answer at the moment but I've been looking into it.
It seems that the deployment of throttling limits for Outlook services to v1 is what impacted the behavior and this is why you are hitting the 4 concurrent requests behavior.
Can you confirm you are using the v1 endpoint? And that the behavior changed around the beginning of June?

[svrooij]

We can confirm both.
It was unclear to us that this also affected batching.

Going from EWS (up to 80 changes in one request) to Outlook api (20 requests) to Graph api (first 15) to graph api (4 changes) seems like a limit going the wrong direction.
As of now it doesn't really matter anymore if we even batch the requests, because now the individual requests in a batch are counted, correct?

[baywet]

It looks to me that the individual requests in a batch are counted but I'm trying to get a confirmation on that.

[baywet]

Update: I have confirmation that requests are counted individually at this very point, we're looking at what we can do to address the problem.

[garrettm]

I'm also hitting this problem with https://graph.microsoft.com/v1.0/$batch, it seems to be new. I don't know exactly when it started, (as I took a several month break from working on the API) but I can confirm that I wasn't seeing this behavior several months ago. I had set our batch limit to 16, based on documentation here: https://docs.microsoft.com/en-us/graph/known-issues#limit-on-batch-size

Now it has to be 4, or I get a bunch of 429s.

[garrettm]

I would be surprised if this wasn't affecting a pretty good set of people/users? Is the Graph API not in wide use yet?

[svrooij]

Would it be possible to allow some change with the following restriction:

• All the outlook changes in the batch are against the same mailbox
• Set the X-AnchorMailbox like we used to with Exchange

So I'm not asking for a higher limit when changing items in several calendars, just for an exception when changing several items in the same calendar.

[garrettm]

Any updates here? This is a pretty significant issue for us.

[baywet]

Hi everyone,
This issue is currently on the backburner due to other engineering priorities.
I just want to clarify a potential misunderstanding in terms of processes here. Sorry if my previous comments were misleading. I understand the situation can be frustrating, I personally have been in very similar situations not that long ago from your perspective.

GitHub issues on public documentation are available for the community to help Microsoft identify and address any documentation issue.
Although it is sometimes used to flag some service issues, it is not what it it designed for and a service issue signaled by GitHub issues carries less weight than support tickets, even if it's being pushed by a PM.
If this is a production impacting issue for you, please go ahead and open a support ticket, mention than an incident (ICM) is already open for is and ask the support clerk to link your ticket with the incident: 197442070.
This will help prioritize the investigation (and resolution) of this issue.

[garrettm]

This isn't impacting a production app for us (yet) -- but it is a potential blocker for something we hope to release. Should I still file there @baywet ?

[baywet]

Impacting production or blocking adoption yes

[garrettm]

@baywet do I need a paid account to file a support ticket? I only see Billing/Subscription options. I don't have a paid account (yet). I followed a link to tweet at Azure Support: https://twitter.com/glmitchy/status/1286829243494617088

[baywet]

If your office 365 subscription is a paid one (not a developer tenant), you should be able to open a support ticket from the office 365 admin portal.

[garrettm]

We don't have an office 365 subscription. AzureSupport told me on Twitter that it was an Office 365 issue and I needed to open a ticket at support.microsoft.com, but that looks rather consumer focused, not developer/API. This is all a bit baffling...

[Herz3h]

I can confirm same issue here regarding the 4 batch limit for calendars. Appeared very recently, we have been using this batch method for roughly more than 2 years and obviously noticed the 15 limit first. Now it's 4.

And every time we have opened a support ticket using paid support, it told us we don't have the right subscription or whatever....... Never seen something as complicated as figuring out how to get in contact with microsoft office 365 support..

[garrettm]

Sorry to bother, but are there any updates here? This is really hurting our development.

[baywet]

Hey everyone,
We're still working on it and trying to figure out a solution for this issue.
I'll be changing roles soon, so @dkershaw10 will be taking it from here.
Thanks for your patience!

[garrettm]

Thanks for keeping us updated @baywet!

[garrettm]

@dkershaw10 any updates on timing for resolving this?

[Herz3h]

@dkershaw10 still facing this issue, any news ?

[garrettm]

@dkershaw10 sorry to bother you but is there any news here? This one is pretty important for us.

[svrooij]

@baywet and @dkershaw10 I opened this issue over 3 months ago, hoping to get a quick solution (or explaination if cannot get it resolved).

But after months of hearing that you're working on it, I'm hoping to get a timeframe when this will be fixed (or at least be reverted to the original 15)

[garrettm]

@dkershaw10 sorry to bug, but any estimate on this?

[dkershaw10]

@Harini84 can you take this up with the Calendar engineering team please?

[garrettm]

@dkershaw10 @Harini84 FYI this affects all of Graph, not just calendar. For us it’s mail and calendar.

[Harini84]

@dkershaw10 / @baywet
As mentioned in the above thread this throttling limit is applicable for a mailbox on MS-Graph and not specifically for calendar.

The following is quoted in the blog post:
https://developer.microsoft.com/en-us/graph/blogs/beta-outlook-api-throttling-limits-deploying-to-v1-0-on-microsoft-graph-may-2020/

_Concurrent requests
Each application is limited to 4 concurrent requests per mailbox._

In documentation, the Outlook service limits are specified. Kindly refer the same:
https://docs.microsoft.com/en-us/graph/throttling#outlook-service-limits

_JSON batching allows you to optimize your application by combining multiple requests into a single JSON object. Requests in a batch are evaluated individually against throttling limits and if any request exceeds the limits, it fails with a status of 429 and an error similar to the one provided above. The batch itself fails with a status code of 424 (Failed Dependency). It is possible for multiple requests to be throttled in a single batch. You should retry each failed request from the batch using the value provided in the retry-after response header from the JSON content._

[Herz3h]

After further testing with creating Outlook events using batch endpoint, I suspect there is another limitation on how many batches you can send in X seconds. For instance I had to add 5 sleep between each batch of 4 requests, for the ApplicationThrottled error to not appear: Application is over its MailboxConcurrency limit.

I don't see this documented anywhere, could we get more info on this ?