Onedrive-api-docs: nextLink Bug in Delta API: Sync Changes

[GaganLohia]

When in Delta API(https://graph.microsoft.com/v1.0/me/drive/root/microsoft.graph.delta(token='someValue')?token=latest) we try to give a top parameter in the request we get back a list of results with the nextLink property that should point us to the next set of changes from the drive. But when we try to request using that link the API returns the same set of data again and again even though it has a new nextLink with a new skipToken every time.

This Bug is there for personal accounts. I am not sure if the same is the case with Business Accounts.

request-id: e2ff45b9-4b07-46ad-b56d-61f443edfc40
client-request-id: e2ff45b9-4b07-46ad-b56d-61f443edfc40
Date: Fri, 15 Jan 2021 07:36:09 GMT

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: f139d8af-bd5e-3289-daec-a101e7071fbf
• Version Independent ID: 69ddaa8b-120b-14e7-643e-dfa921d196fb
• Content: Sync the contents of a drive - OneDrive API - OneDrive dev center
• Content Source: docs/rest-api/api/driveitem_delta.md
• Product: onedrive
• GitHub Login: @rgregg
• Microsoft Alias: rgregg

[msftbot[bot]]

Thank you for your contribution to OneDrive API Docs. We will be triaging your incoming issue as soon as possible.

[omer-chase]

We are experiencing the same problem for at least ~100 OneDrive Personal Account users that are integrated into our system.
I think it may relate to #1408, Cause we are getting those URLs for those users.
We were able to continue syncing their data by fixing the malformed URL by replacing delta\(.+\) with delta

[laurent22]

We are having the same issue in our app - https://github.com/laurent22/joplin/issues/4305#issuecomment-761906930

---

What we know so far:

• The problem appears to have started on 5 Jan 2021, based on this thread, and another thread on the forum
• It's most likely not a bug in Joplin because users have reported it with both the latest version 1.6.6, and older versions 1.5, which for sure never had any issue.
• The delta link returned by the OneDrive API appears to be invalid - it keeps returning changes and never reaches the end of it. For example, it might return 16000 changes, even though there's only 6000 files on OneDrive. Eventually OneDrive will throw a bunch of errors, like timeout, "not found" (for a file that's definitely there) or non-sensical HTML (even though that's supposed to be a JSON API), and Joplin will stop the sync process.

What to do about it?

On our side, it doesn't look like there's anything we can do. The sync algorithm has been working fine for years, and it's not clear there's anything we can change to make it work again with OneDrive. So we can only hope this is a bug in OneDrive and that Microsoft will fix it.

If it doesn't happen we might have to discontinue OneDrive support as we can't reasonably propose it to our user as it's a completely unusable at the moment.

[abraunegg]

@laurent22

On our side, it doesn't look like there's anything we can do.

Actually there is something Joplin developers can do, as per https://github.com/OneDrive/onedrive-api-docs/issues/1408#issuecomment-740819783

The solution that Joplin needs to implement is this:

• If the provided 'deltaLink' / 'nextLink' URL generates a HTTP 400 (Bad Request), then, Joplin should retry the query without the bad provided URL.

Right now, it appears that Joplin is expecting a URL in the API response and using that URL, without any return validation as to if the URL itself generates a bad response.

Yes - the API is again broken, but it would appear that Joplin also is broken because it does not handle the error response well.

This is how this was solved for the Linux OneDrive Client. Currently, users who use the Linux OneDrive Client are impacted by this 'bug', but are not affected because the client application gracefully handles the 400 response and queries the data again without the bad URL.

[abraunegg]

@JeremyKelley

Can you please investigate this issue

[laurent22]

If the provided 'deltaLink' / 'nextLink' URL generates a HTTP 400 (Bad Request), then, Joplin should retry the query without the bad provided URL.

If the provided URL is invalid, it's not clear how the app can retry? If you mean by starting over from the beginning, that's what we do on an unhandled error.

But anyway the issue here is that the delta link doesn't actually generate a 400 - it keeps returning valid data and new delta links, all valid, except that following these links never ends. I've checked the links and they change on each delta call, so it's not some buggy behaviour of the app that would keep fetching the same URL. It's using what OneDrive provides and that appears to be invalid.

[JeremyKelley]

Tagging the team to take a look

[GaganLohia]

If the provided 'deltaLink' / 'nextLink' URL generates a HTTP 400 (Bad Request), then, Joplin should retry the query without the bad provided URL.

If the provided URL is invalid, it's not clear how the app can retry? If you mean by starting over from the beginning, that's what we do on an unhandled error.

But anyway the issue here is that the delta link doesn't actually generate a 400 - it keeps returning valid data and new delta links, all valid, except that following these links never ends. I've checked the links and they change on each delta call, so it's not some buggy behaviour of the app that would keep fetching the same URL. It's using what OneDrive provides and that appears to be invalid.

Yes as @laurent22 mentioned it keeps returning the valid data again and again with new links that point to the same data.

[zachzi]

The change that caused this behavior has been disabled, and this API is working again.

[ankittoshniwal]

@zachzi can you share details on what was the cause that led to this behavior? BTW we are still seeing this issue in our environment and for business accounts.

[abraunegg]

@JeremyKelley , @zachzi
Reports are still occurring that this issue is impacting API consumers across many ecosystems.

Can you please investigate further and provide some feedback.

[Konstantin-Margorin]

Hello

I'm from Acronis. We provide Microsoft 365 backup solution.
https://www.acronis.com/en-us/solutions/backup/office-365/

Just want to report that both personal and business drives are affected by the issues right now exactly with the same symptoms as described here https://github.com/OneDrive/onedrive-api-docs/issues/1424#issue-786687272

Personal drives were affected starting from 5 Jan 2021. Business drives affected from 22 Jan 2021.

Please ask me if you need more details.

[JeremyKelley]

Are people still seeing this against OneDrive consumer this morning? We're working on a fix for Business now and would appreciate if someone is still seeing the problem on their consumer accounts after the change we made on Friday.

Thanks!

[abraunegg]

@JeremyKelley
The issue still persists: https://github.com/OneDrive/onedrive-api-docs/issues/1408#issuecomment-766845886

With Business accounts, this is generating a 504 error at the moment - so the impact is increasing rapidly.

[JeremyKelley]

Late afternoon update, we're continuing to work on fixes for both SharePoint and OneDrive consumer.

We believe that the issues still being experienced are (as in a previous issue) related to URLs having been returned that can now not be processed correctly even after reverting the change that caused them to be emitted. We are working to accept the "bad" URLs (I use that term loosely) to get people moving again. You should find that Delta calls that are "fresh" return links that are currently working, this can be problematic if you have a lot of content to process and a lot of threads already going that you can't necessarily do a full re-scan of. Hence us working to get them processed with the links you already have.

[abraunegg]

@JeremyKelley
Many thanks for the update and providing details as to how the issue is being worked around.

[JeremyKelley]

We are getting the SharePoint Online and OneDrive for Business fixes rolled out. I'm curious if people are still significantly impacted on the OneDrive personal side? We should be pushing out new URLs that work although anyone holding links from the broken period could still be in an infinite loop.

[GaganLohia]

@JeremyKelley
OneDrive personal users are still going into an infinite loop if we are using nexLink with a _top_ param in the request.

[westleyMS]

@JeremyKelley does this mean all clients using this sync will have to do a full sync?

[shayan-brcm]

Hi @JeremyKelley we are also facing this issue. Is there any ETA for this fix or any work around?

[GaganLohia]

Hey @JeremyKelley
Any updates on this issue?

[westleyMS]

Just to be clear, the issue was fixed last week, and if you get a new token delta?token=latest, it works fine.
We are only looking for a way to get the data between when it broke and it was fixed.

[shayan-brcm]

@westleyMS thanks for the update. Is existing token who was stuck will work or not? or do we need a new token to make it work?

[westleyMS]

@shayan-brcm you need a new token.