Onedrive-api-docs: Sharepoint / O365: Session uploaded file size reported by MS Graph does not equal local file size

[abraunegg]

Category

• [ ] Question
• [ ] Documentation issue
• [X] Bug

Expected or Desired Behavior

OneDrive should report the correct file size when uploaded

Observed Behavior

When uploading a file via a session upload, to an O365 sharepoint site, the file size of the object returned does not match the actual file which was uploaded.

Files that appear to report incorrect sizes are:

• PDF
• MS Office Documents
• HTML files

Files that do not appear to report incorrect sizes are:

• Text files

When uploading to a standard OneDrive Business Account, this behavior does not occur.

Steps to Reproduce

• Upload a file via API using a session upload
• Review session response for file upload completion and note the 'size' value
• Compare the 'size' value to the actual file size

For further details, refer to https://github.com/abraunegg/onedrive/issues/205 which provides additional diagnostic information

[abraunegg]

@ificator
Any update on this issue?

[abraunegg]

@ificator
Updated from an affected user:

Background: I just started using this client two weeks ago when I got a new machine at work. I saw this issue the very first day I tried using it on a plain .txt file. Today the problem came back in an interesting way.

I have a file called 2018-10-12 Burn Down Rate.xlsx that I initialized locally and then uploaded using this tool. It was one of 6 Excel sheets that I initialized today, but the only one to throw the file size does not match error.

My local copy of this file is 20,143 bytes. The Sharepoint Documents repository shows 25.6 KB with no obvious way of seeing the exact number of bytes. However, when I use my web browser to download the file, it comes out to 26,194 bytes! The visible content within this Excel sheet is identical, but the files themselves must be different.

A cursory look shows that Sharepoint is modifying the files after they are uploaded. I extracted each of the .XLSX files as ZIP archives and I can see that Sharepoint has injected a new file at the path docProps/custom.xml that does not exist in my original upload. There is an entire extra folder named customXml filled with 6 additional files and a subfolder as well.

So Sharepoint is modifying these files after they are uploaded. That's.... interesting. My current workaround is to delete the local file after I see the error and allow it to re-sync the updated copy from Sharepoint. I don't know if that is a sane default but it is the workflow I have adopted.

[abraunegg]

@ificator
Any update on this issue?

[abraunegg]

@ificator, @chackman
Any update on this issue? Files should not be modified after they are uploaded with data. Can this be looked at with some urgency?

[chackman]

Were these files opened or viewed between the times that they were uploaded and then downloaded? Have you checked file history to see if the file has been modified by another application?

Could you please provide information about your original upload request:

• Date (in UTC, please)
• request-id
• SPRequestGuid (for requests made to OneDrive for Business)

[abraunegg]

@chackman

Were these files opened or viewed between the times that they were uploaded and then downloaded? Have you checked file history to see if the file has been modified by another application?

No I do not believe so. The response from OneDrive in the response json is immediate after session upload is complete, and this is where the size difference is being noted. The file is being modified by OneDrive / MS Graph immediately after upload. As the change is size is right after the upload to indicate that the file was successfully uploaded - there is 'zero' chance something else is modifying this other that the OneDrive backend.

Could you please provide information about your original upload request:

• Date (in UTC, please)
• request-id
• SPRequestGuid (for requests made to OneDrive for Business)

Will ask the users to generate some requests using additional debugging to capture this.

[abraunegg]

@chackman
Tracker case against 'onedrive' client application for Linux: https://github.com/abraunegg/onedrive/issues/205

[abraunegg]

@chackman
Below are 2 different users logs with the data requested:

Log 1: https://pastebin.com/SNCnpM1n
Log 2: https://pastebin.com/57B0e0Vh

[chackman]

Thanks; I'm waiting for more information from a partner team about this issue.

[abraunegg]

@chackman
Any update here - Microsoft should not be modifying user's files when uploaded to OneDrive service. This is impacting a significant number of users, using various Linux clients to sync data to OneDrive.

[chackman]

I'm still waiting for more information from a partner team about this issue.

[abraunegg]

@chackman
Can this please be somehow escalated? User files should not be modified when there is zero information that this would be occuring - and by doing so this breaks user trust in the service to keep data safe, secure and in its original format as the service might be used for archival purposes.

[JeremyKelley]

After doing a little research this is normal behavior for SharePoint and has been since at least SharePoint 2010. When uploading a file that can contain metadata SharePoint will associate some metadata from the library the file is uploaded to directly in the file. Office files are probably the most typical files where this will happen but it's not unreasonable to assume that this will happen for other rich file types as well.

Since basic text files don't have in-file metadata they don't see this behavior.

If you're syncing files to a SharePoint document library using the sync client the initial sync will have the enriched file sync'ed rather than the original.

[abraunegg]

After doing a little research this is normal behavior for SharePoint and has been since at least SharePoint 2010. When uploading a file that can contain metadata SharePoint will associate some metadata from the library the file is uploaded to directly in the file. Office files are probably the most typical files where this will happen but it's not unreasonable to assume that this will happen for other rich file types as well.

Whilst it might be considered 'normal behaviour' to Microsoft to modify a file after upload - either by enrichment or other - it is still a file modification that breaks file validation mechanisms to indicate that the file was uploaded successfully. This modification would constitute unacceptable behaviour because it is happening without user knowledge or acceptance.

Just because an uploaded file 'can' be enriched, does not mean that it should be 'automatically' enriched.

Is there any way to disable this 'automatic' SharePoint document enrichment?

[chackman]

This is a feature in SharePoint document libraries that has been around for some time. With respect to validation, have you encountered cases where a file upload appears to have succeeded, but in fact, it did not succeed?

If upload didn't work even though it appeared to work, then please provide more information about the upload that silently failed.

[abraunegg]

This is a feature in SharePoint document libraries that has been around for some time. With respect to validation, have you encountered cases where a file upload appears to have succeeded, but in fact, it did not succeed?

Any file that is successfully uploaded, but modified by Microsoft & file size changed (due to modification or enrichment) when comparing to that of the actual file on my local disk - should in my mind constitute a 'failed upload' as what I 'uploaded' is not what is present on OneDrive - the Microsoft modified version is what exists. If I was then to use any sort of 'checksum' to validate if my files are 'in sync' - I would have different files which would then need to either:

• reupload (which again would get modified)
• replace with MS modified / enriched version .... why would I want a MS modified version of my file?

I would suggest that this 'feature' - despite its 'age' either needs to be clearly documented somewhere, or somehow switched off by default for SharePoint libraries and enabled by some sort API flag if wanted.

[derrix060]

Agree with @abraunegg.

Looks very odd for my can't trust on the host. I want to upload a file and make sure it is the same. If I also have a text file with the checksum, and when download I use the content for this checksum file to detect if I downloaded the exact file that I've upload, won't be possible.

I strongly suggest being able to disable this "feature".

[peter-sabath]

This "enriched" information should be stored in a separate/hidden data stream or blob, so "User" files are never changed.

I took a look what changes on HTML files and found added "mso:" markup in the file.
Every Developer get crazy if he finds such in handwritten file(s).

[chackman]

Unfortunately, as stated earlier, this is how the document library features work.
We will look into documentation feedback; however, UserVoice is the appropriate place to suggest feature enhancements and improvements.

[0x3333]

Does this behavior happen on a user's OneDrive for Bussiness?