Cli-microsoft365: 'spo file add' fails to upload files larger than 250 MB

[muecs]

When trying to upload a file of ~300 MB size using spo file add I get the following error message:

{
    "odata.error": {
        "code": "-1, Microsoft.SharePoint.Client.InvalidClientQueryException",
        "message": {
            "lang": "en-US",
            "value": "The request message is too big. The server does not allow messages larger than 262144000 bytes."
        }
    }
}

It appears that files of 250 MB or more can't be uploaded in the body of a single request, as done currently.

In old REST API documentation there is mention of endpoints StartUpload, ContinueUpload, FinishUpload and CancelUpload which allow for chunked file uploads.

Would it be possible to switch to a chunked upload approach for large files? Or are there other methods to accomplish this using the CLI?

[VelinGeorgiev]

Thanks for your input @muecs!

This is interesting, at the moment there aren't other options for file add in the CLI, but I will try to investigate and enhance it. I will turn that issue into enhancement request so we bring the option to upload files bigger than 250 mb.

I will dedicate some time in the next days to investigate what options we have (including your suggestion), then we can bring change in the CLI.

In addition: We are using the newest SharePoint apis for copy and our expectation was that this api can handle any load since it is decentralized and lives in Azure. However, this seem not to be the case so we can investigate and see how to enhance the CLI.

[muecs]

Thank you Velin, that sounds great.

The Microsoft.SharePoint.Client.dll that you refer to in your blog post seems to offer methods for chunked uploads matching the REST API docs, so these endpoints might still be supported.

[VelinGeorgiev]

@muecs, I did a quick research. It seems that is the preferred way to do it even from the modern UI of the Document libraries

So, my suggestion would be the file add command to check the file size before any other operations and if it is bigger than 250mb to execute the StartUpload(GUID, stream1),ContinueUpload(GUID, 10 MB, stream2),FinishUpload(GUID, 30 MB, stream4).

This is a change that can be done and I can pick it up when I have free time, _but of course we are open for new contributions just in case you would like to help_. :)

[muecs]

Thanks @VelinGeorgiev for looking into this. I'm not awfully familiar with the stack but I'll see what I can do.

[VelinGeorgiev]

That is great @muecs ! Thank you! I will assign that to you then (no rush, no deadlines) . Feel free to ask questions of any kind, I would be happy to help.
The stack is nodejs + typescript + bunch of other js libraries. You can find some useful information on how to get started here: Getting started.
There is also a gitter channel https://gitter.im/office365-cli/cli for more realtime communication.

Just additional note is that you should branch out of the dev branch and because it is v2.0 (master is still v1.23) there are some differences the way you login and the way you execute commands. The docs for v2.0 are here: https://github.com/pnp/office365-cli/tree/dev/docs/manual/docs

[muecs]

@VelinGeorgiev, sorry for the radio silence, I've only just now gotten around to working on this.

Just to give you an update, I've got chunked file uploads implemented (3acc20b0ee8b36111cd88dd840df271415a99711) and been running a couple manual tests successfully. I still need to look into updating the unit tests before I can open a PR.

Do you have an ETA on a v2 release yet?

[VelinGeorgiev]

That is great @muecs, thank you ! ETA on v2 release is September. @waldekmastykarz when in September? First Sunday of Sept ? :D

[waldekmastykarz]

😄

[waldekmastykarz]

Sorry for chiming in late and I hope it doesn't change things too much, but is there a downside to using one upload method no matter the file size as opposite to checking the file size and using either one or the other? Using the same method for all uploads would simplify our code and maintenance of the command.

[muecs]

I don't mind, it's a trivial change. The only downside I can see is that we'd always be using at least 2 API calls rather than just one, even for the tiniest file. But since I've implemented the StartUpload call to have no payload (rather than the first chunk), at least there is immediate feedback whether an upload is actually permitted/possible.

[VelinGeorgiev]

I would also recommend keep it to one upload method to keep less code in the command. I know that I initially said to be a check (up in the comments). I now realized that the less code we have, the more easily to read and maintain is.

[VelinGeorgiev]

This could be potential issue: https://github.com/SharePoint/sp-dev-docs/issues/2813
So perhaps staying with old approach for small files can reduce the issues raised on our site. @waldekmastykarz what do you think?

[waldekmastykarz]

Hmmm, good catch @VelinGeorgiev. If there is no one API that supports everything then I'm afraid we'd need to settle on using two separate calls. It will complicate the implementation but in the end will make things better for the user.

[garrytrinder]

Hey @muecs 👋🏻 are you still working on this?

[muecs]

@garrytrinder Not at present, although I have been using chunked uploads in production for over two months now. To open a PR only tests are still missing which I need to make time for.

[garrytrinder]

Thanks @muecs 👍🏻

Is your branch available for someone to potentially pick up, if you aren’t able to find the time to finish the tests?

[muecs]

@garrytrinder Of course; my work so far is on this branch: https://github.com/muecs/office365-cli/commits/file-add-chunking

[VelinGeorgiev]

Hey guys, I am working on this to finish it.