Uppy: AwsS3Multipart + Drive/Dropbox stalls after ~21mb

[yonahforst]

Uppy v0.30.4 on the client (React) and @uppy/companion 0.17.4 on the server (Nodejs+Heroku).

S3 multipart uploads work like a charm when uploading local files, however when uploading files from Drive or Dropbox, uploads stall after about 21mb.

No errors in the console, nor on the server.

Just the following WS message:

{"action":"progress","payload":{"progress":"17.37","bytesUploaded":20971520,"bytesTotal":120758940}}

[andrebastie]

I'm experiencing this issue as well

[JuanValencia]

I'm seeing a similar issue except that the upload finishes successfully, but with a partial upload. Looking at S3, I see the same file uploaded several times, but with different sizes:
2019-07-26 14:12:15 44020 1564175531587.595481.Getting started
2019-07-26 14:13:14 47901 1564175590968.507706.Getting started
2019-07-26 14:15:56 15134 1564175753296.786573.Getting started
2019-07-26 14:33:22 14750 1564176796862.861128.Getting started
2019-07-26 14:39:44 118981 1564177180441.140060.Getting started
2019-07-26 14:52:15 25862 1564177931471.697703.Getting started
2019-07-26 14:58:48 16160 1564178324452.946449.Getting started
2019-07-26 15:12:38 72621 1564179154172.450885.Getting started
2019-07-26 15:20:45 72608 1564179641379.45989.Getting started
2019-07-26 15:23:38 15136 1564179814751.143882.Getting started

If I trace what's happening, essentially, the companion/lib/server/uploader is creating the multipart uploader to S3 as soon as the first chunk is downloaded from drive/dropbox.

uploadS3() creates "tailReadStream" (file) which is what the multipart uploader uses. However, it closes the tailReadStream as soon as the download stream from drive/dropbox is closed. This essentially finishes the upload even though the stream is only partially completed.

[JuanValencia]

to confirm, pausing the @uppy/fs-tail-stream in the debugger at .once('end', function (e) { pos = bytesRead + (options && options.start || 0); the upload finishes. Continuing the run after the upload is complete has a successful upload... Now if only I could figure out how to make that work consistently.

[huydod]

I'm facing the same issue with uppy 1.5.2 on client and uppy/companion 1.5.0 on server.
It just uploads a part of the downloaded file to S3. Below is the logs from companion server

Welcome to Companion!
Listening on http://0.0.0.0:3020
companion: 2019-10-23T11:07:13.561Z [debug]  Instantiating uploader.
companion: 2019-10-23T11:07:13.563Z [debug]  Waiting for socket connection before beginning remote download.
companion: 2019-10-23T11:07:13.563Z [debug] 2825184b uploader.socket.wait waiting for connection
companion: 2019-10-23T11:07:13.594Z [info] socket.connect connection received from 2825184b-5390-44dc-b806-be2cecdcbe89
companion: 2019-10-23T11:07:13.594Z [debug]  Socket connection received. Starting remote download.
companion: 2019-10-23T11:07:14.178Z [debug] 2825184b uploader.download.progress 757 bytes
companion: 2019-10-23T11:07:14.182Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.182Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.182Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.182Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.183Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.183Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.183Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.183Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.183Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.183Z [debug] 2825184b uploader.download.progress 14327 bytes
companion: 2019-10-23T11:07:14.201Z [debug] 2825184b uploader.download.progress 15117 bytes
companion: 2019-10-23T11:07:14.206Z [debug] 2825184b uploader.download.progress 16474 bytes
companion: 2019-10-23T11:07:14.206Z [debug] 2825184b uploader.download.progress 17831 bytes
companion: 2019-10-23T11:07:14.206Z [debug] 2825184b uploader.download.progress 19188 bytes
companion: 2019-10-23T11:07:14.209Z [debug] 2825184b uploader.download.progress 20545 bytes
companion: 2019-10-23T11:07:14.209Z [debug] 2825184b uploader.download.progress 21902 bytes
companion: 2019-10-23T11:07:14.211Z [debug] 2825184b uploader.download.progress 23259 bytes
companion: 2019-10-23T11:07:14.211Z [debug] 2825184b uploader.download.progress 24616 bytes
companion: 2019-10-23T11:07:14.213Z [debug] 2825184b uploader.download.progress 25973 bytes
companion: 2019-10-23T11:07:14.213Z [debug] 2825184b uploader.download.progress 27330 bytes
companion: 2019-10-23T11:07:14.214Z [debug] 2825184b uploader.download.progress 28687 bytes
companion: 2019-10-23T11:07:14.215Z [debug] 2825184b uploader.download.progress 30044 bytes
companion: 2019-10-23T11:07:14.217Z [debug] 2825184b uploader.download.progress 31401 bytes
companion: 2019-10-23T11:07:14.217Z [debug] 2825184b uploader.download.progress 32758 bytes
companion: 2019-10-23T11:07:14.219Z [debug] 2825184b uploader.download.progress 34115 bytes
companion: 2019-10-23T11:07:14.219Z [debug] 2825184b uploader.download.progress 35472 bytes
companion: 2019-10-23T11:07:14.221Z [debug] 2825184b uploader.download.progress 36829 bytes
companion: 2019-10-23T11:07:14.221Z [debug] 2825184b uploader.download.progress 38186 bytes
companion: 2019-10-23T11:07:14.235Z [debug] 2825184b uploader.download.progress 39543 bytes
companion: 2019-10-23T11:07:14.235Z [debug] 2825184b uploader.download.progress 44971 bytes
companion: 2019-10-23T11:07:14.235Z [debug] 2825184b uploader.download.progress 44971 bytes
companion: 2019-10-23T11:07:14.235Z [debug] 2825184b uploader.download.progress 44971 bytes
companion: 2019-10-23T11:07:14.235Z [debug] 2825184b uploader.download.progress 44971 bytes
companion: 2019-10-23T11:07:14.235Z [debug] 2825184b uploader.download.progress 46328 bytes
companion: 2019-10-23T11:07:14.239Z [debug] 2825184b uploader.download.progress 47685 bytes
companion: 2019-10-23T11:07:14.239Z [debug] 2825184b uploader.download.progress 50599 bytes
companion: 2019-10-23T11:07:14.239Z [debug] 2825184b uploader.download.progress 50599 bytes
companion: 2019-10-23T11:07:14.239Z [debug] 2825184b uploader.download.progress 50599 bytes
companion: 2019-10-23T11:07:14.239Z [debug] 2825184b uploader.download.progress 51956 bytes
companion: 2019-10-23T11:07:14.241Z [debug] 2825184b uploader.download.progress 53313 bytes
companion: 2019-10-23T11:07:14.241Z [debug] 2825184b uploader.download.progress 54670 bytes
companion: 2019-10-23T11:07:14.243Z [debug] 2825184b uploader.download.progress 56027 bytes
companion: 2019-10-23T11:07:14.243Z [debug] 2825184b uploader.download.progress 57384 bytes
companion: 2019-10-23T11:07:14.245Z [debug] 2825184b uploader.download.progress 58741 bytes
companion: 2019-10-23T11:07:14.245Z [debug] 2825184b uploader.download.progress 60098 bytes
companion: 2019-10-23T11:07:14.246Z [debug] 2825184b uploader.download.progress 61455 bytes
companion: 2019-10-23T11:07:14.249Z [debug] 2825184b uploader.download.progress 62812 bytes
companion: 2019-10-23T11:07:14.249Z [debug] 2825184b uploader.download.progress 65614 bytes
companion: 2019-10-23T11:07:14.249Z [debug] 2825184b uploader.download.progress 65614 bytes
companion: 2019-10-23T11:07:14.249Z [debug] 2825184b uploader.download.progress 65614 bytes
companion: 2019-10-23T11:07:14.250Z [debug] 2825184b uploader.download.progress 66971 bytes
companion: 2019-10-23T11:07:14.250Z [debug] 2825184b uploader.download.progress 68328 bytes
companion: 2019-10-23T11:07:14.252Z [debug] 2825184b uploader.download.progress 69685 bytes
companion: 2019-10-23T11:07:14.252Z [debug] 2825184b uploader.download.progress 71042 bytes
companion: 2019-10-23T11:07:14.254Z [debug] 2825184b uploader.download.progress 72399 bytes
companion: 2019-10-23T11:07:14.256Z [debug] 2825184b uploader.download.progress 73756 bytes
companion: 2019-10-23T11:07:14.258Z [debug] 2825184b uploader.download.progress 75113 bytes
companion: 2019-10-23T11:07:14.258Z [debug] 2825184b uploader.download.progress 76470 bytes
companion: 2019-10-23T11:07:14.273Z [debug] 2825184b uploader.download.progress 77827 bytes
companion: 2019-10-23T11:07:14.274Z [debug] 2825184b uploader.download.progress 87326 bytes
companion: 2019-10-23T11:07:14.274Z [debug] 2825184b uploader.download.progress 87326 bytes
companion: 2019-10-23T11:07:14.274Z [debug] 2825184b uploader.download.progress 87326 bytes
companion: 2019-10-23T11:07:14.274Z [debug] 2825184b uploader.download.progress 87326 bytes
companion: 2019-10-23T11:07:14.274Z [debug] 2825184b uploader.download.progress 87326 bytes
companion: 2019-10-23T11:07:14.274Z [debug] 2825184b uploader.download.progress 87326 bytes
companion: 2019-10-23T11:07:14.274Z [debug] 2825184b uploader.download.progress 87326 bytes
companion: 2019-10-23T11:07:14.275Z [debug] 2825184b uploader.download.progress 90040 bytes
companion: 2019-10-23T11:07:14.275Z [debug] 2825184b uploader.download.progress 90040 bytes
companion: 2019-10-23T11:07:14.275Z [debug] 2825184b uploader.download.progress 94111 bytes
companion: 2019-10-23T11:07:14.275Z [debug] 2825184b uploader.download.progress 94111 bytes
companion: 2019-10-23T11:07:14.275Z [debug] 2825184b uploader.download.progress 94111 bytes
companion: 2019-10-23T11:07:14.276Z [debug] 2825184b uploader.download.progress 95468 bytes
companion: 2019-10-23T11:07:14.277Z [debug] 2825184b uploader.download.progress 97025 bytes
companion: 2019-10-23T11:07:14.277Z [debug] 2825184b uploader.download.progress 97025 bytes
companion: 2019-10-23T11:07:14.277Z [debug] 2825184b uploader.download.progress 99739 bytes
companion: 2019-10-23T11:07:14.277Z [debug] 2825184b uploader.download.progress 99739 bytes
companion: 2019-10-23T11:07:14.278Z [debug] 2825184b uploader.download.progress 101096 bytes
companion: 2019-10-23T11:07:14.278Z [debug] 2825184b uploader.download.progress 102453 bytes
companion: 2019-10-23T11:07:14.284Z [debug] 2825184b uploader.download.progress 103810 bytes
companion: 2019-10-23T11:07:14.284Z [debug] 2825184b uploader.download.progress 111952 bytes
companion: 2019-10-23T11:07:14.284Z [debug] 2825184b uploader.download.progress 111952 bytes
companion: 2019-10-23T11:07:14.284Z [debug] 2825184b uploader.download.progress 111952 bytes
companion: 2019-10-23T11:07:14.284Z [debug] 2825184b uploader.download.progress 111952 bytes
companion: 2019-10-23T11:07:14.284Z [debug] 2825184b uploader.download.progress 111952 bytes
companion: 2019-10-23T11:07:14.284Z [debug] 2825184b uploader.download.progress 111952 bytes
companion: 2019-10-23T11:07:14.285Z [debug] 2825184b uploader.download.progress 113309 bytes
companion: 2019-10-23T11:07:14.285Z [debug] 2825184b uploader.download.progress 117380 bytes
companion: 2019-10-23T11:07:14.285Z [debug] 2825184b uploader.download.progress 117380 bytes
companion: 2019-10-23T11:07:14.285Z [debug] 2825184b uploader.download.progress 117380 bytes
companion: 2019-10-23T11:07:14.286Z [debug] 2825184b uploader.download.progress 118737 bytes
companion: 2019-10-23T11:07:14.286Z [debug] 2825184b uploader.download.progress 121451 bytes
companion: 2019-10-23T11:07:14.286Z [debug] 2825184b uploader.download.progress 121451 bytes
companion: 2019-10-23T11:07:14.291Z [debug] 2825184b uploader.download.progress 122808 bytes
companion: 2019-10-23T11:07:14.292Z [debug] 2825184b uploader.download.progress 126879 bytes
companion: 2019-10-23T11:07:14.292Z [debug] 2825184b uploader.download.progress 126879 bytes
companion: 2019-10-23T11:07:14.292Z [debug] 2825184b uploader.download.progress 126879 bytes
companion: 2019-10-23T11:07:14.292Z [debug] 2825184b uploader.download.progress 131150 bytes
companion: 2019-10-23T11:07:14.292Z [debug] 2825184b uploader.download.progress 131150 bytes
companion: 2019-10-23T11:07:14.292Z [debug] 2825184b uploader.download.progress 131150 bytes
companion: 2019-10-23T11:07:14.293Z [debug] 2825184b uploader.download.progress 131150 bytes
companion: 2019-10-23T11:07:14.293Z [debug] 2825184b uploader.download.progress 135221 bytes
companion: 2019-10-23T11:07:14.293Z [debug] 2825184b uploader.download.progress 135221 bytes
companion: 2019-10-23T11:07:14.293Z [debug] 2825184b uploader.download.progress 135221 bytes
companion: 2019-10-23T11:07:14.293Z [debug] 2825184b uploader.download.progress 136578 bytes
companion: 2019-10-23T11:07:14.293Z [debug] 2825184b uploader.download.progress 140649 bytes
companion: 2019-10-23T11:07:14.294Z [debug] 2825184b uploader.download.progress 140649 bytes
companion: 2019-10-23T11:07:14.294Z [debug] 2825184b uploader.download.progress 140649 bytes
companion: 2019-10-23T11:07:14.294Z [debug] 2825184b uploader.download.progress 142006 bytes
companion: 2019-10-23T11:07:14.294Z [debug] 2825184b uploader.download.progress 143363 bytes
companion: 2019-10-23T11:07:14.295Z [debug] 2825184b uploader.download.progress 144720 bytes
companion: 2019-10-23T11:07:14.295Z [debug] 2825184b uploader.download.progress 148791 bytes
companion: 2019-10-23T11:07:14.295Z [debug] 2825184b uploader.download.progress 148791 bytes
companion: 2019-10-23T11:07:14.295Z [debug] 2825184b uploader.download.progress 148791 bytes
companion: 2019-10-23T11:07:14.297Z [debug] 2825184b uploader.download.progress 150148 bytes
companion: 2019-10-23T11:07:14.297Z [debug] 2825184b uploader.download.progress 154219 bytes
companion: 2019-10-23T11:07:14.297Z [debug] 2825184b uploader.download.progress 154219 bytes
companion: 2019-10-23T11:07:14.297Z [debug] 2825184b uploader.download.progress 154219 bytes
companion: 2019-10-23T11:07:14.326Z [debug] 2825184b uploader.download.progress 155576 bytes
companion: 2019-10-23T11:07:14.327Z [debug] 2825184b uploader.download.progress 156933 bytes
companion: 2019-10-23T11:07:14.329Z [debug] 2825184b uploader.download.progress 158290 bytes
companion: 2019-10-23T11:07:14.329Z [debug] 2825184b uploader.download.progress 165275 bytes
companion: 2019-10-23T11:07:14.330Z [debug] 2825184b uploader.download.progress 165275 bytes
companion: 2019-10-23T11:07:14.330Z [debug] 2825184b uploader.download.progress 165275 bytes
companion: 2019-10-23T11:07:14.330Z [debug] 2825184b uploader.download.progress 165275 bytes
companion: 2019-10-23T11:07:14.330Z [debug] 2825184b uploader.download.progress 165275 bytes
companion: 2019-10-23T11:07:14.330Z [debug] 2825184b uploader.download.progress 165275 bytes
companion: 2019-10-23T11:07:14.330Z [debug] 2825184b uploader.download.progress 166632 bytes
companion: 2019-10-23T11:07:14.338Z [debug] 2825184b uploader.download.progress 167989 bytes
companion: 2019-10-23T11:07:14.339Z [debug] 2825184b uploader.download.progress 173417 bytes
companion: 2019-10-23T11:07:14.339Z [debug] 2825184b uploader.download.progress 173417 bytes
companion: 2019-10-23T11:07:14.339Z [debug] 2825184b uploader.download.progress 173417 bytes
companion: 2019-10-23T11:07:14.339Z [debug] 2825184b uploader.download.progress 173417 bytes
companion: 2019-10-23T11:07:14.339Z [debug] 2825184b uploader.download.progress 177488 bytes
companion: 2019-10-23T11:07:14.339Z [debug] 2825184b uploader.download.progress 177488 bytes
companion: 2019-10-23T11:07:14.339Z [debug] 2825184b uploader.download.progress 177488 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 189701 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 196686 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 196686 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 196686 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 196686 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 196686 bytes
companion: 2019-10-23T11:07:14.340Z [debug] 2825184b uploader.download.progress 196686 bytes
companion: 2019-10-23T11:07:14.341Z [debug] 2825184b uploader.download.progress 198043 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 206185 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 206185 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 206185 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 206185 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 206185 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 206185 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 211853 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 211853 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 211853 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 211853 bytes
companion: 2019-10-23T11:07:14.342Z [debug] 2825184b uploader.download.progress 211853 bytes
companion: 2019-10-23T11:07:15.113Z [debug] 2825184b uploader.upload.progress 14327 14327 100.00%

is there anyone has fixed this issue?

[huydod]

@yonahforst @andrebastie @JuanValencia

The problem is read tail stream is closed right after the WriteStream is finished so it push null to the upload stream and finish it while the readChunk function hasn't accomplished reading all of downloaded chunks from the file

So my solution is emit an event eof whenever the read chunk process reached end of file. And once the write stream is finish, add a listener to event eof to close the read tail stream

I also encountered another issue when the file is large enough so the read chunk process pushes a lot of chunks with chunk size is 65536 bytes to upload stream, result in the upload process is stucked at first few parts. I've not figured it out why it stucked but I've fixed it by only push a chunk to upload stream if it's larger or equal to partSize of the s3 upload config.

One another issue is the fs.watch can't recognizes the downloaded is growing up to trigger the readChunk function to uploading the while it's downloading. My solution for this case is create an interval to trigger readChunk function and clear it once the file is finish with change event from watcher of fs.watch

That's all and it works fine to me now. Hope this helpful.

[mihalo]

I'm having the same problem with @uppy/core 1.3.0, @uppy/google-drive 1.3.3.

Companion is running on Node v12.13.0, @uppy/companion 1.6.0

No error messages on client or server.

[yonahforst]

@huydod - it sounds like you patched some Uppy's code. Would you consider opening a PR? 😸

[mihalo]

Any updates on this one? Seems to be same case with the latest versions.

[goto-bus-stop]

not yet sorry :( i'll make this higher priority and try to get it done before next month's release 1.9.

[massens]

I'm have this issue as well

[goto-bus-stop]

Eek! gonna open up a PR today. Sorry that this has sat for so long…

Thanks @huydod for explaining what's going wrong! We'll just drop the parallel download/upload feature for S3 Multipart. I think even if we fix the issue @huydod pointed out, the tail-stream package is still quite brittle (just because fs.watch is that way), and AFAICT sees no production use outside of Companion. We dropped the feature for Tus uploads just before 1.0, and honestly should've done the same for S3 back then. If we just download the entire file before uploading it, we save a bunch of complexity.