Server: Chunked transfers lead to 0 byte files

[ylangisc]

We hit the annoying 0 byte files issue when using a PUT operation with a chunked transfer encoding to modify an existing 0 byte file. After successfully executing the PUT the server replies with a 204 but the file is still empty. Not sure if this is a NC or sabre/dav issue.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @report.csv "http://ubuntu.local/remote.php/webdav/ylatst.txt" -u admin:admin
*   Trying 192.168.48.135...
* TCP_NODELAY set
* Connected to ubuntu.local (192.168.48.135) port 80 (#0)
* Server auth using Basic with user 'admin'
> PUT /remote.php/webdav/ylatst.txt HTTP/1.1
> Host: ubuntu.local
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/7.54.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 204 No Content
< Server: nginx/1.13.8
< Date: Mon, 22 Jan 2018 16:41:33 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=xlmQsVcgE%2FcbE8%2FmB8bvsVG3fGrkDGB%2Fg5wHy5sJqrIjfeVotxGf4US%2BnLdGj6xNmzPjH3NzYgm09Qy%2FnkqKQA59LI9qQEmY0mkZZs17wwCvcubybbqiCS83FHYGdvhe; path=/; HttpOnly
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Set-Cookie: nc_sameSiteCookielax=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Content-Security-Policy: default-src 'none';
< Set-Cookie: oc4nuj3qa5xi=upmvsv2vtom8dg52tn72r3g4a0; path=/; HttpOnly
< Set-Cookie: cookie_test=test; expires=Mon, 22-Jan-2018 17:41:33 GMT; Max-Age=3600
< OC-FileId: 00000180oc4nuj3qa5xi
< ETag: "46ca66c845120b954836fb1dc6fb4af9"
< OC-ETag: "46ca66c845120b954836fb1dc6fb4af9"
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none

I'm using nginx 1.13.8 as a webserver (nginx 1.10.3 doesn't work either). My nginx configuration is quite default. The relevant part:

location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+)\.php(?:$|/) {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        fastcgi_pass      unix:/var/run/php/php7.0-fpm.sock;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
    }

Some observations:

• changing fastcgi_request_buffering off to fastcgi_request_buffering on solves the issue but this is not an option for large streamed files.
• it does happen for files larger than 7kb in my local environment. Smaller files work fine. For servers not being in my local network the files have to be even larger to show the issue. It's much more difficult to replicate the issue in such an environment. Due to bandwidth limitation and latency the server probably has enough time to process the data and a buffer overflow or similar is less likely to happen.

Background: I'm a developer of Mountain Duck which uses the create/update file pattern described here http://sabre.io/dav/0bytes/. According this document nginx in a newer version should work fine. We have several users hitting this issue though.

Steps to reproduce

1. Upload a 0 byte file
2. Try to update the file with chunked transfer encoding (see above for a curl example)

Expected behaviour

Files has updated content and correct size

Actual behaviour

Remote file is not touched at all and you still see 0 bytes

Server configuration

Operating system:
Ubuntu 16

Web server:
nginx

Database:
MariaDB

PHP version:
7.0.22

Nextcloud version: (see Nextcloud admin page)
12.0.4

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh

Where did you install Nextcloud from:
Official download

[ylangisc]

After some more testing I've noticed that it's even worse. It does not matter if you create a 0 byte file prior updating it. Basically all PUT operations with a chunked transfer encoding lead to 0 byte files in low latency and high bandwidth environments for files that are greater than a certain size (see above). In my local environment it's easy to replicate with a simple curl command. E.g.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @testfile.bin "http://ubuntu.local/remote.php/webdav/testfile.bin" -u admin:admin

[nextcloud-bot]

Hey, this issue has been closed because the label stale is set and there were no updates for 14 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[dkocher]

Please reopen this issue.

[Greek64]

I too am having this issue with webdav.
After disabling encryption and sniffing the traffic going to and from the nextcloud webserver (nginx 1.14.0 in my case), I too came to the conclusion, that after a PUT command with chunked transfer encoding, nextcloud responses with 204 no content and creates only 0 byte files.

I noticed this behavior when trying to export my Kodi library via webdav to nextcloud.

I can also provide a sanitized HTTP communication if needed, but I guess that will not help much.

[jospoortvliet]

I just tested on nextcloud 15 with a 700 kb image. 256 kb of the image made it to the server. a 7.5 mb ODP file was also cut in half - 3.3 mb on the server.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @test.odp "http://127.0.0.1/nextcloud-15/remote.php/webdav/test.odp" -u jos:jos
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'jos'
> PUT /nextcloud-15/remote.php/webdav/test.odp HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic am9zOmpvcw==
> User-Agent: curl/7.62.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
< HTTP/1.1 201 Created
< Date: Thu, 03 Jan 2019 11:22:50 GMT
< Server: Apache
< X-Powered-By: PHP/7.3.0
< Set-Cookie: oczswzpgpvdq=9qbdh7bhvlq6s9um76vqfs7dlg; path=/nextcloud-15; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=I1gqnptmzWjmuBJhZqrfPOeNr6LIhTAn6OWTsFxQhflPr2EuxY2LtSmdoIMc0bz4ehfVuMBRmLO%2FZBBQG6FEDfgc0Q7qupNRWiHZPR%2BxESiNJg5Vh1%2Fajn6geMMRJNWv; path=/nextcloud-15; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: no-referrer
< Set-Cookie: nc_sameSiteCookielax=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Set-Cookie: oczswzpgpvdq=pj5kbp17hlf3btfob60d2meh7q; path=/nextcloud-15; HttpOnly
< Set-Cookie: cookie_test=test; expires=Thu, 03-Jan-2019 12:22:50 GMT; Max-Age=3600
< OC-FileId: 00001151oczswzpgpvdq
< Content-Length: 0
< ETag: "7ef58d6f933a4e3387fdf1418f188772"
< OC-ETag: "7ef58d6f933a4e3387fdf1418f188772"
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host 127.0.0.1 left intact

[ldipenti]

I just tested on nextcloud 15 with a 700 kb image. 256 kb of the image made it to the server. a 7.5 mb ODP file was also cut in half - 3.3 mb on the server.

curl -v -X PUT --header "Transfer-Encoding: chunked" -d @test.odp "http://127.0.0.1/nextcloud-15/remote.php/webdav/test.odp" -u jos:jos
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
* Server auth using Basic with user 'jos'
> PUT /nextcloud-15/remote.php/webdav/test.odp HTTP/1.1
> Host: 127.0.0.1
> Authorization: Basic am9zOmpvcw==
> User-Agent: curl/7.62.0
> Accept: */*
> Transfer-Encoding: chunked
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
< HTTP/1.1 201 Created
< Date: Thu, 03 Jan 2019 11:22:50 GMT
< Server: Apache
< X-Powered-By: PHP/7.3.0
< Set-Cookie: oczswzpgpvdq=9qbdh7bhvlq6s9um76vqfs7dlg; path=/nextcloud-15; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=I1gqnptmzWjmuBJhZqrfPOeNr6LIhTAn6OWTsFxQhflPr2EuxY2LtSmdoIMc0bz4ehfVuMBRmLO%2FZBBQG6FEDfgc0Q7qupNRWiHZPR%2BxESiNJg5Vh1%2Fajn6geMMRJNWv; path=/nextcloud-15; HttpOnly
< Content-Security-Policy: default-src 'none';
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: no-referrer
< Set-Cookie: nc_sameSiteCookielax=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/nextcloud-15; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Set-Cookie: oczswzpgpvdq=pj5kbp17hlf3btfob60d2meh7q; path=/nextcloud-15; HttpOnly
< Set-Cookie: cookie_test=test; expires=Thu, 03-Jan-2019 12:22:50 GMT; Max-Age=3600
< OC-FileId: 00001151oczswzpgpvdq
< Content-Length: 0
< ETag: "7ef58d6f933a4e3387fdf1418f188772"
< OC-ETag: "7ef58d6f933a4e3387fdf1418f188772"
< Content-Type: text/html; charset=UTF-8
< 
* Connection #0 to host 127.0.0.1 left intact

I have been doing some related tests on another webdav service with curl, and for me it started to work ok when using other Content-Type header, like "application/octet-stream" instead of "application/x-www-form-urlencoded".

[rblayzor]

Also running into this issue on Nextcloud 16.04. Nginx 1.16.1

macOS finder DAV client.

With "fastcgi_request_buffering off" uploads in Finder to the Nextcloud server over a DAV mount always end up "0 size".

If we enable "fastcgi_request_buffering on" it's hit or miss. Smaller files generally work, but larger files almost always end up truncated. (0 size).

[Subito]

Is this getting worked on? I would appreciate an update apart from the modified milestones. If I can help with debugging in any way (the first few posts do a good job on explaining how to reproduce the issue, IMHO), please let me know. Its quite an awful feeling that a part of my data gets destroyed once I use something like the iOS-App to Update files and it keeps me away from using anything other than the webinterface.

[skjnldsv]

@Subito sorry, not assigned yet.

[tonihoang]

I cannot push data from my QNAP with Hybrid Backup Sync, due to this, have not found any option to unchunk it yet.

[schmatzler]

This bug is still present with nginx 1.18.0.

With Apache + PHP-FPM, chunked transfers destroy all of my larger files very reliably.
But that seems to be related to a different bug:
https://bz.apache.org/bugzilla/show_bug.cgi?id=57087

So I've put nginx behind an Apache reverse proxy and then set these options in the nginx configuration so it doesn't error out on uploading larger files:

client_max_body_size 50000M;
client_body_buffer_size 400M;
fastcgi_buffers 64 4K;

I've tried uploading 30 files that are under <10k and half of them ended up being 0 bytes.

With fastcgi_request_buffering on only one time a file was truncated to 0 bytes in three subsequent test runs and very large files all transferred fine, too.

So while that option makes it a LOT better it is still not reliable and I cannot trust my data. mod_php being the only option to run Nextcloud reliably makes this very unsuitable for a VPS. It's already a bit painful that I have to run two web servers to make it _almost_ work, but the blame on that can be put on the Apache team too, I guess.

I hope someone will fix this soon.

[dkocher]

We have documented our deployment workarounds in Zero byte file truncate issue with Nextcloud and ownCloud deployed with FastCGI.

[j-dimension]

Adding myself @j-dimension to the list of watchers. Happens to me with Nextcloud 18.06

[stephenitpro]

@dkocher thanks for the tip, I enabled this too on my Nginx Nextcloud config which improved the situation. Originally the documentation for Nextcloud calls for this fastcgi_request_buffering to be turned off. I had to turn it on to improve the situation since I'm waiting for NC19.0.1 to hopefully better address these type of issues. So much tinkering is required to make this setup work.

[stephenitpro]

Nevermind, I spoke too soon, it still is happening. The file uploads properly to the data directory but webdav/webui still report it as 0KB. The only option I have to run a files:scan --all but that takes forever for 50TB of data. :(

[enoch85]

files:scan --all

Does that get you "proper" files back?