Couchdb: Couchdb sends more than requested range in attachments

[sploders101]

Description

When performing a request for a partial attachment, Couchdb starts at the correct file location, but sends more data than the Content-Length header suggests, causing an error in some http clients.

Steps to Reproduce

Upload a large text file as an attachment to a couchdb instance
Run:

nc 127.0.0.1 5984

and send this data, substituting your information.

GET /db/doc/attachment.ext HTTP/1.1
Range: bytes=1-4
Host: 127.0.0.1:5984
Connection: close

Expected Behaviour

CouchDB should send only the requested bytes.

Your Environment

Fresh install

• CouchDB Version used: 2.3.1
• Browser name and version: N/A
• Operating System and version: Linux (docker image couchdb:latest)

Additional context

The specific type of attachment I'm using is an asar archive. I am attempting to read files from it without downloading the entire thing, and storing it to the fs/ram. It has a Content-Type of application/octet-stream in the database.

I am using node.js http.request function as my http client, but this is reproducible with any client that doesn't close the connection after Content-Length bytes.

[janl]

it doesn’t look like I can reproduce this:

> curl -vX PUT http://a:[email protected]:15984/a
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 15984 (#0)
* Server auth using Basic with user 'a'
> PUT /a HTTP/1.1
> Host: 127.0.0.1:15984
> Authorization: Basic YTph
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 201 Created
< Cache-Control: must-revalidate
< Content-Length: 12
< Content-Type: application/json
< Date: Fri, 25 Oct 2019 09:05:44 GMT
< Location: http://127.0.0.1:15984/a
< Server: CouchDB/3.0.0-be2364d9c (Erlang OTP/21)
< X-Couch-Request-ID: de4cfb3575
< X-CouchDB-Body-Time: 0
< 
{"ok":true}
* Connection #0 to host 127.0.0.1 left intact
jan@rose (11:05:44) ~ 
> curl -vX PUT http://127.0.0.1:15984/a/a/a -Hcontent-type:application/octet-stream -d 'abcde'
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 15984 (#0)
> PUT /a/a/a HTTP/1.1
> Host: 127.0.0.1:15984
> User-Agent: curl/7.54.0
> Accept: */*
> content-type:application/octet-stream
> Content-Length: 5
> 
* upload completely sent off: 5 out of 5 bytes
< HTTP/1.1 201 Created
< Cache-Control: must-revalidate
< Content-Length: 64
< Content-Type: application/json
< Date: Fri, 25 Oct 2019 09:05:51 GMT
< Location: http://127.0.0.1:15984/a/a/a
< Server: CouchDB/3.0.0-be2364d9c (Erlang OTP/21)
< X-Couch-Request-ID: ea824e1e65
< X-CouchDB-Body-Time: 0
< 
{"ok":true,"id":"a","rev":"1-3f5a3d76ee90dc21a1bc013c8010446c"}
* Connection #0 to host 127.0.0.1 left intact
jan@rose (11:05:52) ~ 
> curl -v http://127.0.0.1:15984/a/a/a -HRange:bytes=1-2
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 15984 (#0)
> GET /a/a/a HTTP/1.1
> Host: 127.0.0.1:15984
> User-Agent: curl/7.54.0
> Accept: */*
> Range:bytes=1-2
> 
< HTTP/1.1 206 Partial Content
< Accept-Ranges: bytes
< Cache-Control: must-revalidate
< Content-Length: 2
< Content-Range: bytes 1-2/5
< Content-Type: application/octet-stream
< Date: Fri, 25 Oct 2019 09:06:00 GMT
< ETag: "q1a02StAcTrMWviZhdS3hg=="
< Server: CouchDB/3.0.0-be2364d9c (Erlang OTP/21)
< 
* Connection #0 to host 127.0.0.1 left intact
bc

[kocolosk]

Curl is too smart for its own good sometimes -- it's ignoring the bytes on the wire that are past the Content-Length.

Turns out this is the cause of the recent FreeBSD build failures. Newer versions of libcurl that reject HTTP/0.9 content are seeing the remaining bytes on the wire and bailing out.

Here's some tcpdump output using data generated by the attachment_ranges test suite. Note the 29 byte response packet with the full "This is a base64 encoded text" response:

10:33:01.411874 IP 127.0.0.1.57526 > 127.0.0.1.15984: tcp 152
E.....@.@.............>p
....z.............
O...O...GET /test_suite_db_yqfppndr/bin_doc/foo.txt?anti-cache=70594 HTTP/1.1
Host: localhost:15984
User-Agent: curl/7.64.1
Accept: */*
Range: bytes=0-3


10:33:01.411895 IP 127.0.0.1.15984 > 127.0.0.1.57526: tcp 0
E..4..@.@...........>p...z..
..^.....(.....
O...O...
10:33:01.434856 IP 127.0.0.1.15984 > 127.0.0.1.57526: tcp 294
E..Z..@.@...........>p...z..
..^.....N.....
O...O...HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Cache-Control: must-revalidate
Content-Length: 4
Content-Range: bytes 0-3/29
Content-Type: application/octet-stream
Date: Wed, 13 Nov 2019 15:33:01 GMT
ETag: "aEI7pOYCRBLTRQvvqYrrJQ=="
Server: CouchDB/3.0.0-18db801b7 (Erlang OTP/22)


10:33:01.434891 IP 127.0.0.1.57526 > 127.0.0.1.15984: tcp 0
E..4..@.@.............>p
..^.z.......(.....
O...O...
10:33:01.435501 IP 127.0.0.1.15984 > 127.0.0.1.57526: tcp 29
E..Q..@.@...........>p...z..
..^.....E.....
O...O...This is a base64 encoded text
10:33:01.435533 IP 127.0.0.1.57526 > 127.0.0.1.15984: tcp 0
E..4..@.@.............>p
..^.z.......(.....
O...O...
10:33:01.435714 IP 127.0.0.1.57526 > 127.0.0.1.15984: tcp 0
E..(..@.@.............>p
..^.z..P.......

[kocolosk]

Also, great find @sploders101