Spark: Spark incorrectly decodes params from the url path

[dmacvicar]

We have a controller that serves package files. Some rpms use + signs in their versions, e.g.: libgomp1-5.2.1+r226025-4.1.ppc64le.rpm.

Our router looks like this:

// Download endpoint
get("/manager/download/:channel/getPackage/:file",
    DownloadController::downloadPackage);
get("/manager/download/:channel/repodata/:file",
    DownloadController::downloadMetadata);

When doing a request like:

GET /rhn/manager/download/sles12-sp1-pool-ppc64le/getPackage/libgomp1-5.2.1+r226025-4.1.ppc64le.rpm?eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjE0OTAxODA5NTgsImlhdCI6MTQ1ODY1Mzk1OCwibmJmIjoxNDU4NjUzODM4LCJvcmciOjExImp0aSI6InpqODd1RE1UTm5OUHNQUm5BQW5HTGcifQ.u42yvUR5r2tx4Pxy2EQpyzc5HX4sw78is1-H7H3e7xM

We realized Spark decodes the path, and they arrive to the parameter map incorrectly, with + replaced as (spaces). so that:

String channel = request.params(":channel");
String filename = request.params(":file");

request.params(":file") has getPackage/libgomp1-5.2.1 r226025-4.1.ppc64le.rpm

The culprit line is:

https://github.com/perwendel/spark/blob/master/src/main/java/spark/servlet/FilterTools.java#L55

We can workaround it doing the parsing of params again ourselves in the controller, luckily, request.url() is fine.

[dmacvicar]

I am saying that Spark is using it wrong. + in the path should be treated literally. + is a special character in the query component.

The program doing the download is not encoding it. Nor does curl.

[jwgmeligmeyling]

This is not a bug in Spark. Nor it is a problem with java.net.URLDecoder. According to RFC-3986 + is a sub delimiter and thus a reserved character. The character should be escaped in URL's. This is therefore a client side problem, and not related to Spark in any way.

[lucidd]

@JWGmeligMeyling it's not that simple.

First some observations java.net.URLDecoder decodes + into space.
This is the + encoding defined for application/x-www-form-urlencoded source. It's based on the URI percent encoding but adds modifications like the + to space one. Therefore its not meant for decoding a URI. The Java Doc of URLDecoder even says that its for HTML Form / application/x-www-form-urlencoded decoding.

Secondly URLDecoder seems to be inconsistently used anyway since Request#uri() gives the decoded result while Request#url() still has the undecoded result.

As for encoding + in a URL: (From linked RFC)

sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="

source

path          = path-abempty    ; begins with "/" or is empty
                    / path-absolute   ; begins with "/" but not "//"
                    / path-noscheme   ; begins with a non-colon segment
                    / path-rootless   ; begins with a segment
                    / path-empty      ; zero characters

path-abempty  = *( "/" segment )
path-absolute = "/" [ segment-nz *( "/" segment ) ]
path-noscheme = segment-nz-nc *( "/" segment )
path-rootless = segment-nz *( "/" segment )
path-empty    = 0<pchar>
segment       = *pchar
segment-nz    = 1*pchar
segment-nz-nc = 1*( unreserved / pct-encoded / sub-delims / "@" )
                    ; non-zero-length segment without any colon ":"

pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"

source

This shows that sub-delims (and therefore +) can be used unencoded inside the path component of the URI by using it with its special meaning as delimiter. Since URLDecoder changes all instances of + special meaning or not, this should already be enough to prove that URLDecoder does not do the right thing here.

Now the part that makes me believe that + can also be used unencoded as data in the path under certain conditions.

URIs include components and subcomponents that are delimited by
characters in the "reserved" set. These characters are called
"reserved" because they may (or may not) be defined as delimiters by
the generic syntax, by each scheme-specific syntax, or by the
implementation-specific syntax of a URI's dereferencing algorithm.
source

So sub-delims do not have any special meaning unless specified by one of:

• generic syntax
• scheme-specific syntax
• implementation-specific syntax of a URI's dereferencing algorithm (to be honest i don't understand this part so i will overlook this for now)

The generic syntax being the one defined in this RFC where we already saw that the path component does not specify special meaning to + but simply includes it as one of the allowed segment characters.

The scheme specific syntax in this case would be for http[s].

The definitions of "URI-reference", "absolute-URI", "relative-part",
"scheme", "authority", "port", "host", "path-abempty", "segment",
"query", and "fragment" are adopted from the URI generic syntax. An
"absolute-path" rule is defined for protocol elements that can
contain a non-empty path component. (This rule differs slightly from
the path-abempty rule of RFC 3986, which allows for an empty path to
be used in references, and path-absolute rule, which does not allow
paths that begin with "//".) A "partial-URI" rule is defined for
protocol elements that can contain a relative URI but not a fragment
component.
source

So the http[s] scheme adopts largely the URI generic syntax and does not seem to assign + and special meaning in the context of the path component or as delimiter of components.

Lastly following three parts of the RFC suggest that only if a character is outside the allowed charset (which + is not see path grammar above) or if it has a reserved purpose as delimiter (which + is neither defined as component delimiter nor as subcomponent delimiter in the context of the path component). So since + does not have any special purpose in the path context is can safely be used as data without percent encoding.

If data for a URI component would conflict with a reserved
character's purpose as a delimiter, then the conflicting data must be
percent-encoded before the URI is formed.
source

A percent-encoding mechanism is used to represent a data octet in a
component when that octet's corresponding character is outside the
allowed set or is being used as a delimiter of, or within, the
component.
source

If a reserved character is found in a URI component and
no delimiting role is known for that character, then it must be
interpreted as representing the data octet corresponding to that
character's encoding in US-ASCII.
source

Additional resources supporting this observation.

https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters
http://stackoverflow.com/a/2678602/166041
http://stackoverflow.com/a/1006074/166041

[tipsy]

@dmacvicar Would you like to create a PR for this?

[Yqwed]

@tipsy https://github.com/perwendel/spark/pull/872