Racket: string->url doesn't handle whitespace at the ends of a URL string

[nikofeyn]

The string->url procedure inside collects/net/url-string.rkt does not properly handle input strings with whitespace at the ends of the string. So it reports an invalid URL string error "host provided with non-absolute path (i.e., missing a slash)" for strings like "http://google.com ". Real-world error:

This should be a simple fix to process the input string "str" to trim any whitespace from the end, but the string->url function is fairly complex with multiple regular expression matches and str sprinkled throughout it. The easiest fix is to wrap the entire body with a let expression replacing str with a trimmed string.

It is probably the case that url-regexp and any related regular expressions should be fixed as well to still match strings with whitespace on the ends. If there are any tests for this library, URL strings with whitespace on the ends should be added.

I encountered this with an http_proxy setting. Since the setting's value was copied from another document, a trailing whitespace was entered when pasted into the environment variable dialog. In this case, Dr. Racket showed the error command window upon starting up, and then failed to start (the startup window exits and closes once the command window is closed). In this case, I would rank this fairly high in terms of priority since it prevents Dr. Racket from starting up and should be relatively easy to fix.

[mflatt]

I can see how stripping whitespace from a dialog input field can be a good idea, but I'm skeptical that string->url should have the job of stripping whitespace.

Attempts to build "do what I mean" behavior into an API usually does not go well in my experience. For example, Windows strips whitespace from paths at a fairly deep level, and it causes all sorts of trouble with reliable path parsing (e.g., "a /b" is "b" with a subdirectory whose name ends with a space, but "a " by itself won't access that subdirectory).

Even though a space is not supposed to be in a URL, given that we haven't ruled it out in string->url where other non-special characters are allowed, it should stay allowed (which I think is a typical choice). And then it should be treated consistently—even at the end of the string.

[nikofeyn]

1. Is there ever a use case where a string with whitespace at either or both ends that represents a valid or invalid URL is changed to an invalid or valid URL, respectively, when the whitespace is removed?
2. Is there a use case where a string with whitespace at either or both ends representing a URL is changed to a _different_ URL when the whitespace is removed?

If the answer is no to both, then I think the case is strong that string->url should handle whitespace at the ends. That is a fairly weak application of do what I mean if a question about the input arguments can be explicitly answered. If the answer is yes to either, I guess it would be good to know what those cases are, as it seems possible that string->url is handling a lot of different types of URLs and not just web URLs because to my knowledge web URLs can't have non-encoded spaces in them (and we're just talking about at the ends here). If string->url is handling a lot of different URL types, it isn't clear to me how to handle it. Is it possible to update string->url to handle whitespace correctly for web URLs? I would guess for such URLs, the answer to both questions above is no.

Although these are applications and not APIs, all of Chrome, Firefox, and Windows explorer seem to correctly handle whitespace at the ends of URLs. Chrome doesn't crash if I type " google.com ". (This is more addressing the overall behavior rather than an API decision I suppose.) In my personal experience, removing whitespace at the ends of an input string is actually the typical decision at the level of APIs, especially when reading in values and properties from INIs, environment variables, etc. in situations that the above questions have explicit answers. If they don't, then the padded whitespace is left alone. This is done for robustness reasons because it is typical in these situations that inadvertent whitespace characters get entered in.

If you still think string->url shouldn't handle it, there is an issue elsewhere then that a whitespace at the end of an environment variable containing a web URL prevents Dr. Racket from starting up. This is complicated by my example because http_proxy was the URL with a whitespace at the end, so it prevented Dr. Racket from starting up. But Racket (specifically the package manager / raco) doesn't work behind proxies, so something part of a feature that doesn't really work prevented Dr. Racket from starting up.

[lexi-lambda]

It is true that URLs do not allow literal whitespace characters, according to RFC 3986. In that sense, it’s true that Racket’s string->url function is “too permissive,” since it accepts URLs with any characters at all. However, this is both a very conservative extension to the specified syntax of URLs and an implication of the regular expression given by the RFC itself in Appendix B.

This modification generalizes the unreserved character set from [-a-zA-Z0-9._~] to “any Unicode character than isn’t reserved by the RFC for some other purpose.” This generalization is reasonable, and I’d bet that a lot of Racket programmers (implicitly or not) rely upon it, but it’s incompatible with your proposal of stripping whitespace: it makes whitespace a perfectly legal character in URLs, so it would mean dropping a valid character in the input string.

I agree with Matthew’s sentiment: I think string->url is supposed to be a pretty low-level, non-DWIM operation that just means “please interpret this set of characters as a URL.” The characters are essentially assumed to be the proper format, and the string itself might even be machine-generated.

What you are asking for, though, seems to be a different function that converts a string typed by a human into a URL, which could perform the kinds of normalizations that a web browser’s address bar performs. Appendix C even recommends that kind of normalization:

For robustness, software that accepts user-typed URI should attempt to recognize and strip both delimiters and embedded whitespace.

I think a separate human-string->url function or a #:normalize? option to string->url would be a reasonable way to implement this in one place, which could do the trimming you suggest, and it would make sense for DrRacket to use that function instead of string->url (which is really for machines, not people).

[samth]

Better than adding a new function to net/url is treating the whole thing as a low-level implementation detail and writing a new library for making http requests a la Python's requests library or Rust's reqwest.

[mflatt]

I've at least adjusted net/url to at least not fail on startup. If the http_proxy environment variable is allowed by other applications to have trailing whitespace, then I think it makes sense to add string trimming when the environment variable is looked up (but I didn't make that change, yet).

[nikofeyn]

I think all this makes sense. Thank you for the change @mflatt!