Ky: Unhandled Rejection (Error): `input` must not begin with a slash when using `prefixUrl`

[Spomky]

Hi,

I receive this exception when trying to call an API resource at https://example.com/foo/id

My ky instance is created as follow:

import ky from 'ky';

const config = {
    prefixUrl: process.env.API_ENTRYPOINT,
    headers: {
        'accept': 'application/ld+json'
    }
};

export default ky.extend(config);

I read the discussion related to the #11 and especially the one you wrote https://github.com/sindresorhus/ky/pull/11#issuecomment-424838175

Sorry to be rude, but I disagree with the choice you made.
From my point of view, the only acceptable format should be await ky('/unicorn', {prefixUrl: 'https://cats.com'});.

It is not a matter of taste, just a choice guided by the RFC3986:
As per the section 3.3:

If a URI contains an authority component, then the path component
must either be empty or begin with a slash ("/") character.

For the url https://cats.com/unicorn we have the following parts:

• The scheme: https
• The : separator
• The authority //cats.com (no userinfo, no port and host=cats.com)
• The path to the resource: /unicorn

Except if the path is empty, a path must start with a slash.

Can you please reconsider the choice you made in #11 and modify this library according to that RFC?

[szmarczak]

The RFC describes no prefixURL. It is not an authority component (e.g. HTTP2 has) - please do not confuse it.

[Spomky]

You are right, the RFC does not describe any prefix URL, it just describes what parts are available in an URL.
What I try to explain is that the choice made in #11 is a wrong choice.
In an HTTP call

• The host never ends with a slash,
• Unless empty, the path always starts with a slash

Ex.

GET /hello-world.htm HTTP/1.1
Host: www.example.com
Accept-Encoding: gzip, deflate
Connection: Keep-Alive

Here we configure the prefix and input with www.example.com/ and hello-world.htm which is not compliant with the standards we have.

As asked earlier, can you please reconsider your choice?

Ping @sindresorhus

[szmarczak]

I understand you, but prefixURL is not host nor authority. Don't confuse it.

which is not compliant with the standards we have

Funny. Firstly, you say it isn't, then you confirm there's no such thing in RFC and again you say it isn't compatible with RFC. Let me clarify something:

The URL is calculated by merging prefixURL with the input. So, if you provide prefixURL https://somesite.com/api/v1 and input hello the result is https://somesite.com/api/v1/hello. It's the same if you pass https://somesite.com/api/v1/hello directly.

THEN, the output is translated by JavaScript into these parts and a request is made:

• protocol: https:
• host: somesite.com
• path: /api/v1/hello

[Spomky]

Funny. Firstly, you say it isn't, then you confirm there's no such thing in RFC and again you say it isn't compatible with RFC.

😒
I don’t see any contradiction in the comments I wrote. I have just confirmed there is no concept of prefixURL in the RFC, but still maintain there is something wrong with the choice made in this library.
prefixURL is not host nor authority, it generally contains the scheme, the authority (wich is composed with a host and the optional userinfo and port) and may contain part of the path (/api/v1) in your example.

if you provide prefixURL https://somesite.com/api/v1 and input hello the result is https://somesite.com/api/v1/hello

I would like to provide https://somesite.com/api/v1 and /hello.
The problem is that you decided to append a slash at the end of the prefix and forbid the one at the beginning of the path.
The path defined in the RFC mentioned above and every http calls (including the one from your example) always starts with with a slash.

https://somesite.com/api/v1/ + hello has the same result as https://somesite.com/api/v1 + /hello but the first one is counterintuitive and causes troubles when dealing with relative links in jsonld responses: it is then mandatory to rewrite every links and remove the slash.

Even if I understand you won't change your mind, you may also have a look at https://restfulapi.net/resource-naming/

• Use forward slash (/) to indicate a hierarchical relationships
• Do not use trailing forward slash (/) in URIs

[szmarczak]

I would like to provide https://somesite.com/api/v1 and /hello.

Okay, but the problem is we don't know if by / you're referring to the prefixURL or the main root. It's ambiguous. But if the prefixURL ends with / and the input doesn't begin with / then it's clear what the result is :)

[Spomky]

Okay, but the problem is we don't know if by / you're referring to the prefixURL or the main root.

Does the library have to manage that?
The library should only concatenate the prefix with the input.

When I work on my books, the prefix could be https://api.example.com/books.
With an empty input, I should be able to send GET requests (all books) or POST requests (create a new one).
By using /{id} as input, a GET, DELETE or PUT request should reach https://api.example.com/books/{id}.

At the moment https://api.example.com/books cannot be reached because a slash is added (side effect).
To reach my resources I have to use https://api.example.com/withbooksorbooks/{id}instead of just https://api.example.com/books and /{id}.

[sholladay]

The simple answer is that if you want to reach https://api.example.com/books while using a prefix, then your prefix should be https://api.example.com and your input should be books.

await ky('books', { prefixUrl : 'https://api.example.com' });

If you want to reach https://api.example.com/books/1234 while using a prefix, then your prefix can be either https://api.example.com or https://api.example.com/books.

await ky('books/1234', { prefixUrl : 'https://api.example.com' });
await ky('1234', { prefixUrl : 'https://api.example.com/books' });

As an aside, your prefixUrl can optionally end with a slash, Ky will behave the same regardless.

Right now, to reach https://api.example.com/books/ (note the trailing slash), you could even do this:

await ky('', { prefixUrl : 'https://api.example.com/books' });

We could probably change that to not append the trailing slash if input is empty. But I'm not sure it's worth it. Some people would probably prefer the old behavior. I think it would be better for users to instead use a prefix with one less path component and then use a more explicit input that includes or excludes the trailing slash, if it matters to the server.

[Spomky]

Hi @sholladay and thank you for this detailed answer.
I understand how this library works. The purpose of this issue and the discussion is not about to use it but the way it works.

On one hand, by adding a slash at the end of the base URI, it creates a side effect.
On the other hand, I have to rewrite any single resource path and remove the forward slash (resource objects I receive are JSON-LD objects and the @id parameter is often a relative one).

That is why I see the following possibilities to solve that:

1. Concatenate the complete path to the resource on my side (the prefixUrl becomes completely useless)
2. Propose a PR to remove this limitation and the side effect (done in #71 but refused)
3. Find another library/another way to communicate with the API (and loose the benefits of the library)

At the moment, I've directly integrated a custom version of this library where these lines have been removed.

To be honest and following the posts hereabove, my conclusion is that the library should just concatenate the prefix with the input and make no modification or assumption on the final result.
But it appears to me that there is no way to change these lines here so the third solution looks to be the way to go for me.

[sholladay]

I understand where you're coming from. FWIW, as you probably saw in #11, when I wrote the prefixUrl code, I initially wanted it to be baseUrl and for it to behave exactly like the <base> tag. That would have made cases like this a bit easier to deal with since the URL resolution would work exactly the same as all of the browser APIs you are used to. But as Sindre mentioned, it has drawbacks too.

That being said, I don't fully understand why you even need a prefix if you are fetching resources at many different path levels. What is the prefix you actually want to use that is common to all of your requests? If you fetch resources at the root level, it doesn't seem to me that a prefix is needed unless the requests are always going to a different origin.

To put that another way in case it isn't clear, these things are equivalent:

await ky('/some-root-level-resource')
await ky('some-root-level-resource', { prefixUrl : location.origin });

If there is any way you can make your paths be origin-relative (as in, relative to the root, by starting with a /), then they should "just work" as input without a prefix (again, unless cross-origin). If, on the other hand, you are trying to "follow" a URL from one resource to another using a path that is relative to the first resource (seems like this may be the case?), then yeah that's not generally going to work unless the input is specially crafted - the prefixUrl option is not designed for this.

I think there's a reasonable chance we can fix this for you somehow. The first thought that comes to mind is that maybe Sindre would reconsider having a baseUrl option if the documentation were very high quality explaining the difference between prefixUrl and baseUrl. I'll think about some other ways to solve it too.

In the meantime, how about a beforeRequest hook? I think this should work. It's a bit of a hack, but IMO worth it to keep using Ky.

const myClient = ky.extend({
    prefixUrl : 'https://api.example.com/books',
    hooks : {
        beforeRequest : [
            (option) => {
                if (option.prefixUrl.endsWith('/')) {
                    option.prefixUrl = option.prefixUrl.slice(0, -1);
                }
            }
        ]
    }
});

[Spomky]

I understand where you're coming from. FWIW, as you probably saw in #11, when I wrote the prefixUrl code, I initially wanted it to be baseUrl and for it to behave exactly like the tag. That would have made cases like this a bit easier to deal with since the URL resolution would work exactly the same as all of the browser APIs you are used to. But as Sindre mentioned, it has drawbacks too.

Yes I read #11. I mentioned it in my initial post. I am not sre to clearly see what the drawbacks are.
I will take time to read all posts carefully.

That being said, I don't fully understand why you even need a prefix if you are fetching resources at many different path levels. What is the prefix you actually want to use that is common to all of your requests? If you fetch resources at the root level, it doesn't seem to me that a prefix is needed unless the requests are always going to a different origin.

That is exactly my problem. The requests are sent to an entrypoint on another domain.
The behaviour is not the same when the prefix is set: await ky('/some-root-level-resource') works only if no prefix is set.

If there is any way you can make your paths be origin-relative (as in, relative to the root, by starting with a /), then they should "just work" as input without a prefix (again, unless cross-origin). If, on the other hand, you are trying to "follow" a URL from one resource to another using a path that is relative to the first resource (seems like this may be the case?), then yeah that's not generally going to work unless the input is specially crafted - the prefixUrl option is not designed for this.

Resource paths are all relative to the main entrypoint (e.g. entrypoint is https://api.example.com/v1 and resource path is /books/{id}
I thought the prefixUrl was designed to ease the concatenation of the entrypoint and the path and avoid calls like await ky(API_ENTRYPOINT + '/books/{id}') by having an instance of ky with a predefined base using extend.

I think there's a reasonable chance we can fix this for you somehow. The first thought that comes to mind is that maybe Sindre would reconsider having a baseUrl option if the documentation were very high quality explaining the difference between prefixUrl and baseUrl. I'll think about some other ways to solve it too.

I am confused, it looks like there was a misunderstanding between prefixUrl and baseUrl.
Whatever the way this is implemented, you should keep it simple. From my POV baseUrl + prefixUrl + input should be "https://api.example.com" + "/v1" + "/books/{id}".

In the meantime, how about a beforeRequest hook? I think this should work. It's a bit of a hack, but IMO worth it to keep using Ky.

I am pretty sure it will work. That’s just quite strange to me to create a hook to remove the added trailing slash.
Even if it is not a good way to deal with that issue, I prefer to keep my custom implementation at the moment.

[sholladay]

Resource paths are all relative to the main entrypoint (e.g. entrypoint is https://api.example.com/v1 and resource path is /books/{id}

I think this is the problem. You are specifying the resource path as /books/{id}, which doesn't really make sense. URLs that start with a single/ are origin relative, not resource relative. Yet you are treating them as resource relative for some reason.

If ky supported a baseUrl option that followed the standard rules for <base> tags and you did this ...

await ky('/books/{id}', { baseUrl : 'https://api.example.com/v1' });

... you would end up fetching https://api.example.com/books/{id} instead of what you _really_ wanted which is https://api.example.com/v1/books/{id}. This wouldn't be ky's fault - that's just how URL resolution works. URLs can only be considered resource relative if they do not begin with a / and do not begin with a scheme.

For this reason, ky disallows input to begin with a / when using prefixUrl, precisely because we want to avoid being misleading. Ky doesn't perform URL resolution against prefixUrl, so all input values are inherently resource relative, not origin relative. It's best to make that clear by ensuring that your input URLs are correctly formed as resource relative, rather than origin relative.

I do see how it's inconvenient that we don't support origin relative URLs when using an extended ky instance with a prefix. That feels like a genuine problem to me. However, I'm personally against treating origin relative URLs like /books/{id} as if they are resource relative. The user of your instance should be using books/{id} instead (no leading slash), not because ky dictates that, but because that's how URLs work - /books/{id} indicates they want to break out of /v1. This is also how filesystem paths behave.

[Spomky]

Obviously this is an endless debate. From my POV, a third party library should not be opinionated. It is up to the user of the library to prepare its requests depending on the API.
I prefer to deal with my own implementation.

[sholladay]

If ky were unopinionated, it would implement browser URL resolution, and your requests would still go to an incorrect URL, as I described above, because you are treating origin relative URLs as though they are resource relative, which is going to break on basically any sane library.

I am hopeful that we can take care of this part of my last comment though:

I do see how it's inconvenient that we don't support origin relative URLs when using an extended ky instance with a prefix. That feels like a genuine problem to me.

Anyway, I'm glad you've got something that works for you. I hope this was still useful in some way. :)

[aintHuman]

This error is totally nuts and completely insane, forward slash at the start of the resource path is 100% and COMPLETELY valid.

Lack of a forward slash can introduce problems when it comes to relative URL's.

Using a forward slash works fine when urlprefix is not defined, and that is indeed what I want in production, but in development I cannot proxy requests to the express server, and I therefore I need to use the prefixurl which in turn would require the production-ready resource-paths to be broken by insisting on them not beginning with a forward slash.

So what, am I supposed to keep two sets of api endpoints? One for development, and one for production, only differing by a forward-slash?, all because of your narcissistic desire to control something so trivial?

Please fix this.

[sholladay]

Lack of a forward slash can introduce problems when it comes to relative URL's.

When prefixUrl is being used, then input is _not_ a relative URL, but rather a suffix. That is an important and meaningful distinction and it's one of the main reasons we don't allow the forward slash, to avoid further ambiguity. If it had the semantics of a relative URL, we would allow the forward slash, I promise.

So what, am I supposed to keep two sets of api endpoints?

No, instead the recommendation is to avoid the forward slash in both production and development. If you _need_ a forward slash in production because input must be a root-relative URL (as opposed to page-relative), then that is understandable. However, the problem is Ky's prefixUrl option is merely a prefix, so relative URLs don't get resolved against it, and therefore they probably won't work as expected. We chose to throw an error to avoid the implication that relative URLs are supported.

What I think many users want, and which I am personally in favor of implementing, is a baseUrl option that _does_ do URL resolution and _does_ support relative URLs. It would behave much like the HTML <base> tag. However, there was some minor disagreement about it, mostly related to it being confusing to have both options. I suspect we could work through those issues, though

[aintHuman]

So are you saying use prefix for both production and development:

lets say previously, my path was /api/xyz

now, the setup goes to:

1. in production the prefix might be /,
2. development prefix becomes http://localhost:3000/ (ie to proxy the request)

and then the path changed from /api/xyz to api/xyz, and be used for both?

[Spomky]

@aintHuman,

Don’t loose your time here, this is an endless debate.
You actually don’t need any library like this one.
The “simplicity” hides lots of troubles with side effects.
Better go with bare fetch function.