Kong: allow for parameter mapping in path based routing

[ahmadnassri]

e.g. allow request_path to contain values such as /api/{foo}/bar which map to the upstream_url /upstream/my/api/bar/{foo}

this could also be more globally addressed with regex matching groups e.g.

request_url = '/api/(.+)/bar`
upstream_url = '/upstream/my/api/bar/$1'

[ayush]

Native support for path parameters in Kong seems like a pretty significant missing piece.

Right off the bat, I want to say that it is clear to me that Kong can proxy to path parameter based APIs. Let us say I have two backend GET APIs /pet/{petId}/get-image and /pet/{petId} If I want Kong to be able to proxy to both, I can simply create an API with request_path as /pet. This will make sure that http://kongserver/pet/123 will be forwarded to http://upstreamserver/pet/123 and http://kongserver/pet/123/get-image will be forwarded to http://upstreamserver/pet/123/get-image (thanks to @travisyates for clarifying this)

However, not having native support for path parameters is problematic. There is probably more to it but I see two resulting side effects of Kong not supporting path parameters:

Kong Blindness

In the above example, Kong is blind to the difference between these two APIs. Even though these are two very different APIS, Kong cannot tell them apart. This leads to problems with using plugins and other features of Kong effectively.

Let us say that Kong did support path parameters and let me define these two APIs separately:

• Get Image of any pet: /pet/{petId}/get-image
• Get all details of a /pet/pet/{petId}

In my use case, I want to allow anonymous access to /pet/{petId}/get-image while securing access to /pet/{petId}. Using ACL plugin I would be able to do it but only if Kong allowed me to add these two as different APIs. Right now, Kong cannot tell the difference between these two. And thus I cannot secure these two separately.

Additional complexity of collapsing multiple APIs

As a developer using Kong, I now have to parse all my API definitions and if there are situations like the above /pet/{petId}/get-image and /pet/{petId}, I'll need to deal with the complexity of collapsing them into one API. This kinda sucks. I was working on a project which takes swagger api definitions and puts them in Kong with the right security. This limitation of Kong introduces an impedance mismatch which does not have any ideal solution other than Kong natively supporting path parameters.

[ahmadnassri]

while this is a limitation of the functionality, i wouldn't say its a "significant missing piece"

Kong aspires to be a solution for ALL API providers, and the described is just one of many abstractions and methods used. we'll have to balance that equally against other feature requests and with the community's feedback and support we can prioritize appropriately :)

[ayush]

@ahmadnassri Path parameters are widely used and are even part of open standards like swagger. Completely ignoring them in an API middleware does not seem right. Even Amazon API Gateway has some support for them.

I don't wish to be incessant about how we classify this :) but in the interest of _community's feedback and support_ to help Kong developers prioritize, I'd certainly qualify it as a "significant missing piece".

[ahmadnassri]

nobody said anything about "Completely ignoring" them :) just talking about prioritizing features.

there. are. many. feature. requests so we're looking for community feedback for help in prioritization!

[ahmadnassri]

related: the official spec: RFC 6570 describing proper URL Templating usage.

[sidharthachatterjee]

@ahmadnassri Any update on when we can expect this?

[BrianHutchison]

+1

This would meet a major use-case/need of ours.

[3dbrows]

+1

[eitanpo]

+1

[RossJayJones]

+1

[haikuo81]

+1

[andyfleming]

Wildcard/parameter mapping would be helpful. In my specific use case I'm looking to do something like:

/something/{alphanumeric}/whatever to be routed to /whatever and I want to add the header x-id: {alphanumeric}

[bradwood]

+1

[matangiz]

+1

[lucas-cegatti]

+1

[vitorsdcs]

+1

[merlindorin]

+1, really need it, we are stucked now with this missing feature and we have review our mircoservice architecture :'(

The main problem is related to API depth in our services. For example, I can't have different microservice for an endpoint like /users/john/organizations and /users/john/projects. The unique base path that we can handle with kong is /users so... we can have just one service behind (and not one for /users/:username/organizations and /users/:username/projects).

[merlindorin]

+1 for RFC6570

[kintetsu]

+1

[dankennedy]

+1

[goraniliev]

+1

[erutherford]

@ahmadnassri is there any movement on this? We're currently implementing a work around with a nginx proxy in between kong and our services, but this is a less than ideal setup. We'd definitely prefer to have kong handle this path matching on it's own. If you could provide the likelihood or progress on this issue it'd aid us greatly in planning out our future direction.

[thibaultcha]

This is still on our roadmap but no work has been done as of today yet. Expect such a feature for 0.11 or 0.12, eventually.

[Roam-Cooper]

Any word on this? Kong is unusable for modern apis without this feature. It's 2016; any petty arguments about this are null and void, in my opinion. Almost every major api in the world uses url parameters.

[drabiter]

+1

in other case you may want to expand metrics from statsd per uri, so this could help that also.

[kvaillant]

+1

[tookko]

+1

[thedodd]

+1

[rudijs]

+1

[nemccarthy]

+1

[Krieke]

+1

[SriramKeerthi]

Any updates on timelines for this issue? We're about to invest a lot of time into Kong in our stack and this is a deal-breaking issue for us.

[Krieke]

@SriramKeerthi +1 We are also investigating different API gateway sollutions: Kong vs Tyk vs ...
I has come to my attention that some pull request remains open for several months in despite.

[thibaultcha]

Hey all, just a quick update on this as we realize it's a pretty big deal for a lot of people:

We're slowly making our way through our list of priorities, but now that 0.10 is out and contains the new router, we're planning to build this by Q2 this year. The new router is built with extensibility in mind, especially for regex and parameters mapping, so we're now on track for this!

Cheers

[alfredogordian]

Nice, I really looking forward for Q2 :) , In the meantime, does anybody has a partial solution to this issue? I would like to read some ideas

[ikouchiha47]

@driagongit one solution is to have a router before the kong clusters, which supports such url redirection. And then redirect url to kong ..

[merlindorin]

clearly :/ ... In HA architecture, this need a set of extra services... more services, more complexity, more complexity, more 💸

(#useless answer, just 😢 )

[hizorro88]

+1

[nizq]

+1

[mrproper]

I ended up finding this after running into the same issue....to be honest i was a little surprised that path parameters dont exist.
for example
api.tld/foo/$id/bar I want restricted to GET
api.tld/foo/$id/betsy I want restricted to POST

in current kong i basically have to allow:
api.tld/foo GET,POST

Seems like a bit of an oversight or am i just missing something completely?

[HenryChelsea]

+1. Modern apis need this. (RFC6570)

[remihillairet]

+1

[serghei-ilin]

+1

[Jimflip]

"we're planning to build this by Q2 this year. " <---- so what's the new estimate?

[nehasupnfan]

Any movement on this??

[thibaultcha]

Hi,

Yes, we are still planning on building this soon. This estimate is still somewhere around this month or the next one.

After updating the router to support much better use cases already, our focus has been more on things like removing the Serf dependency, improving our new DNS resolver, improving our configuration options for better usability (considering the current use cases), and fixing a good amount of bugs. We've also invested a good amount of time on-boarding new developers on the team (hi @p0pr0ck5 @bungle :wink:), and researching/designing our next features and refactors. On top of that, as a company Mashape also has its own business requirements and priorities on which focus was also given on.

That said, our Q2, 0.11/0.12 estimation is still standing, but I hope that this highlights not the fact that we are disinterested in this feature, but rather that we are/were taking care of higher priorities at this point :)

[ghost]

+1

[ramo-karahasan-riechardt]

+1

[lodow]

Please do not comment with "+1".
Use thumbs emoji instead so we can keep important information visible.
Thank you 😃

[williamhorstkamp]

This would be exceptionally useful. Looking forward to this release!

[thibaultcha]

Hi everyone! I just submitted #2681 which implements this :)

It contains support for regex URIs in the router, and provides PCRE groups captures as well. Feedback, testing and reviews are much welcome!

Edit: we will probably include this feature in our 0.11 release family (upcoming rc2, our subsequent 0.11.1 release).

[rdzak]

Hi @thibaultcha!

First of all - thank so much for working on it. Do I understand correctly that this feature allows dynamically building upstream URLs from available data such as consumer_ids? For instance, when using auth plugin, consumer_id is sent currently in the header to upstream API. Will this allow me to dynamically build upstream routes using this variable like that: /customers/{consumer_id}/something?
Also I'm super curious when is the next release planned :)

Kind regards.

[thibaultcha]

@rdzak Hi,

Do I understand correctly that this feature allows dynamically building upstream URLs from available data such as consumer_ids?

It does not. This work enabled URIs regexp matching and capturing groups extraction. It then exposes the captured groups to the plugins for the lifecycle of the request. It is now up to the plugin to be updated and to use those capturing groups to do something. That is beyond the scope of the proposed PR, and still in the scope of this issue. We are working on it.

Additionally, if the value you want to use comes from the request headers, then it is not dependent on the regexp PR in any way, but entirely on the request-transformer plugin.

Also I'm super curious when is the next release planned :)

We have an RC 2 coming up next week, and stable 0.11 in the upcoming weeks if RC 2 is successful.

[thibaultcha]

Hi everyone!

As of today, all the work that we planned for this so far has been released in 0.11.0. So, before closing the conversation here, I would like to recap a few things.

This issue actually described two features:

1. The ability to set regexes in an API's uris attribute (and match an API with not only prefixes of URIs, but also dynamic path segments).
2. Extracting the dynamic path segments (whether named regex capturing groups or not), and substitute them in the upstream call URI.

As of today, 1. is available starting from Kong 0.11.0 🎉, and 2. is available as part of our Mashape Enterprise subscription. In our most recent release for our Enterprise customers, we've improved the request-transformer plugin to support substitution for Lua land values, such as: config.uri.replace=/users/$(uri_captures.user_id)/profile.

You can find more information about our Mashape Enterprise subscription and its additional features on top of Kong at: https://mashape.com/enterprise.

As a "bonus feature", I would like to highlight that Kong 0.11.0 also includes:

1. Consumption of the API router matches from the plugins via the new ngx.ctx.router_matches variable.

This should allow for some good hacking sessions in your Kong custom plugins :wink:

With that said, I believe we can now close this conversation and consider it implemented under the scope provided by this issue! You can checkout Kong 0.11.0 here: https://github.com/Mashape/kong/releases/tag/0.11.0

Thanks!

[merlindorin]

At least, we don't forget that kong is also an entreprise product 🤔

Thx for the work, this feature really change the game for us. For the first time we can purpose your product to our customer because it fits our development. 💌

I hope that the next big step will be an integration of Swagger (at least params/body/query sync validation) 👍

[mrproper]

What is the correct way to do a regex in a uris attribute, when i do it via curl it does not yield the results im after:
curl http://127.0.0.1:8001/apis/foo -X PATCH --data 'uris=/v1/foo/\d+/bar' curl http://127.0.0.1:8001/apis/foo -X GET { ... "uris" : [ "/v1/foo/\\d /bar" ], ... }

[RolphH]

@mrproper see https://github.com/Mashape/kong/pull/2681#issuecomment-314228542

[techtyler]

Is it possible to use this feature with QueryString parameters? I'm having issue with getting the regex to match the "?" character in the request uri. I want to set up 2 apis with the only difference being the presence of the query string (such as "?debug=true") is added.

Example (get user by ID with debug querystring)
RequestURI: /User/[0-9]{8}\?debug=true

I've attempted to escape and double escape the '?' but it seems to be ignored. If I use another character such as '&', everything works fine. There seems to be some issue with the reserved regex characters. I've also tried using the url encoded "%3F" but that doesn't get sent properly to my service and it does not pick up the query string parameters.

[thibaultcha]

@techtyler Copying here my answer to you from Gitter for future reference to this thread (please avoid cross-posintg in the future):

@techtyler Kong’s router evaluates URIs after stripping out the request arguments from the request line, which is why this regex will never match an incoming URI
@techtyler No plugin or core feature would allow you to route based on querystring args quite yet. You’d have to develop some in house solution in a plugin

Also, I will now be locking this issue to avoid further noise. The usual support channels (Gitter/Freenode/Mailing list) are the preferred channel of communication for getting help and asking questions. Today, GitHub issues are only looked at when actually reporting bugs. Thank you for your cooperation!