Kong: Service URL formation changed when using routes with strip_path=false

[dliberman]

Summary

Service request URL formation changed after KongCE 0.15 if you create routes with strip_path=false. If you create the service as url=https://mockbin.org/request (without the trailing slash), for example, and add a route /something to it, it will form the url as mockbin.org/requestsomething

Steps To Reproduce

1. Create a service:

curl -i -X POST \
  --url http://localhost:8001/services/ \
  --data 'name=potato' \
  --data 'url=https://mockbin.org/request'

1. Create a simple route for the service, with strip_path=false

curl -i -X POST \
  --url http://localhost:8001/services/potato/routes \
  --data 'strip_path=false' \
  --data 'paths[]=/fries'

1. Make an http call to this server/path (using httpie here):

http localhost:8000/fries

1. Check result JSON:
2. On 0.14.1 and before (also tested and works on Kong EE 0.33-2), request url formed is http://localhost/request/fries. On 0.15 and after (tested also on 1.1.0) the url formed is http://localhost/requestfries"

Additional Details & Logs

• Kong version; 0.15 and after, tested until 1.1.0
• As mockbin.org accepts requests like mockbin.org/request, mockbin.org/requestsomething, mockbin.org/request/something, this is easy to reproduce there.
• As stated in the summary, it works if you create the service with the trailing slash at the url. But this will break some services we have in place already (when migrating from <0.15 to >=0.15) that have more complex url syntax.
• Full result from example request:

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: host,connection,x-forwarded-for,x-forwarded-proto,x-forwarded-host,x-forwarded-port,x-real-ip,kong-cloud-request-id,kong-client-id,user-agent,accept-encoding,accept,x-request-id,via,connect-time,x-request-start,total-route-time
Access-Control-Allow-Methods: GET
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 996
Content-Type: application/json; charset=utf-8
Date: Wed, 03 Apr 2019 17:35:26 GMT
Etag: W/"3e4-Sq0Wd/KoVkJPqkIVPN3ULQ"
Kong-Cloud-Request-ID: df23405223db9dce258be56fc49df266
Server: openresty/1.13.6.2
Vary: Accept, Accept-Encoding
Via: kong/0.34-1-enterprise-edition
X-Kong-Proxy-Latency: 217
X-Kong-Upstream-Latency: 557
X-Kong-Upstream-Status: 200
X-Powered-By: mockbin

{
    "bodySize": 0,
    "clientIPAddress": "192.168.0.1",
    "cookies": {},
    "headers": {
        "accept": "*/*",
        "accept-encoding": "gzip, deflate",
        "connect-time": "1",
        "connection": "close",
        "host": "mockbin.org",
        "kong-client-id": "mockbin",
        "kong-cloud-request-id": "df23405223db9dce258be56fc49df266",
        "total-route-time": "0",
        "user-agent": "HTTPie/0.9.8",
        "via": "1.1 vegur",
        "x-forwarded-for": "192.168.0.1, 10.1.192.50, 18.204.28.183",
        "x-forwarded-host": "localhost",
        "x-forwarded-port": "80",
        "x-forwarded-proto": "http",
        "x-real-ip": "189.112.8.241",
        "x-request-id": "26d71355-f5a0-430e-abda-96151519790b",
        "x-request-start": "1554312926176"
    },
    "headersSize": 535,
    "httpVersion": "HTTP/1.1",
    "method": "GET",
    "postData": {
        "mimeType": "application/octet-stream",
        "params": [],
        "text": ""
    },
    "queryString": {},
    "startedDateTime": "2019-04-03T17:35:26.176Z",
    "url": "http://localhost/requestfries"
}

• Kong configuration:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 6103
Content-Type: application/json; charset=utf-8
Date: Wed, 03 Apr 2019 17:29:37 GMT

{
    "configuration": {
        "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
        "admin_access_log": "/dev/stdout",
        "admin_error_log": "/dev/stderr",
        "admin_listen": [
            "0.0.0.0:8001"
        ],
        "admin_listeners": [
            {
                "http2": false,
                "ip": "0.0.0.0",
                "listener": "0.0.0.0:8001",
                "port": 8001,
                "proxy_protocol": false,
                "ssl": false,
                "transparent": false
            }
        ],
        "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
        "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
        "admin_ssl_enabled": false,
        "anonymous_reports": false,
        "cassandra_consistency": "ONE",
        "cassandra_contact_points": [
            "127.0.0.1"
        ],
        "cassandra_data_centers": [
            "dc1:2",
            "dc2:3"
        ],
        "cassandra_keyspace": "kong",
        "cassandra_lb_policy": "RequestRoundRobin",
        "cassandra_port": 9042,
        "cassandra_repl_factor": 1,
        "cassandra_repl_strategy": "SimpleStrategy",
        "cassandra_schema_consensus_timeout": 60000,
        "cassandra_ssl": false,
        "cassandra_ssl_verify": false,
        "cassandra_timeout": 60000,
        "cassandra_username": "kong",
        "client_body_buffer_size": "8k",
        "client_max_body_size": "0",
        "client_ssl": false,
        "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
        "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
        "custom_plugins": [
            "ifood-kong-remote-addr-plugin",
            "ifood-kong-jwt-plugin",
            "ifood-kong-rbl-plugin",
            "ifood-kong-request-transformer-plugin"
        ],
        "database": "postgres",
        "db_cache_ttl": 0,
        "db_resurrect_ttl": 30,
        "db_update_frequency": 5,
        "db_update_propagation": 0,
        "dns_error_ttl": 1,
        "dns_hostsfile": "/etc/hosts",
        "dns_no_sync": false,
        "dns_not_found_ttl": 30,
        "dns_order": [
            "LAST",
            "SRV",
            "A",
            "CNAME"
        ],
        "dns_resolver": {},
        "dns_stale_ttl": 4,
        "dns_valid_ttl": 120,
        "enabled_headers": {
            "Server": false,
            "Via": false,
            "X-Kong-Proxy-Latency": true,
            "X-Kong-Upstream-Latency": true,
            "X-Kong-Upstream-Status": false,
            "latency_tokens": true,
            "server_tokens": false
        },
        "error_default_type": "text/plain",
        "headers": [
            "latency_tokens"
        ],
        "kong_env": "/usr/local/kong/.kong_env",
        "loaded_plugins": {
            "acl": true,
            "aws-lambda": true,
            "azure-functions": true,
            "basic-auth": true,
            "bot-detection": true,
            "correlation-id": true,
            "cors": true,
            "datadog": true,
            "file-log": true,
            "hmac-auth": true,
            "http-log": true,
            "ifood-kong-jwt-plugin": true,
            "ifood-kong-rbl-plugin": true,
            "ifood-kong-remote-addr-plugin": true,
            "ifood-kong-request-transformer-plugin": true,
            "ip-restriction": true,
            "jwt": true,
            "key-auth": true,
            "ldap-auth": true,
            "loggly": true,
            "oauth2": true,
            "post-function": true,
            "pre-function": true,
            "prometheus": true,
            "rate-limiting": true,
            "request-size-limiting": true,
            "request-termination": true,
            "request-transformer": true,
            "response-ratelimiting": true,
            "response-transformer": true,
            "statsd": true,
            "syslog": true,
            "tcp-log": true,
            "udp-log": true,
            "zipkin": true
        },
        "log_level": "debug",
        "lua_package_cpath": "",
        "lua_package_path": "./?.lua;./?/init.lua;",
        "lua_socket_pool_size": 30,
        "lua_ssl_verify_depth": 1,
        "mem_cache_size": "128m",
        "nginx_acc_logs": "/usr/local/kong/logs/access.log",
        "nginx_admin_directives": {},
        "nginx_conf": "/usr/local/kong/nginx.conf",
        "nginx_daemon": "off",
        "nginx_err_logs": "/usr/local/kong/logs/error.log",
        "nginx_http_directives": [
            {
                "name": "lua_shared_dict",
                "value": "prometheus_metrics 5m"
            }
        ],
        "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
        "nginx_kong_stream_conf": "/usr/local/kong/nginx-kong-stream.conf",
        "nginx_optimizations": true,
        "nginx_pid": "/usr/local/kong/pids/nginx.pid",
        "nginx_proxy_directives": {},
        "nginx_worker_processes": "auto",
        "origins": {},
        "pg_database": "kong",
        "pg_host": "kong-database",
        "pg_port": 5432,
        "pg_ssl": false,
        "pg_ssl_verify": false,
        "pg_timeout": 60000,
        "pg_user": "kong",
        "plugins": [
            "bundled",
            "ifood-kong-remote-addr-plugin",
            "ifood-kong-jwt-plugin",
            "ifood-kong-rbl-plugin",
            "ifood-kong-request-transformer-plugin"
        ],
        "prefix": "/usr/local/kong",
        "proxy_access_log": "/dev/stdout",
        "proxy_error_log": "/dev/stderr",
        "proxy_listen": [
            "0.0.0.0:8000",
            "0.0.0.0:8443 ssl"
        ],
        "proxy_listeners": [
            {
                "http2": false,
                "ip": "0.0.0.0",
                "listener": "0.0.0.0:8000",
                "port": 8000,
                "proxy_protocol": false,
                "ssl": false,
                "transparent": false
            },
            {
                "http2": false,
                "ip": "0.0.0.0",
                "listener": "0.0.0.0:8443 ssl",
                "port": 8443,
                "proxy_protocol": false,
                "ssl": true,
                "transparent": false
            }
        ],
        "proxy_ssl_enabled": true,
        "real_ip_header": "Incap-Client-IP",
        "real_ip_recursive": "on",
        "ssl_cert": "/usr/local/kong/ssl/kong-default.crt",
        "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
        "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
        "ssl_cert_key": "/usr/local/kong/ssl/kong-default.key",
        "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
        "ssl_cipher_suite": "modern",
        "ssl_ciphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256",
        "ssl_preread_enabled": true,
        "stream_listen": [
            "off"
        ],
        "stream_listeners": {},
        "trusted_ips": [
            "0.0.0.0/0"
        ],
        "upstream_keepalive": 60
    },
    "hostname": "c080b425404f",
    "lua_version": "LuaJIT 2.1.0-beta3",
    "node_id": "ebc03f07-c063-4042-9982-feabe056a249",
    "plugins": {
        "available_on_server": {
            "acl": true,
            "aws-lambda": true,
            "azure-functions": true,
            "basic-auth": true,
            "bot-detection": true,
            "correlation-id": true,
            "cors": true,
            "datadog": true,
            "file-log": true,
            "hmac-auth": true,
            "http-log": true,
            "ifood-kong-jwt-plugin": true,
            "ifood-kong-rbl-plugin": true,
            "ifood-kong-remote-addr-plugin": true,
            "ifood-kong-request-transformer-plugin": true,
            "ip-restriction": true,
            "jwt": true,
            "key-auth": true,
            "ldap-auth": true,
            "loggly": true,
            "oauth2": true,
            "post-function": true,
            "pre-function": true,
            "prometheus": true,
            "rate-limiting": true,
            "request-size-limiting": true,
            "request-termination": true,
            "request-transformer": true,
            "response-ratelimiting": true,
            "response-transformer": true,
            "statsd": true,
            "syslog": true,
            "tcp-log": true,
            "udp-log": true,
            "zipkin": true
        },
        "enabled_in_cluster": []
    },
    "prng_seeds": {
        "pid: 136": 199937721731,
        "pid: 143": 198253806061,
        "pid: 144": 215187153518,
        "pid: 145": 190112150124,
        "pid: 146": 180157229128,
        "pid: 147": 823018214214,
        "pid: 148": 132422391495,
        "pid: 149": 394317206200,
        "pid: 150": 178471351020
    },
    "tagline": "Welcome to kong",
    "timers": {
        "pending": 3,
        "running": 0
    },
    "version": "0.15.0"
}

• Operating system: centos docker image (kong:0.15-centos)
• If I may, looking at https://github.com/Kong/kong/compare/0.14.1...0.15.0, I didn't revert the change and didn't debug enough to say it could be it, but 62422bf6fadd56122fc5b42c7ec20727e263292e looks related to it.

[Tieske]

That was an intentional change. See tests here: https://github.com/Kong/kong/blob/master/spec/01-unit/08-router_spec.lua#L1773-L1858

Because matching is not bound to a segment, so neither is path construction

[bungle]

@dliberman,

what you want is probably this in service:

curl -i -X POST \
  --url http://localhost:8001/services/ \
  --data 'name=potato' \
  --data 'url=https://mockbin.org/request/'

See the added / at the end.

EDIT: Nevermind, you already mentioned it:

As stated in the summary, it works if you create the service with the trailing slash at the url

[carnei-ro]

@Tieske and @bungle the thing is: not every upstream works with a trailing slash. See the example of https://httpbin.org/anything/ vs https://httpbin.org/anything, and so do us have services like that behavior. How do we proceed? Kong used to fit in our environment.

[dliberman]

Also, I'm searching through lots of REST examples and guidelines and there is no consensus: although some places suggest to use a trailing slash when the API is related to a collection of elements, some places strongly suggest for API URLs/URIs never to end with a trailing slash to avoid confusion. @carnei-ro 's example above is a good one - httpbin.org/anything/ does not respond, when httpbin.org/anything/something will respond well (which means it still accepts a path hierarchic definition, but not necessarily accepts a request with a trailing slash if there's nothing to define/query under this "collection").
Enough said, it will definitely break several service endpoints for us here. :)

[bungle]

@carnei-ro / @dliberman,

that httpbin example is a really good one, thank you. I need to think this more, is there way to somehow support both (without new config params), or are the two use cases mutually exclusive.

[bungle]

At the moment I can only think of something like:

service.path="/request[/]"

Or adding another config parameter:

service.path="/request"
service.path_separator="/"

Where request path (if not equal to /) is appended to service.path with the specified path_separator.

Or something similar, there are many ways if we go a route of adding config parameter here.

[carnei-ro]

@bungle is there any way to test it with the mainline version? I've just tried with "/anything[/]" but got "invalid path: '/anything[/]' (characters outside of the reserved list of RFC 3986 found)"

[bungle]

@carnei-ro,

no, I was just thinking a path forward here. e.g. just dropping ideas. Maybe someone can suggest something better. Also what is the default behavior we need to decide. In general I think the old behavior is perhaps a bit more sane and expected.

[bungle]

@carnei-ro and @dliberman,

I added proposal for discussion here:
https://github.com/Kong/kong/pull/4538

With that you could create service like this:

curl -i -X POST \
  --url http://localhost:8001/services/ \
  --data 'name=potato' \
  --data 'url=https://mockbin.org/request'
  --data 'path_separated=true'

[dliberman]

Thank you @bungle!
I was just looking at your commit - wondering what would be the reason to have this new config property set as default to false and not true. Do you think this could be a breaking change if set as default to true?

[bungle]

@dliberman,

yes that is about to be debated. the sad situation is that we broke it with 1.0 to 1.1 (I am not sure if that was overlooked, or just that we weren't fully aware of the nature of the breakage). Now if we set this to true we will be breaking again from 1.1 to 1.2, while false keeps it. So question is whether the first breakage was a mistake (afaik it was not). We do probably have more users on pre-1.1 still, so by changing it to true, will serve them.

[carnei-ro]

@bungle why not maintain as the old versions (until 0.14.1)?

[hishamhm]

This is finally fixed with #5360 ! Both route path construction algorithms will be supported in Kong 2.0 (and the default reverted back to the 0.x behavior)

[marckhouzam]

This is finally fixed with #5360 ! Both route path construction algorithms will be supported in Kong 2.0 (and the default reverted back to the 0.x behavior)

To clarify, by looking at https://github.com/Kong/kong/pull/5463/commits/0dbf8521c51b475afc984eb3227d284b8a483014, I believe the default behaviour was kept to be the 1.x behaviour. Is that right?

In that case, IIUC, it means that someone upgrading to Kong 1.x from 0.x must manually fix any service that:

• uses a path without a trailing / AND
• has a route with strip_path=false

Is this correct?

I'm preparing our migration, that is why I'm asking. Thanks in advance for any confirmation or correction to my understanding.

[hishamhm]

@marckhouzam I understand this might be confusing, so here's a breakdown:

• Running Kong 1.5.0:
  
  
  
  • When migrating from 0.x, the migrations of 1.5.0 will detect that and set the routes to have the 0.x behavior ("path_handling": "v0") so that they keep working just like the user's previous version 0.x.
  
  
  • When migrating from 1.x into 1.5, the migrations of 1.5.0 will detect that and set the existing routes to have the 1.x behavior ("path_handling": "v1") so that they keep working just like the previous version 1.x.
  
  
  • When running 1.5.0, any new routes created will have the 1.x behavior by default ("path_handling": "v1"), but the user can set either behavior explicitly when creating a route.
  
  
• Running Kong 2.0.0:
  
  
  
  • When migrating from 1.x into 2.0.0, the migrations of 2.0.0 will detect that and set the existing routes to have the 1.x behavior ("path_handling": "v1") so that they keep working just like the previous version 1.x, unless those are routes migrated by 1.5.0 and set to have 0.x behavior, in which case that will be preserved.
  
  
  • When running 2.0.0, any new routes created will have the restored 0.x behavior by default ("path_handling": "v0"), but the user can set either behavior explicitly when creating a route.
  
  

In short:

• both 1.5.0 and 2.0.0 have smart migrations which detect which behavior the existing routes expected at the time of their creation, and will set path_handling accordingly so that those routes preserve their existing behavior.
• From Kong 2.0.0 onwards the new default behavior is the same behavior we used to have in Kong 0.x.

[marckhouzam]

Thanks @hishamhm for the detailed answer, it helps a lot.

From what I undersand, migrating from 0.x to 1.5, will keep working as is, thanks to the migrations of 1.5.
However, new routes will behave differently than old routes (this could cause confusion to my users).
Also, 2.0.0 switches back to the 0.x behaviour.

I'm therefore thinking that I should force "path_handling": "v0" to all new routes deployed to 1.5. That would be more future-proof (in preparation for 2.0.0) and consistant (to have old routes and new routes behave the same). Does that seem like a sound approach?

Thanks again.