Kong: [community feedback requested] A new way to install and configure Kong

[thibaultcha]

Summary

1. Problem
2. Proposed new approach
   a. Installing
   b. Configuring
3. Pros and Cons
4. Community Feedback

1. Problem

Today's Kong can be quite un-intuitive and painful to both install and configure:

• For a source install, it requires installing separately several third-party software components (Serf/OpenResty/LuaRocks).
• Said source installs need a few flags during the Nginx compilation that Kong requires and that users need to add manually.
• Customizing the Nginx configuration requires creating a Penlight template with its own syntax and quirks, and users must update their template manually between Kong versions if new directives are added.
• When using a custom Nginx configuration, users have to maintain 2 configuration files: kong.conf, and nginx.conf.
• The Kong CLI is a process sitting between the user and the Nginx master process, and makes things like stdout logging or non-daemon mode harder for containers setups.
• Several Kong instances cannot run on the same machine at the same time.
• Installing the Kong Lua sources and dependencies requires an Internet access to luarocks.org and hosting code services like github.com.

Source installs are quite common because they allow building Nginx with custom flags (and custom Nginx C modules), and custom Nginx configurations are even more common, because kong.conf does not (and should not because it is not "scalable") support each and every Nginx directive.

2. Proposed new approach

a. Installing

Release Kong as an archive like kong-0.12.0.tar.gz containing:

• The OpenResty sources (which themselves contain the Nginx sources)
• LuaRocks
• The Kong core Lua sources
• The Kong community-edition, OSS plugins (currently, they live in the same repository but not for long)

The installation process for Kong would consist of:

$ wget http://getkong.org/download/kong-0.12.0.tar.gz
$ tar -xvf kong-0.12.0.tar.gz
$ cd kong-0.12.0
$ ./configure --prefix=/usr/local/kong
$ make
$ [sudo] make install

And that's it. Let's walk through the steps and what they achieved here:

$ wget http://getkong.org/download/kong-0.12.0.tar.gz
$ tar -xvf kong-0.12.0.tar.gz
$ cd kong-0.12.0

Here, we just downloaded the latest release archive and extracted it. Such an archive should be PGP signed by the core Kong contributors for each release, ensuring users get what they want.

$ ./configure --prefix=/usr/local/kong

This is a configure script that wraps the OpenResty one (which installs LuaJIT + Nginx). It allows any OpenResty flag, and thus, any Nginx flag as well. It also adds the flags required by Kong automatically, (such as --with-pcre-jit or --with-http_realip_module) so users don't have to worry about it. One can compile Kong like so:

$ ./configure --prefix=/usr/local/kong \
    --user=foobar \
    --with-http_v2_module \
    --with-http_geoip_module \
    --add-module=/path/to/custom/nginx/module \
    --add-module=/path/to/custom/nginx/module_2 \
    --with-debug

We now allow Kong users to very easily customize the underlying Nginx instance running Kong, without compromising the minimum flags required by Kong.

This step will also take care of installing LuaRocks with the newly compiled LuaJIT (from the OpenResty ./configure).

$ make
$ [sudo] make install

Just compilation and installation steps. During installation, we copy the Kong Lua sources into the prefix's lualib (the place where Lua modules required from an OpenResty installation should go to). LuaRocks mostly is bundled for future kong install [plugin] features relying on it.

b. Configuring

Let's see what the previous installation step actually installed:

bin    luajit    lualib    nginx    pod    luarocks    nginx.conf

Here, most of those files come from a regular OpenResty installation, but luarocks and nginx.conf are added by Kong (among other files in lualib and bin like the Lua sources and the CLI, but not detailed here).

What interests us here is the nginx.conf file, because it would be the only configuration file needed. That's right, no kong.conf.

Here is what such a nginx.conf file might look like, but take it with a grain of salt as it is a very early, draft document:

error_log logs/error.log notice;
access_log logs/access.log;

events {

}

http {
    server {
        listen 0.0.0.0:8000;

        location = / {
            # serve a webpage, like a developer portal at "api.example.com"
            # with Nginx settings optimized for assets serving/caching.
        }

        location / {
            # serve the API itself, at "api.example.com/v1"
            # with Nginx settings optimized for an API's reverse-proxy.

            kong_cassandra_contact_points 10.0.0.1,10.0.0.2;
            kong_cassandra_keyspace       kong;
            kong_cassandra_user           kong_user;
            kong_cassandra_password       password;

            kong_serve_proxy;
        }
    }


    # un-comment this block if you wish this node to also expose an Admin API
    # on localhost:8001
    #
    # server {
    #    listen 127.0.0.1:8001;
    #
    #    location / {
    #        # serve the Kong API, and maybe use any Nginx module to protect it,
    #        # such as HTTP Basic Auth if desired.
    #        # auth_basic_user_file conf/htpasswd;
    #
    #        kong_serve_admin_api;
    #    }
    #  }
}

This is achieved through a custom Nginx C module, that is transparent to the user because it is added under the hood by the Kong install phase (shipped in kong-0.12.0.tar.gz and automatically added to ./configure flags).

Thus, we allow any user to easily customize their Nginx instance to do anything they'd like:

• run another Nginx server block
• use a customized setting for a given block (tweak buffer size? use third-party module? serve a website? etc...)
• ability to enable/disable the Admin API on given nodes, chosen by the user

This is very beneficial in the sense that users now don't need to maintain two configuration files. The Kong C module will try to avoid as many required changes to the Nginx configuration as possible (like manually adding lua_shared_dict directives, and so).

Kong is now not started from its CLI anymore, but directly from Nginx, as in:

$ nginx -p /usr/local/kong

3. Pros and Cons

Going down this road has numeral benefits for users, power-users and contributors. Here is a non-exhaustive list:

• Easy and fast source installation for power-users and contributors, everything-in-one-(signed)-archive
• Easier maintainability for official Kong package maintainers (rpm, Homebrew, etc)
• Easier customization of the Nginx configuration, and avoid the 2 config files pattern
• No need to learn a new config file format (Kong's) or a new CLI (Kong's). We are Nginx-like.
• No process in between the user and the Nginx master process. We easily integrate into containers, and users can apply any Nginx technique they know (daemon off;, error_log stdout;...). This could potentially mean that Kong could come back on Heroku as well.
• No need for an access to luarocks.org and github.com when installing from source.
• Less need for users to track configuration changes: the C modules offloads as much as possible
• Ability to include any OpenResty or Nginx custom match that we'd like. No need for long waits between releases if we need something.
• Ability to use Nginx idioms like init.d scripts or monitoring solutions out of the box.
• Lock on our side the OpenResty/Nginx version shipped with Kong to ensure the best compatibility. Users cannot use Kong with just any version out there and end up with issues they are not supposed to.

There are a few losses as well that I will try to list objectively:

• Less "Kong brand" (our config lives in nginx.conf).
• The Nginx configuration might be harder to grasp for users who have never seen it before.

Those lists should be maintained and updated (edit this issue).

---

4. Community Feedback

Would love community feedback on this one! Thoughts? Concerns? Better ideas? Please comment and share!

[bungle]

We could still have kong.conf like this or users may create their own:

kong_cassandra_contact_points 10.0.0.1,10.0.0.2;
kong_cassandra_keyspace       kong;
kong_cassandra_user           kong_user;
kong_cassandra_password       password;

And a location:

location / {
    include kong.conf;
    # include kong-debug.conf;
    # include kong-production.conf;
    kong_serve_proxy;
}

[coopr]

Will the suggested changes make it notably easier, or harder, to discover, install, configure, upgrade, and remove plugins (Kong CE plugins, Kong Enterprise plugins, Community-supported plugins, custom plugins built and used by a single entity, etc)?

[thibaultcha]

We could still have kong.conf like this or users may create their own:

@bungle Good idea! Definitely a nice side-effect that we can take advantage of.

[thibaultcha]

Will the suggested changes make it notably easier, or harder, to discover, install, configure, upgrade, and remove plugins (Kong CE plugins, Kong Enterprise plugins, Community-supported plugins, custom plugins built and used by a single entity, etc)?

@coopr I don't think those two topic (the one covered in this issue and external plugins) relate very much. The idea would be to have a set of chosen plugins (living in other repositories) being fetched and bundled when the kong-0.12.0.tar.gz archive is being built, and installed in the LuaRocks tree. The blocker for this is having the plugins out of the core, and not this new Nginx packaging. If the plugins were out of the core, what you described would already be doable, but differently: we would add core CE plugins to the Kong rockspec and let LuaRocks install them when executing luarocks install kong. (Oh, that makes me think that another benefit of this entire method I forgot to add was that source install won't require Internet access anymore if one has the release archive). Only this newly described method is, I believe, cleaner and less dependent on LuaRocks. But that doesn't mean it is a blocker to "plugins granularity" (Maybe we should refer to plugins out of the core as "plugins granularity").

[ciberado]

Looks very nice. May a suggest it should be an easy way to specify the configuration when running Kong with Docker? something like docker run .... kong --config-uri http://config.local/production.conf?

[peterbsmith2]

Regarding the nginx config being harder to grasp, this is something that devs are used to coming over. With regard to this project specifically, a kong user is likely to be getting into nginx configuration at some point anyway as their demands on kong grow. This only makes the issue of learning nginx syntax come quicker.

[lsh-0]

I don't know lua and I didn't 'get' the magic that was happening inside Kong when I needed to add our strong SSL configuration. I'm sure it was possible, but it was easier to deal with vanilla nginx and proxy traffic to Kong. The above approach reads as becoming more nginx-orientated and removing some of the magic that would otherwise frustrate those (like me) already familiar with nginx. I could see this approach as shifting the responsibility during the long-Kong (the post-honeymoon period where you've sold your team on Kong) from Kong and Lua and deep internals to familiar and trusted Nginx. I hope that makes sense.

Lock on our side the OpenResty/Nginx version shipped with Kong to ensure the best compatibility. Users cannot use Kong with just any version out there and end up with issues they are not supposed to.

I'd be happy for Kong to be the nginx provider if I can be certain it's keeping up with security patches. Unless I read that incorrectly.

[thibaultcha]

@peterbsmith2

This only makes the issue of learning nginx syntax come quicker.

Definitely 👍

@lsh-0 Thanks for the feedback!

I'd be happy for Kong to be the nginx provider if I can be certain it's keeping up with security patches. Unless I read that incorrectly.

Yes you did read that correctly. Good point. At the same time, it is worth noting that OpenResty is our Nginx provider (Kong > OpenResty > Nginx), and as such, Kong keeps up with security patches as much as OpenResty does. OpenResty generally does a good job at staying up to date in terms of security patches. At the same time, the good news is that since we can also add custom patches to the Nginx core, we can provide a release before OpenResty does so, if it ever happens that it is lagging behind (which I doubt :) ).

[samrose]

My opinion FWIW:

Overall bringing Kong closer to openresty > nginx patterns into kong by way of using the nginx config + making it a better practice to make Kong vs install via package sound like simpler way to develop Kong, and a more flexible way to use/configure/deploy it in more environments

[jussiarpalahti]

I like this direction overall. If Kong were more "nginx-like" we could use regular Ansible scripts instead of having to maintain custom module + playbook. Also, there's wealth of documentation for nginx and experienced developers available.

Could you also explain how upgrading is done with this model? Download new Kong release, make install and then just start that nginx with your config? I've understood that migrations are run "automatically" when upon startup Kong sees that database has older schema.

How does this affect Kong clustering and its use of serf?

[thibaultcha]

@samrose @jussiarpalahti Thanks for taking the time to read this and share your thoughts.

@jussiarpalahti

Could you also explain how upgrading is done with this model?

Eventually, the migrations will keep being part of the CLI. Said CLI does not start and stop Kong directly anymore, but provides an utility belt (migrations, add-api, remove-api, version...). To upgrade, I hope we can somewhat follow the same steps as the Nginx binary upgrade path: download Kong, install, run $ kong-cli migrations up (this makes the running Kong node in migration mode, aka stops talking to the DB), and $ /usr/local/kong/sbin/nginx reload (with the appropriate conf flags) would inspect the DB on startup, and judge it to be up-to-date for its version. Sounds reasonable?

How does this affect Kong clustering and its use of serf?

Serf is currently being removed from Kong. That is the main blocker as of today for implementing this. Clustering will really be stateless, and events will be propagated with some database polling. More on that very shortly :)

[hungpt91]

Sorry but I $ wget http://getkong.org/download/kong-0.12.0.tar.gz ---> not found.
Did you mean wget https://github.com/Mashape/kong/archive/0.10.1.tar.gz ?
But i can't run command "./configure" in 0.10.1 folder . Plz help, tks so much.

[thibaultcha]

@hungpt91 This issue is to discuss the idea presented above, none of this exists as of today.

[jussiarpalahti]

@thibaultcha, upgrade model sounds good. re:clustering: interesting... :)

[francois-maillard]

Customizing the Nginx configuration requires [...] users must update their template manually between Kong versions if new directives are added.

indeed, that's why I proposed #3010. Comparing our nginx conf to the new templates was one of the most time consuming task when preparing upgrades. We mitigated by patching the official template instead of having a separate version, but that's hardly a clean solution. The above-proposed process seems like a much more elegant solution.

The Nginx configuration might be harder to grasp for users who have never seen it before.

On the other hand, it makes it easier for nginx users to grasp kong integration.

[bungle]

This may represent a long-term plan, but as we have decided to improve on injections, and pursuing into clearer dataplane / control separation, I will close this now. Thank you all for the feedback.