Kong: Storing sensitive information in database

[mleace]

Hello,

as far as I know, sensitive informations (like API keys, OAuth client secrets, OAuth access & refresh tokens etc.) are stored in database in plain text, which might be a security issue. Is there any way to store these informations encrypted (or hashed when applicable). If not, is there a plan to implement such feature? I know there will be a performance penalty, but still this might be necessary in some environments.

Thank you!

[arsdehnel]

curious about this as well, it'll be a deal breaker on Kong if there isn't a way to make this happen. Guessing I could create a new OAuth plugin that uses a hash algorithm on save and then again on authorization but having never written Lua that seems like a bad place to start (with cryptography).

[t1tcrucible]

This is no real comment, but sharing some experience. Maybe a Hashicorp vault can play a role, we had the same reflection. But until now no issue at our customers storing jwt, oauth tokens. This is because the Kong gateway port 8001 is only accessible from our own application, and infrastructural restrictions apply. We have a use case were we had to enrich payment transactions with merchant ID's; we are still in a conceptual phase of maybe using Hashicorp vault.

[jmdacruz]

It's important to separate "data at rest encryption" (which can be handled by the database directly and transparently) from one-way hashing. The first tries to mitigate the issue in which an attacker gains physical access to the database, so that the database files cannot be read directly. The latter tries to mitigate the issue in which an attacker gains access to an application server which can query the database, and hence gets access to some of the data via the query APIs of the database.

One-way hashing makes a lot of sense for long-lived sensitive information (e.g., passwords, tokens with long TTLs). One way hashing should be applied in a way that every single entry on the table that is hashed has a unique salt, which is also stored on the table along with other parameters for the algorithm. You would end up with something like this on the table (at a very high level):

• hashed_password = hash(original_password, salt, algorithm, [parameters, ...])
• salt
• algorithm
• parameters

Where salt is a random string, algorithm is your hashing algorithm of choice (e.g., PBKDF2), and parameters are parameters for the hashing algorithm (e.g., number of hashing rounds in PBKDF2, length of derived key)

This will allow you to change the hashing algorithm incrementally (remember, you'll not have access to the original information anymore), when for instance people start changing their passwords (you'll validate against the old hash, and then store the new password using a new algorithm with new parameters).

[arsdehnel]

I agree @jmdacruz that it is important to distinguish those two things. Due to contracts we have with our clients we need one-way hashing of this sort of information.

I like the idea of storing the algorithm components in the record for incremental hashing possibilities but I'd be concerned about performance of such a solution. Maybe that's unwarranted?

[jmdacruz]

What I meant by "incrementally" is that when you figure out that XYZ hashing algorithm is broken (i.e., somebody find a cryptographic attack on it making it easy to find collitions) and needs to be replaced, you can make this change incrementally when you ask your users to update their passwords... because you'll have some users with their old password and old hashing algorithms and some users with their new passwords and new hashing algorithms all at the same time in the same database. Having the data on the table is the only way to be prepared for this.

Also, having a salt per password is a best practice: Having a global salt for all passwords is the same as not having a salt at all, because an attacker can read your code and figure out that you have a global salt. With the approach above, if an attacker gets a hold of your database dump he'll see that you have a salt per password and this makes the search space for combinations of password and salt much bigger (an attacker would have to test every password in a dictionary with all of the salts on the database... so the search space increases with the number of users you have). Of course there could be other problems in the code that may reduce the search space, but that's a different problem.

In terms of performance, there is no much impact of having all this in the database and configurable vs. hardwired in your code... you would still have to use a reasonable hashing algorithm with a certain number of rounds.

[t1tcrucible]

@jmdacruz I agree aswel but if a user can access the data storage underneath Kong, I have much more concerns than only how to store tokens, keys, etc. A user can change the upstream uri's as well and perform MIT attacks (tampering with the provided headers for example). This is why from my point of view, Kong instances should be well protected in the infrastructure. In our architecture right now, Kong is only accessible from a business layer, managing those instances, an only those service instances can connect to the admin port (8001). Introducing authentication for the 'admin' connection is much more relevant in my opinion. The underlying Cassandra cluster is also enforced only to connect with Kong instances.
Can I state that we have setup Kong instances secured? I guess it is the case at the moment but this issue triggers some doubt (understanding your concern). Other architecture can require other security approaches towards Kong.

[arsdehnel]

Thanks again @jmdacruz I totally get what you meant by incrementally and like it from that standpoint. Actually really wish that someone had done that on a system I recently inherited that we're working on moving off MD5 hashing.

Working on coming to terms with the fact that I need to replace Kong or get much more into Lua than I'd planned when I got up this morning (which was 0% into Lua). Anyone have any experience with Lua crypto options that can recommend one?

[jmdacruz]

@t1tcrucible Absolutely agree in that you have to treat your Kong cluster as the corner stone of your deployment and protect it accordingly, no question about it. You have to follow network protection best practices, close your ports, implement ACLs, only allow the Kong admin API to be used from within your business environment, etc. All of those are must-haves and any decent network environment should have them in place.

Now, you have to prepare yourself for an attack... you have to think what will happen when an attacker gets in your system (not "if", but "when"... modern systems are comprised of a lot of software you don't control... you have nginx, OpenSSL, etc. which all have vulnerabilities waiting to be discovered). When somebody breaks into your system, you have to assume they'll get to your database... and at that point, the only defense you have is making it hard for attackers to get the original data. Worst thing that could happen to a company is being compromised and attackers stealing all the passwords in clear text... a nightmare because most people reuse their passwords in multiple sites.

Bottomline: _If you don't store the data in it's original form, then there is nothing for the attacker to exfiltrate._

Once an attacker is in your system, the fact that they can change the upstream URLs in Kong/Cassandra/Postgres should be the least of your concerns (it will be the last place where an attacker will look for sure): They could just replace your entire Kong installation with a simple server using a remote code execution vulnerability in nginx (just as an example... but you get my point). In order to prevent exfiltration of data, you should setup network/firewall ACLs so that the servers hosting Kong can only connect to well-known addresses and URLs.

[jmdacruz]

@arsdehnel Lua is actually a simple an elegant language once you start using it, there is no complex syntax and the semantic is very straightforward... My recommendation is that you work with the guys at kong in a contribution model (say, if you wanted to contribute modifications that help storing information with hashes, etc.). In terms of libraries, since Kong is built on top of OpenResty (which is a set of Lua modules for nginx), you should stay in that family of libraries ideally... and luckily there is a lot of support:

• https://github.com/openresty/lua-resty-string: The string library for OpenResty supports hashing algorithms already, which apparently includes some hashing with salt and multiple rounds already (probably not PBKDF2, but close... haven't looked close enough myself)
• https://github.com/openresty/encrypted-session-nginx-module: This gives you support for some encryption primitives (although I don't think you'll need them)

[arsdehnel]

I probably came off as annoyed or frustrated in my previous post, I'm more surprised that this hasn't already come up. I will definitely pull this down and start figuring out how to adjust the plugin to have crypto as an option, involving the Mashape team when appropriate or needed.

[jmdacruz]

Not at all @arsdehnel, I think you do have a valid point.

[mleace]

Quite a rich discussion with valuable opinions, thank you.

I agree that securing access to Kong cluster is a MUST, but I have seen too many security breaches to be satisfied just with what I would call an "infrastructure defense". @jmdacruz went much further than I originally planned, as my first thoughts was to store access & refresh tokens and client secrets (I'm mostly concerned by OAuth2 plugin) as plain SHA-256 hash, without salt. I know that salting makes it more difficult for potential attacker to misuse gained data, but I guess that using strong-enough hashing algorithm gives me enough time to change/disable these data before they can be misused.

This should not be very difficult to implement, considering that JWT plugin already depends on crypto module. I'm also new to Lua, but I think I will be able to create such implementation.

I started this thread mostly because I wanted to ask creators of Kong whether this consideration was made at all and if there are any plans to change storage in future - and if not, ask for a suggested way how to implement this ;)

[arsdehnel]

I think that's a good point @mleace where you're asking how Mashape feels about this and if it was already in any plans. Maybe @sinzone can help answer that (asking you since I noticed you gave a 👍 to the original question)?

As for the implementation details the straight SHA-256 method seems like a suitable first round of this and simplifies the implementation quite a bit. It would be enough for "my security guy" that I referenced as well.

[markyjones]

Other plugins that store user/human created credentials like the basic auth module will require a KDF like PBKDF2 etc.

This comment is focused specifically at the OAuth 2.0 plugin where keys are defaulted by the application and only supplied by the user for 'migration' reasons.

• access_tokens
• refresh_tokens
• client_secrets is interesting as a lot of OAuth providers seem to store these in a reversible/retrievable way - if this is required then AES 256 Encryption is likely the way forward; however, when used with the Client Credential Grant in OAuth 2.0 this secret is effectively a password and I would argue could be hashed - perhaps as a configurable option.

For OAuth 2.0 (access token and refresh) +1 for SHA-256 for storage.

If we have system generated 128bit cryptographically random passwords we can get away with a lightweight SHA 256 hash to make the data statistically irretrievable by virtue of its extreme entropy yet be quickly compared/validated on every web request.

Using a salt here isn't really going to help as passwords are already truly random thus there is no concern over dictionary and pre-computed rainbow tables.

Key stretching techniques (KDF) like PBKDF2, scrypt, bcrypt etc make less sense on system generated ultra high entropy keys and in the case of access_tokens especially cost a lot of CPU validating on every request.

Some other suggestions from our team include perhaps using of a Pepper or a final AES 256 encryption of the values but this will involve keeping a key safe on the Kong server for this purpose and support rotation etc.

Concerns about the key generation

The above approach depend on the quality of the key which is not quite perfect in Kong as it stands.

The tokens are generated by Kong via Lua calls to a function random_string() here which creates a V4 uuid using a Lua library called lua-uuid which in tern uses hooks to libuuid which reads cryptographically random bytes (CSPNG) from /dev/random or /dev/urandom in code here and manual here.

Lua-uuid appears to fall back to a _non_ CSPNG approach if run on an OS without /dev/random or /dev/urandom which would be bad as it would then be merely unique and not random.

The OpenSSL libraries here and here may be a better way to get the key so we are explicitly getting a CSPNG derived key.

[djMax]

Honestly, to me this issue is very high priority and the fact that secrets and provisioning keys are in the clear in the db is amateur hour. These fields need to be bcrypted on their way into the db and validation functions updated. We're going to try and give this a shot because there's no way we'd go live with an implementation like this...

[hutchic]

@djMax are you thinking Kong itself needs to encrypt / decrypt those fields?

I personally feel like between encryption at rest / one of the postgres encryption options and SSL to postgres that should cover the threat vectors people seem to be worried about.

Encrypting / decryption persistent datastores can sometimes just end up being security theater when the decryption key is immediately accessible.

[djMax]

That's why we should use bcrypt and not encryption

[djMax]

Any plan to address this? Are there good docs on setting up an env for core Kong development?

[djMax]

I've got a fork that fixes this for provision_key which for us is probably the worst offender. For refresh_token I assume this is much harder because of the lookup?

[hutchic]

If you have a fork feel free to PR :+1:

[ginnavaram]

are there any updates by Mashape/Kong on implementing this?

[bungle]

I did a quick check, e.g. OAuth2 plugin and client_secret, I don't see a place where we need to access the client_secret in its original / reversible form (did I miss something). I think it would be a good idea to add config option for hashing / key derivation. It is reasonable to support both one-way hashing like sha1/sha2/sha3 (aka fast), and key derivation like pbkdf2, bcrypt, scrypt, argon2 (aka slow), also as @markyjones said, we could auto generate it from random by default.

[ginnavaram]

In "client credentials flow" client_secret might be used in its original/reversible form while sending the request but its still a good idea to support hashing....

[lvillarino]

are there any updates by Mashape/Kong on implementing this?

[thibaultcha]

@lvillarino This is definitely on our radar.

[p0pr0ck5]

From @calvinlfer:

Tools like Kongfig and Biplane give you the ability to back up your Kong configuration so that it can be versioned and maintained. One problem I noticed when doing dumps/backups is that for basic authentication, it will dump out the encrypted password for basic authentication consumer credentials. When a synchronization/upload is done, it will actually use the already encrypted credentials and further encrypt it which causes your actual credentials to stop working. It would be very helpful when dumping out the credentials via GET consumers/:consumerId/basic-auth to include the fact that you were obtaining encrypted credentials and also have an encrypted flag in the configuration to allow you to upload the backed up encrypted data so that your credentials wouldn't get incorrectly updated.

[Witchfinder]

Are we any closer to this being resolved? It appears to have been an open issue and "under discussion" for 15 months now.

[ginnavaram]

@thibaultcha Any updates on this?

[siamaksym]

+1. Seems this is a necessary feature as storing API keys in plain text in the DB is unacceptable in many organizations.

[jharitesh108]

@thibaultcha Any ETA for this feature?

[p0pr0ck5]

Just as an update, this functionality is available as an alpha-quality feature made available in the most recent release of Kong Enterprise. See https://docs.konghq.com/enterprise/1.3-x/db-encryption/ for an overview of functionality.