Kong: [request] support for DynamoDB

[nickveenhof]

Cassandra is quite a beast to setup and maintain. If Kong runs in EC2,it would be great if it could also write it's data to DynamoDB (very similar to Cassandra) and use that as a service. This allows engineers to have less maintenance on Cassandra and focus more on Kong.

Great project!

[thibaultcha]

We are aware of this, and support for other databases is on the roadmap, but we have a lot of things to do before that.

[jakehow]

:+1:

[DavidTPate]

For me, supporting another database which I can have managed (see RDS/DynamoDB/etc) would be awesome.

[sonicaghi]

btw.. @DavidTPate @nickveenhof you can have Cassandra in the cloud through https://www.instaclustr.com/

[jasonfill]

+1

[berkay]

+1
IMO, Cassandra dependency is a major problem. It would be unwise for anyone to build such a critical service (API) with a dependency on a data store that one does not fully grok but have to manage. DynamoDB would be a good fit.

[desmondmorris]

+1

[glkz]

:+1:

[owahlen]

+1

[vmattos]

:+1:

[kenwarner]

+1

[skippy]

+1

[udleinati]

+1

[vwal]

+1

While PostgreSQL support would be great, support for DynamoDB would be even more critical for AWS applications. If KONG is used on Auto-Scaling Group instances, DynamoDB access would allow much lighter instances to be used as Cassandra is a resource hog.

[sonicaghi]

@vwal that is true. and we're def going to do both. We just prioritized Postgres at the moment given more community demand for it.

[seiffert]

:+1:

[gankkank]

:+1: Glad to hear that

[harlow]

While PostgreSQL support would be great, support for DynamoDB would be even more critical for AWS applications. If KONG is used on Auto-Scaling Group instances, DynamoDB access would allow much lighter instances to be used as Cassandra is a resource hog.

Agreed. :+1: for DynamoDB.

[jtconnor]

+1 for DynamoDB

[RussellLuo]

I think it will be awesome if KONG provides the extension/plugin support for different databases.

[eric1iu]

+1

[forhot2000]

+1

[truman-misfit]

+1

[exquisite2007]

+1

[guangbochen]

+1

[johnzeng]

+1

[akurkin]

+1

[nhuray]

+1

[lindo-jmm]

Installing Cassandra is a complete nightmare.

[sonicaghi]

@lindo-jmm https://www.instaclustr.com/products/kong/

[lindo-jmm]

I saw! Starting at $299/mo is a little steep for us unfortunately. I spun up a few Azure Hosts in Rancher w/ 128GB SSDs so I’ll just hope the pager doesn’t go off in the middle of the night.

[sonicaghi]

@lindo-jmm maybe try w/ three nodes at $20/month? (dev edition)

[lindo-jmm]

Do you think those are production ready, in your opinion?

On Fri, Jan 29, 2016 at 2:25 PM -0800, "Augusto Marietti" <[email protected][e-mail]> wrote:

@lindo-jmmhttps://github.com/lindo-jmm maybe try w/ three nodes at $20/month?

Reply to this email directly or view it on GitHubhttps://github.com/Mashape/kong/issues/183#issuecomment-176997348.

[AndrewDryga]

+1

[SGrondin]

@lindo-jmm There's also the $80/month option here: https://www.instaclustr.com/products/developers/ Feel free to contact [email protected] if you have questions. From what I can tell, the "dev" plans are simply smaller EC2 instances, they're not "lower quality".

[jkwong]

+1

[slater-ben]

Hello all,

I'm Chief Product Officer at Instaclustr so I thought I could provide some info on suitability of our dev offerings for Kong. SGrondin is correct, they are not "lower quality" and are very reliable for many small use cases. The main things to be aware of are:

• Both dev offerings are based on t2 class AWS instances (t2.small and t2.medium). Both of these are CPU credit limited so if you run them too hard for too long they will run out of credits and drastically slow down. Not a problem when you're getting started but you want to keep an eye on it once your API starts to take off (in particular, I wouldn't recommend the t2.smalls for any consistent load - but OK if you're only getting a few calls an minute).
• The dev offerings don't come with an SLA (not because they are less reliable, just because they're cheap).
• The dev offerings don't support SSL connections to Cassandra (due to resource limitations) so we'd recommend using VPC peering for security (you can set this up through our console).

If you do have any questions then you can start a chat with our support staff from support.instaclustr.com or you can email [email protected].

Cheers
Ben

[BrianHutchison]

+1 for DynamoDB support. Please.

[Mgutjahr]

+1

[DavidTPate]

I've gone some more thought to this recently. Not sure what the stance of the Mashape team is, but I'm not sure that DynamoDB is a good fit unfortunately. The reason why is that you have a lot more to manage with something like Kong that DynamoDB just doesn't do for you and could cause negative impacts to the usage of Kong (and likely perceived as an issue with Kong when really it's due to the limitations of DynamoDB).

For example, in order to properly use DynamoDB for Kong you're going to need to implement some sort of auto-scaling for your DynamoDB table(s) which will monitor their usage and increase their provisoning prior to it being an issue. This comes into play for Kong because it is going to need to Read/Write to DynamoDB and it will be constrained by how throughput works in DynamoDB. The issue here is that throughput might be exceeded and the request will fail (and Kong will have to throttle itself somehow for requests to DynamoDB). This sounds like a very poor fit to me because of the need to do all of this.

I wonder if instead utilizing a database like PostgreSQL with the JSON type (or other implementation) would be a better fit.

[grantcarver]

@DavidTPate I disagree a bit with that. I don't see any difference between having to manage a Cassandra cluster, Postgres servers, or DynamoDB. In every case you are responsible for ensuring the data layer's ability to manage the load you are intending to put onto it. In addition those wanting to use DynamoDB as a datastore for Kong are likely to be aware of any limitations as they are familiar with the platform (and not with Cassandra which is why Postgres support was such a welcome addition!). A note in the docs is sufficient in my opinion - us users are pretty capable.

[nickveenhof]

Completely agree with grantcarver. We've switched an application from Cassandra to DynamoDB with great success. The fact that we don't have to manage the beast that cassandra is, was such a relieve for us. That said, obviously we still need to tune how we access that data, in the same was as we had to do with Cassandra.

The Throguhput exceeding issue can be easily resolved with a tool like https://github.com/sebdah/dynamic-dynamodb so that's a non-argument. Postgres is indeed a good addition, as this can actually be started as a saas product that removes some of the maintenance burden from small teams. I still think DynamoDB is a great fit for Kong and it would, in my opinion, also increase its adoption by a fair amount given that it greatly increases the ease of installation.

[lbadger]

Any progress on supporting DynamoDB?

[HitDaCa]

Any progress on supporting DynamoDB?

[vvondra]

This is now even much more interesting with Global DynamoDB tables - you can have regionally distributed Kong deployments as with Cassandra

[aelesbao]

Having DynamoDB support would be amazing!