Webiny-js: Multi-cloud support

[Ashu96]

Add support for GCP and Azure

Among all major cloud providers currently, we only support AWS out of the box.
Support of other cloud providers like GCP and Azure is in our product roadmap.

The goal of this issue is to discuss the support of other cloud providers mentioned above.

[gustavotr]

I want to help with the GCP support, how do we start?

[Pavel910]

@gustavotr First we need to figure out what alternatives do we have for CloudFront, API Gateway and S3 in the GCP world, as those are the main services Webiny depends on. Well, obviously Lambda as well, but for that we have functions.

So first we need a high level plan on how we will deploy Webiny as a whole (or partially?) to particular services.

[gustavotr]

For S3, Cloud Storage is an obvious choice. I know that GCP has an API service called API endpoints but I never used and don't know if if it has what we need. And for CDN we might be able to use Cloud Storage configuration.

[Pavel910]

@gustavotr I'd be really interested in seeing a working PoC, not involving Webiny, just plain GCP setup with some simple code. We need to cover the following request flow:

CDN -> API Gateway -> Lambda -> S3 and back to the browser. Also it's important to be able to flush CDN cache via an API (it's used by our Page Builder, and is very important for SSR of pages).

Once we have that, we'll see what needs to be upgraded on the Webiny code side (we know of several things already, but it's not important until we know how GCP works).

[gustavotr]

I have managed to create a similar API Gateway -> Lambda -> S3 flow. I will add the CDN next but from what I have seen it's not very difficult. How can I share what I have done after I finish?

[Pavel910]

Maybe you could share it as a github repo? Just create a repository on your account and post a link here. Thanks a lot for spending your time on this!! ❤️

[gustavotr]

I finally got some free time so I created a repo with my current progress. I still have to configure the CDN correctly. Probably I will have to configure with a real domain to make it work. Here is my repo -> https://github.com/gustavotr/serverless-api-over-cdn-on-gcp

[gustavotr]

Finally created the CDN and updated the repo. I had to use gcloud cli since the feature that creates a serverless network endpoint on terraform is on their master branch but did not get released yet. I believe that it will be available on the next one.

[Pavel910]

@gustavotr Lovely, thanks a lot! I'll try to take a look at it one of these days, we're a bit overwhelmed with different tasks and calls, but I'm writing this down so it's not forgotten :)

[ramonck]

I'm looking forward to using Azure so looking at static web apps with Cosmos DB.

[Pavel910]

@gustavotr I went through the code, thanks a lot for putting this together! 🚀

It looks pretty straightforward, but will will first need to introduce some improvements on the existing code base to be able to deploy to different cloud providers.

Main things to upgrade:

• invocation of functions - we currently use AWS SDK directly in our code to invoke Lambda functions - we'll need to introduce an "adapter" which would work via a plugin so you can control function invocation depending on what provider you're using.
• file manager - I believe this is something that will have to be recreated for GCP specifically (we'll most likely introduce a new package for GCP), due to differences in S3/Storage. We'll have to review this more in-depth with @doitadrian.

Just so you know, we'll be migrating to Pulumi for infrastructure deployments. They support a wide range of cloud providers, and make it really easy to define infrastructure via code (TS).

I'm going to create a separate issue for Function invocation plugin with some details on how to approach the problem. I'll tag you in the issue and if you're willing to give it a go, we'll be more than happy to help :)

[gustavotr]

Count me in @Pavel910 .

[smartboyathome]

I would love to see webiny running on Azure. It shouldn't be a hugely challenging migration, since it has a lot of equivalent services. Cosmos DB has Mongo support, Blob Storage is similar to S3, CDN and API definitions should be a pretty straight forward port as well. The largest challenge is likely to come from the porting of the Lambda to Azure Functions, since the latter is event-based, requiring you to define bindings and triggers.

[ramonck]

It's almost impossible to have all of these different APIs from PaaS of different companies working under one bucket. The closest you'll get to that is on the serverless product but it's focused only on the Functions part.
It's either a "buy in" into those vendors or have a mesh Docker Swarm/Kubernetes setup, it's the easiest path. And depending on the case can be cheaper to sideload the nodes then to actually have the buy in into those flagship kubernetes, they're charging by the hour on those products it's insane to pay per hour for a controller specially depending on the size of the company it's ridiculous for smaller companies, just cheaper to do a Lightsail sideloading and done for the day.

[Pavel910]

@ramonck the idea here is to have separate configuration templates for each vendor: AWS, Azure, whatever. When you create your Webiny project, we'll ask you which vendor you want to use, and will create infrastructure configuration specific for that vendor. From Webiny v5 we'll be using https://www.pulumi.com/ to handle all the infrastructure, and that will allow us to support different cloud providers.

So your project will not be deployable to any vendor at any time; instead, you will pick which vendor you want to work with, and that's what you'll get.

Other parts that remain to be improved on our side are storage interaction, database drivers, function invocation, but it's part of the challenge, and we're excited to tackle it :)

[ramonck]

@ramonck the idea here is to have separate configuration templates for each vendor: AWS, Azure, whatever. When you create your Webiny project, we'll ask you which vendor you want to use, and will create infrastructure configuration specific for that vendor. From Webiny v5 we'll be using https://www.pulumi.com/ to handle all the infrastructure, and that will allow us to support different cloud providers.

So your project will not be deployable to any vendor at any time; instead, you will pick which vendor you want to work with, and that's what you'll get.

Other parts that remain to be improved on our side are storage interaction, database drivers, function invocation, but it's part of the challenge, and we're excited to tackle it :)

@Pavel910 If you get it working with the main vendors that's huge, it's like the alternative to docker to me at least.
Yes it will not work simultaneously but that's is fine because each project you want to focus on a vendor anyways is not like I want to use CosmosDB with AWS Lambda.. not for now at least :).

But thinking about the storage portion why not try to make it like they do with the docker approach concept where the default is the internal SQLite and then you can use some other storage if you actually configure it for the production scenario? Make it optional is that possible? Will that make life easier? The YAML approach to things.

[Pavel910]

@ramonck by storage I meant file storage (sorry for not being clear). As for DBs, we're now working on supporting DynamoDB, so by default, we'll always setup a project for DynamoDB. Mongo and SQL will also be supported, if you need it. But again, if you're on AWS, Dynamo is an obvious choice.

Also, by supporting VPCs (also coming in v5), we're opening the gates for DocumentDB, Aurora, etc. so there's a lot of cool stuff coming soon :)

[doitadrian]

I had the need to respond to this question as well, just didn't have the time.

@ramonck

The closest you'll get to that is on the serverless product but it's focused only on the Functions part.

Actually, to setup Webiny, you "just" need services like API Gateway, CDN, Storage, Functions. And these kinds of services are provided by every major cloud provider, which means we should be able to deploy Webiny there.

Once a user installs Webiny on his provider of choice, it's up to him to pick more complex services, if needed.

As for databases, just to add to @Pavel910's answer, we did experiment with databases a lot, and for V5, we'll be relying on DynamoDb, which, as @Pavel910 mentioned, is a no-brainer, when it comes to AWS. But we are constantly thinking of multi-database support, because, for example, Azure doesn't have DynamoDb. So, we are trying to model the data in a way that will be acceptable for other databases as well. We will hopefully do some experiments and testing with DocumentDb / MySQL in the following weeks, so we'll see how it all works together.

[doitadrian]

BTW, if anybody is interested in contributing - building Azure or GCP Pulumi infrastructure templates, let us know.

We already have 90% done for the AWS, so with that in hand, it might be a bit easier to do the same for Azure or GCP, whatever you prefer.

If someone is interested / wants to try it, let su know, and we'll provide the necessary links / instructions. Note that it will probably take me a couple of days to wrap some loose ends, but it shouldn't take more than that.

[smartboyathome]

BTW, if anybody is interested in contributing - building Azure or GCP Pulumi infrastructure templates, let us know.

We already have 90% done for the AWS, so with that in hand, it might be a bit easier to do the same for Azure or GCP, whatever you prefer.

If someone is interested / wants to try it, let su know, and we'll provide the necessary links / instructions. Note that it will probably take me a couple of days to wrap some loose ends, but it shouldn't take more than that.

I know I am too late on this particular piece, but while I like the pulumi offering, I have hesitations due to self hosting being restricted to their enterprise offering. Hopefully you guys can document the setup process so that it can be replicated on something more friendly to this like Terraform.

[Pavel910]

@smartboyathome what do you mean by self hosting being restricted to their enterprise offering ? Pulumi is open-source and you are deploying to your own cloud account (AWS, Azure). We're doing that for months already, deploying our stacks to our own AWS accounts.

[doitadrian]

https://www.pulumi.com/docs/troubleshooting/faq/#can-i-use-pulumi-without-depending-on-pulumi-com @smartboyathome

[smartboyathome]

I see that my info was out of date. I was referring to the state storage. Apologies.

[ramonck]

BTW, if anybody is interested in contributing - building Azure or GCP Pulumi infrastructure templates, let us know.

We already have 90% done for the AWS, so with that in hand, it might be a bit easier to do the same for Azure or GCP, whatever you prefer.

If someone is interested / wants to try it, let su know, and we'll provide the necessary links / instructions. Note that it will probably take me a couple of days to wrap some loose ends, but it shouldn't take more than that.

@doitadrian I'm still learning about webiny but I understand the main ideas I believe no clue about the code, if you want some coding help just send me what you can and I will be glad to help and publish in whichever branch so we can make it work. If you can tell me were the AWS part is I can start from there that's fine too. Not sure how pulumi works but if it's NodeJS I will figure out.

[doitadrian]

Cool @ramonck, I'll try to polish up a couple of things over the weekend and in the beginning of the next week, and then provide some instructions.

BTW which cloud provider would you be interested in trying to tackle? And which database?

Thanks a lot! 🍻 🍻

[ramonck]

Cool @ramonck, I'll try to polish up a couple of things over the weekend and in the beginning of the next week, and then provide some instructions.

BTW which cloud provider would you be interested in trying to tackle? And which database?

Thanks a lot! 🍻 🍻

Azure and CosmosDB.

[doitadrian]

Just a short update @ramonck, please hold on for a couple of more days (I've been busy with some other stuff). I will start working on preparing the necessary parts for you tomorrow, with full attention to it.

Will provide updates ASAP.

[ramonck]

Ok no worries @doitadrian , would be nice to maybe put it on the README/Wiki so others can also jump in and help as they can and you can have more people helping the library and growing the features around it. The more you share on this piece the more plugins others can help you guys out there in this perspective so maybe we can be an army of a couple and not only us.

[alanreynosov]

I think this link could be useful to know what are the equivalencies of resources required by webiny in GCP and Azure
https://www.datamation.com/cloud-computing/aws-vs-azure-vs-google-cloud-comparison.html

[doitadrian]

Very nice @alanreynosov!

BTW just a short update from my side...

Still doing some refactors and testing. Actually, this is my main focus this week, and will probably be in the next one as well.

More time was needed for abstracting a couple of places and make them multi-cloud compatible. And there is still a couple of places to check.

But it's going well I'd say and we'll probably have something ready next week.

I will update you as soon as possible.

P.S. The branch I'm working is feat/v5/pulumi, so you can see the activity / changes there.

[ramonck]

Hey no problem, I've been working late so no worries no rush. Crazy times.. lots of demand.

[webiny-bot]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[gustavotr]

Any updates? Can I try integrating GCP with pulumi branching from feat/v5/pulumi, @doitadrian?

[doitadrian]

Hey @gustavotr,

sorry for the delay.

We didn't manage to do the thorough multi-cloud support research, that's why this issue hasn't received any updates.

We're extremely busy with wrapping Webiny v5 up, and I'm sure we won't have the time to concentrate on this until the end of the year.

I think it might be just better to pause this until we officially release v5.

That being said - if you still want to play with it on your own, and slooowly work on a solution, feel free to do so, and we can check it out once you have something. The branch from which you can start is the v5 branch.

Who knows, maybe once we finally reach the multi-cloud, we'll be able to use what you've created as an official solution. But just have in mind that we won't be officially doing multi-cloud for now, and we won't be able to focus a lot of our time on it.

Here's 🍺 to having the Webiny v5 out as soon as possible. 🙂

[gustavotr]

🍺