Webiny-js: Serverless deploy to AWS fails on bootstrapped Webiny api project

[akve]

Hi there! I'm planning to replace all the stupid small Wordrpess projects I have to Webiny, but the plan is to use my existing AWS. And I run into troubles trying to setup API on AWS Lambda.

This is an issue

Steps I do:

1. Set up a new project as in documentation
2. Configure AWS credentials for serverless
3. run yarn sls deploy --verbose

The issues I run into are definitely connected to serverless configuration, but I think it would be great if the issues are fixed "from the box".

When deploying "production" version, AWS says:

Unable to import module 'src/handler': Error
at Function.Module._load (module.js:474:25)

When deploying "development" version
NODE_ENV=development yarn sls deploy --verbose
it starts to work but fails in completely different place. Logging in throws the following exception:
UnhandledPromiseRejectionWarning: SyntaxError: Unexpected token W in JSON at position 0 at JSON.parse (<anonymous>) at graphqlHandler (/var/task/src/webpack:/Users/akve/work/webiny-test/node_modules/apollo-server-lambda/dist/lambdaApollo.js:24:1) at Object.default (/var/task/src/webpack:/Users/akve/work/webiny-test/node_modules/apollo-server-lambda/dist/ApolloServer.js:102:14) at Promise (/var/task/src/webpack:/Users/akve/work/webiny-test/node_modules/webiny-api/lambda/lambda.js:120:1) at new Promise (<anonymous>) at /var/task/src/webpack:/Users/akve/work/webiny-test/node_modules/webiny-api/lambda/lambda.js:105:1

Possible Solution

Though a newbie with Serverless, I'm eager to help to put it right.

Any help is appreciated.
Thanks!
Alex

[Pavel910]

Hi @akve ! We are glad you are interested in migrating to Webiny :)

Before jumping to your problem let me just give you a small overview at how we do it for our Cloud platform, just so we are on the same track:

• we do not use serverless framework to deploy anything, so it is not really an out-of-the-box issue, but rather it wasn't even planned to be deployed that way.
• sls is (was) there just for local development, and we have already managed to remove it entirely in our development branches, so the new development pipeline will be released next week, completely without sls (it is more stable and more performant, and since we do not use sls for deploys, we don't need it at all)
• although this is a serverless project, it is not built with serverless framework in mind, as our goal is to attract as many developers as possible to our Cloud to create a big ecosystem with different solutions that will make everything more stable, fast and less time-consuming.
• in our Cloud we use our own custom proxy to handle requests and route them accordingly, and AWS API Gateway is not involved (unlike in SLS framework) - thus the api (or any function for that matter) in the webiny project is just a plain Lambda function.

We are aware that people want to self-host, and we will make it easier in the future, but at this point our priorities are somewhat different, and with a small team it is hard to manage everything.

Onto the problem

I tried creating a project from scratch and deploying it with sls, and it worked without errors. Could you create a reproduction repository on Github? I could take a look at your exact project then.

EDIT: just figured out you meant the error on Lambda when you do an API request (I'll look into it).

Click here to expand

ℹ Your Webiny project is almost ready... 1) Configure a MongoDB database and update connection parameters in packages/api/.env 2) To finish the installation: cd packages/api && yarn setup 3) Run API: cd packages/api && yarn start 4) Run Admin app: cd packages/admin && yarn start 5) Run Site app: cd packages/site && yarn start ✔ That's it! Now you have your API, admin and site apps up and running! Happy coding :) ✨ Done in 108.91s. ~/webiny/js/sls-deploy  cd packages/api ~/webiny/js/sls-deploy/packages/api  yarn setup yarn run v1.15.2 $ node src/install.js Welcome to Webiny installer! The following will be installed: ➜ Webiny Security ➜ Webiny CMS Installing... Installation completed in 0.815s. ✨ Done in 2.03s. $ ~/webiny/js/sls-deploy/packages/api  yarn sls deploy --verbose yarn run v1.15.2 $ /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/.bin/sls deploy --verbose Serverless: Using configuration: { "webpackConfig": "./webpack.config.js", "includeModules": false, "packager": "yarn", "packagerOptions": {} } Serverless: Removing /Users/paveldenisjuk/webiny/js/sls-deploy/packages/api/build Serverless: Bundling with Webpack... Time: 3987ms Built at: 2019-04-26 16:09:16 Asset Size Chunks Chunk Names handler.js 4.84 MiB 0 [emitted] [big] main Entrypoint main [big] = handler.js [37] external "fs" 42 bytes {0} [built] [67] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-plugins/index.js 1.33 KiB {0} [built] [76] external "path" 42 bytes {0} [built] [463] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/dotenv/lib/main.js 2.02 KiB {0} [built] [464] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api/index.js 289 bytes {0} [built] [469] ./src/configs/index.js 160 bytes {0} [built] [472] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api/plugins/index.js 560 bytes {0} [built] [473] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-security/plugins/index.js 518 bytes {0} [built] [474] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-cms/plugins/index.js 782 bytes {0} [built] [475] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-cookie-policy/index.js 4.37 KiB {0} [built] [476] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-google-tag-manager/index.js 2.55 KiB {0} [built] [477] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-mailchimp/index.js 7.04 KiB {0} [built] [478] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/source-map-support/register.js 25 bytes {0} [built] [479] /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/source-map-support/source-map-support.js 17.7 KiB {0} [built] [1109] ./src/handler.js + 1 modules 799 bytes {0} [built] | ./src/handler.js 314 bytes [built] | ./src/plugins.js 430 bytes [built] + 1095 hidden modules WARNING in /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-mailchimp/node_modules/got/source/request-as-event-emitter.js 72:18-25 Critical dependency: require function is used in a way in which dependencies cannot be statically extracted @ /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-mailchimp/node_modules/got/source/as-promise.js @ /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-mailchimp/node_modules/got/source/create.js @ /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-mailchimp/node_modules/got/source/index.js @ /Users/paveldenisjuk/webiny/js/sls-deploy/node_modules/webiny-api-mailchimp/index.js @ ./src/plugins.js @ ./src/handler.js Serverless: Zip service: /Users/paveldenisjuk/webiny/js/sls-deploy/packages/api/build/service [100 ms] Serverless: Packaging service... Serverless: Remove /Users/paveldenisjuk/webiny/js/sls-deploy/packages/api/build Serverless: Creating Stack... Serverless: Checking Stack create progress... CloudFormation - CREATE_IN_PROGRESS - AWS::CloudFormation::Stack - project-name-dev CloudFormation - CREATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket CloudFormation - CREATE_IN_PROGRESS - AWS::S3::Bucket - ServerlessDeploymentBucket CloudFormation - CREATE_COMPLETE - AWS::S3::Bucket - ServerlessDeploymentBucket CloudFormation - CREATE_COMPLETE - AWS::CloudFormation::Stack - project-name-dev Serverless: Stack create finished... Serverless: Uploading CloudFormation file to S3... Serverless: Uploading artifacts... Serverless: Uploading service project-name.zip file to S3 (1.29 MB)... Serverless: Validating template... Serverless: Updating Stack... Serverless: Checking Stack update progress... CloudFormation - UPDATE_IN_PROGRESS - AWS::CloudFormation::Stack - project-name-dev CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - FilesCreateLogGroup CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - ApiLogGroup CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - FilesReadLogGroup CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::RestApi - ApiGatewayRestApi CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - FilesCreateLogGroup CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - ApiLogGroup CloudFormation - CREATE_IN_PROGRESS - AWS::Logs::LogGroup - FilesReadLogGroup CloudFormation - CREATE_IN_PROGRESS - AWS::IAM::Role - IamRoleLambdaExecution CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - FilesCreateLogGroup CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - ApiLogGroup CloudFormation - CREATE_COMPLETE - AWS::Logs::LogGroup - FilesReadLogGroup CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::RestApi - ApiGatewayRestApi CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::RestApi - ApiGatewayRestApi CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFiles CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFiles CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Resource - ApiGatewayResourceFiles CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFunction CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFunction CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Resource - ApiGatewayResourceFunction CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFilesProxyVar CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFilesProxyVar CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFunctionApi CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Resource - ApiGatewayResourceFilesProxyVar CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFunctionApi CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Resource - ApiGatewayResourceFunctionApi CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFunctionApiProxyVar CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Resource - ApiGatewayResourceFunctionApiProxyVar CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Resource - ApiGatewayResourceFunctionApiProxyVar CloudFormation - CREATE_COMPLETE - AWS::IAM::Role - IamRoleLambdaExecution CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - FilesCreateLambdaFunction CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - ApiLambdaFunction CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - FilesReadLambdaFunction CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - ApiLambdaFunction CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - FilesCreateLambdaFunction CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - ApiLambdaFunction CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Function - FilesReadLambdaFunction CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - FilesCreateLambdaFunction CloudFormation - CREATE_COMPLETE - AWS::Lambda::Function - FilesReadLambdaFunction CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFunctionApiAny CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - ApiLambdaVersion7PhLsWTBURj50JP2qt68dGxozbUJ5mYQUhACB8r6n0 CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - ApiLambdaPermissionApiGateway CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - FilesCreateLambdaVersionFFWMyd6p1DnOoYc2rPEhnluJyCtm1pGm95Xd1v2Dmc CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFunctionApiProxyVarAny CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFunctionApiAny CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - FilesCreateLambdaPermissionApiGateway CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - FilesReadLambdaVersionzhRXzxwqIPAUwlsJxudLXC3XAiWjdMyCpJkd7f3pY CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - ApiLambdaPermissionApiGateway CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - FilesReadLambdaPermissionApiGateway CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFilesProxyVarGet CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFunctionApiProxyVarAny CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethodFunctionApiAny CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - FilesCreateLambdaPermissionApiGateway CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - ApiLambdaVersion7PhLsWTBURj50JP2qt68dGxozbUJ5mYQUhACB8r6n0 CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - ApiLambdaVersion7PhLsWTBURj50JP2qt68dGxozbUJ5mYQUhACB8r6n0 CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - FilesCreateLambdaVersionFFWMyd6p1DnOoYc2rPEhnluJyCtm1pGm95Xd1v2Dmc CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethodFunctionApiProxyVarAny CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFilesProxyVarGet CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Permission - FilesReadLambdaPermissionApiGateway CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - FilesCreateLambdaVersionFFWMyd6p1DnOoYc2rPEhnluJyCtm1pGm95Xd1v2Dmc CloudFormation - CREATE_IN_PROGRESS - AWS::Lambda::Version - FilesReadLambdaVersionzhRXzxwqIPAUwlsJxudLXC3XAiWjdMyCpJkd7f3pY CloudFormation - CREATE_COMPLETE - AWS::Lambda::Version - FilesReadLambdaVersionzhRXzxwqIPAUwlsJxudLXC3XAiWjdMyCpJkd7f3pY CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethodFilesProxyVarGet CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFilesPost CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Method - ApiGatewayMethodFilesPost CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Method - ApiGatewayMethodFilesPost CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1556287750846 CloudFormation - CREATE_IN_PROGRESS - AWS::ApiGateway::Deployment - ApiGatewayDeployment1556287750846 CloudFormation - CREATE_COMPLETE - AWS::ApiGateway::Deployment - ApiGatewayDeployment1556287750846 CloudFormation - CREATE_COMPLETE - AWS::Lambda::Permission - ApiLambdaPermissionApiGateway CloudFormation - CREATE_COMPLETE - AWS::Lambda::Permission - FilesCreateLambdaPermissionApiGateway CloudFormation - CREATE_COMPLETE - AWS::Lambda::Permission - FilesReadLambdaPermissionApiGateway CloudFormation - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - project-name-dev CloudFormation - UPDATE_COMPLETE - AWS::CloudFormation::Stack - project-name-dev Serverless: Stack update finished... Serverless: Cannot read property 'id' of undefined Service Information service: project-name stage: dev region: us-east-2 stack: project-name-dev resources: 25 api keys: None endpoints: ANY - https://dp5lozzcu6.execute-api.us-east-2.amazonaws.com/dev/function/api ANY - https://dp5lozzcu6.execute-api.us-east-2.amazonaws.com/dev/function/api/{proxy+} POST - https://dp5lozzcu6.execute-api.us-east-2.amazonaws.com/dev/files GET - https://dp5lozzcu6.execute-api.us-east-2.amazonaws.com/dev/files/{proxy+} functions: api: api filesCreate: files-create filesRead: files-read layers: None

This, however, doesn't mean it works when you send a request to the API.

Database connection

In the production config file (packages/api/src/configs/production.js:8) you will find we use context to pass database connection to the application: this is made specifically for the Cloud environment. Since you want to host the API yourself - you need to configure this similar to the development config: packages/api/src/configs/development.js:17 and instantiate the Mongo connection yourself (just follow the referenced development config).

Client apps

admin and site apps also need to be configured for the new API endpoint, see production configs to enter your API endpoint (once it is deployed to AWS):
packages/admin/src/config/production.js:20 and packages/site/src/config/production.js:16

API Gateway

Since now you are in the realm of the API Gateway, I can't provide you with any info myself, because I never tried to deploy it up there. However, if you run into more problems, feel free to send more questions our way, and hopefully we'll figure out how to host it using SLS along the way :)

Coming really soon...

In the upcoming weeks we will release a major upgrade to the webiny-cli, apps structure, etc. We are migrating to react-router to fully support server-side-rendering so I think it is best for you to stick to the api part of the project for now, and continue with the rest once it is all released. We are working hard to make it all great on all fronts, users, developers, local, cloud, so please bear with us. Great stuff is on the way :)

Let us know how it goes, and feel free to come back with questions, we're always glad to help!
Cheers 🍺

[Pavel910]

@akve I investigated the problem and managed to tweak the project to get it running on lambda deployed via SLS. The problems you are encountering are related to 1) SLS config for build, and 2) API Gateway and the way it sends event data to Lambda.

That said, I will write a proper documentation article or just summarize steps taken here in this issue, I just need to play with it a little bit more, so please stay patient. I will try to get it done in the following days. I can't promise the exact day, but I will get it done and will notify you.

Thanks for reporting this issue, much appreciated. It will not be a fix since SLS is not our primary concern, but it will be a guide to get you going.

[akve]

Hi @Pavel910!

Thanks for detailed review of the context and direction Webiny is heading! Now a lot more clear, I was pretty sure from the code that Serverless is fully utilized.

I've configured everything fine, and got to the point where the API is rolled out on AWS - but it throws 502 error on any request (same as yours from the first attempt, I suppose). Also, for testing purposes, used MongoLab db - and locally this works fine.

I see that your test endpoint is now working, so you've managed to conquer it! ) Looking forward to your article on this.

I think it's really nice moment for a serverless CMS made for both geeks and normal people. So I'm glad to propose (and help) with other ways of deploying Webiny, cause I enjoyed it )

[Pavel910]

@akve sorry for the delay, we were very busy with the latest release (if you haven't already, check our blog for the details). We have greatly improved the experience for developers, our CLI now handles everything, from development to deploy and there is no serverless framework anymore. I kindly ask you to give it a try and let us know what you think (just install the new CLI and create a project from scratch).

Back on topic

I managed to run our API using the sls and Mongo Atlas database. Turned out to be not that difficult. But - that is just the GraphQL portion. Our webiny-cli is also handling file uploads (for local development) and once you deploy your apps/api to the cloud, our Cloud is handling the file uploads for you.

So - the logical next problem with sls is - how are you going to handle file uploads if you go the self-hosting way ? You could either replace the image plugin on the client and create one that uses Cloudinary for example, or you could try to recreate the files service on your own (the service consists of 3 lambda functions that do reading and upload to S3, with resizing, etc.).

I would like to hear your ideas on this. Cheers!

[akve]

Offtopic:

I personally think the solution with the proprietary platform is not viable (just my own opinion, you and your investors may think the opposite). The power of Wordpress is primarily in easy setup on own server. So the AWS/Google + server-independent frontend, without any hidden third-party services (and your Webiny Cloud Hosting seems so far completely not reliable and additional headache+costs) is best choice. This might be OK for the cash flow, but I think it's better to get money from custom development, and focus on growing the user base.

Could not find a quick way now to deploy with new CLI outside Webiny Cloud, will take a look a bit later on :)

Thanks!
Alex

[Pavel910]

@akve thanks for your opinion! And yes, we are not aiming at deploying to custom environments using our CLI at this point. Same as Netlify - their CLI is for use with Netlify, not an infinite amount of custom environments.

I am afraid, at this point, all you can do is play with it locally and try to figure out how it would work in a custom environment on your own. Because the apps themselves are simple CRA (for frontend) and an API powered by Apollo Server. So on that level it's all pretty simple. The only problem, as I mentioned in the previous post, is image handling. That's the biggest problem to easy-to-deploy self-hosting at this point.

Hopefully you'll keep an eye on Webiny, maybe for future use :)

Thanks for the discussion and your time,
Cheers! 🍺