Bref: Feature request: Make it possible to enable the php-amqp extension

[Nyholm]

Building an API that takes some HTTP parameters and put it on a queue seams to be a perfect scenario for using a Lambda function.

[mnapoli]

Thanks, just to be sure we are clear: php-amqp is for RabbitMQ and similar message queues right?

In that case is there a reason not to use SQS? That choice would also allow to write the queue workers as Lambda (trigger by the SQS events).

[Nyholm]

Yes, that is correct.

SQS is defiantly a replacement RabbitMQ. I currently have all my architecture built around RabbitMQ, I will not simply replace it with SQS. =)

Other possible solutions:

• Use a library instead of the AMQP extension
• Use a custom extension: https://bref.sh/docs/environment/php.html#custom-extensions

[mnapoli]

👍 makes sense. This might not be an extension we want to ship by default though (as it would probably be needed but a small portion of projects) but I'd love for Bref to be able to offer it via a separate (additional) layer.

[Nyholm]

I've tried to understand how to include a new layer. I also saw #351 but Im not sure how I would install a PHP extension with serverless either..

I've read and understood the tutorial at: https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html

What I dont understand is how layer interact with each other. Do each layer need a bootstrap file? Where is PHP? Can I just use a pecl installation? What if I want to install something like NewRelic. (Installation instructions)

[nealio82]

There seem to be 3 distinct cases and associated methods for creating layers:

1) If you want to override the way PHP is loaded (eg, if you wanted to use ReactPHP instead of php-fpm)

• Just zip up and provide your own bootstrap file in its own layer, then include it in your template.yml alongside Bref's PHP layer

2) If you want to add an extension which _could_ be installed with PECL (eg AMQP, Swoole, etc)

• You will need to compile the extension against the Amazon Linux AMI (and, I think, add it to php/init.d/php.ini in your project). Publish your layer, and include it in your template.yml as well as Bref's PHP layer

3) If you want to include an extension which needs to be compiled into PHP (eg GD)

• You can clone brefphp/bref, modify php.Dockerfile and run make publish. Replace Bref's PHP layer with the one you just created. You'll then have to maintain your entire PHP layer yourself :(

You can use up to 5 different layers in a Lambda function. So you _could_ have Bref's PHP, your own AMQP layer, plus your own New Relic layer all at once

[Nyholm]

Thank you. I did not find the runtimes before.

Now it make more sense. I will modify the runtime to make sure it fits me. Thank you

[mnapoli]

@Nyholm just to be sure: are you trying to write a AMQP worker in Lambda? If that's the case I'm not sure it makes sense because a lambda is not supposed to run continuously (e.g. to poll RabbitMQ).

However if you just want to post messages into a queue then there is no problem.

[Nyholm]

I will both produce messages and consume them. I guess I will have to migrate to SQS slowly. For these new applications I’m building I’ll let them communicate over SQS instead of rabbitmq.

I know I can set up lambda to be triggered with SQS, but how would that work with bref? Is that just a console command that get the SQS message on stdin?

[mnapoli]

We don't have to deal with workers and console commands, SQS sends native events to Lambda. Here is a simple JS example: https://github.com/becloudway/aws-lambda-sqs-sam

You can see here how to configure it: https://github.com/becloudway/aws-lambda-sqs-sam/blob/1a204ab4a330f81891bec2659078e2b80ed4988a/lambda-sqs-sam-template.yml#L28-L33

And then you use a PHP function to handle the events:

<?php

require __DIR__.'/vendor/autoload.php';

lambda(function (array $event) {
    return /* response */;
});

Here is an example of what the $event variable will contain: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

[nealio82]

SNS makes more sense to use with Lambda, however, than SQS. With SQS you'll either still have to long-poll the queue, or mess about with configuring triggers to invoke your command when there's something in the queue (in my experience this can be a bit flaky as well). SNS is a pub/sub model which just calls your Lambda when an event happened, and is much more scalable (it effectively does it itself). @eddmann & I set up a system yesterday from scratch in well under 3 hours. We'll follow up with a blog post, but our system looks something like this:

• client calls an API endpoint which is a simple PHP script on Bref with routing its defined in template.yml
• API endpoint validates the data, calls SnsClient::publish()
• Another script defined as a simple 'function' in template.yml is subscribed to the SNS topic, receives the event, and does its job (in our case does some file processing and uploads the result to a pre-signed S3 bucket)
• The entire system (IAM roles, SNS topic, subscription, etc) is defined in template.yml

[deleugpn]

I would disagree with the statement that SNS makes more sense with Lambda. Both SNS and SQS provide the exact same developer experience with Lambda: either SNS invokes your lambda or SQS does, you don't have to deal with any polling whatsoever as Amazon will do it for you.
Both options also allow for a Dead Letter Queue configuration: either on the original Queue itself or in the Lambda, so that failed payloads gets stored for further analyses.
The deciding factor is much more semantic than technical: pub/sub vs asynchronous request/response. SNS Allows multiple subscribers while SQS doesn't.

[mnapoli]

Agreed, @nealio you don’t have to poll SQS

Jeu 20 juin 2019, à 10:02, Marco Aurélio Deleu a écrit :

I would disagree with the statement that SNS makes more sense with Lambda. Both SNS and SQS provide the exact same developer experience with Lambda: either SNS invokes your lambda or SQS does, you don't have to deal with any polling whatsoever as Amazon will do it for you.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/brefphp/bref/issues/345?email_source=notifications&email_token=AAFP3SCQ2ROO5PFKUYHEXXDP3M2QNA5CNFSM4HXI2E2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYETYLY#issuecomment-503921711, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFP3SBSSIUIXFAMYINK2C3P3M2QNANCNFSM4HXI2E2A.

[nealio82]

How do you get SQS to trigger your Lambda event in that case? Last time I did it I had to configure triggers (if there's more than 0 things in this queue, trigger this Lambda. Scale if more than [x] items...), which was an absolute pain in the arse and pretty flaky (sometimes the lambda wouldn't get triggered at all and things were left in the queue)

[deleugpn]

This is how I did it: https://github.com/deleugpn/laravel-lambda/blob/master/serverless.yaml#L67-L72

[mnapoli]

@nealio82 also see my comment above yours: https://github.com/brefphp/bref/issues/345#issuecomment-503882510

[nealio82]

Well, in that case, SQS _might_ be a better fit for our system than I'd realised :D

[Nyholm]

Okey. I will close this issue now.
Thank you for all your input. I ended up forking the repo to install amqp and newrelic in my custom runtime.

I also implemented SNS.

I added some blog posts about it:

• http://developer.happyr.com/symfony-messenger-on-aws-lambda
• http://developer.happyr.com/newrelic-on-bref-aws-lambda

[tristanbes]

@Nyholm: Thanks for your articles, I've got a question though:

We’ve decided to use SNS because pub-sub [...]. We don’t have the issue that “the server might get too busy";

What if your lambda calls an API and you don't want to spam your API with too many concurrent requests, would you recommand to use Amazon SQS instead of SNS based on this scenario ?

[Nyholm]

Excellent point. Yeah, that would make more sense.

[samijnih]

Hi :wave:

I'm facing the same issue because I'm using "bref/php-74-fpm-dev" for my local development but I need php-amqp extension to be installed in order to run my project.

What is the proper way to handle this case with the php-74-fpm-dev image?

We don't want to use SQS or another queueing system on our production. We dispatch messages to an amqp system managed by another project in the company which is not purpose to AWS :/

Sami

[Nyholm]

Hey.

Have a look at this repository: https://github.com/brefphp/extra-php-extensions

Happy coding!

[samijnih]

Hey.

Have a look at this repository: https://github.com/brefphp/extra-php-extensions

Happy coding!

Hey :)

Thanks for the answer.

I've already managed to run my project from aws lambda using serverless deploy with that AMQP extra layer.

My problem is only for local development and test. I have a symfony HTTP app and the extra-php-extensions is only for serverless.

For my development, I can't use serverless like this serverless invoke local -f {my_function} --data {which_data?} because as I said it is a HTTP web app (BTW may be a way to hit my http endpoint with serverless invoke?)

And I don't want to make a symfony server:start because it's "okay" to make developers install symfony CLI but not on our CI

[Nyholm]

There is a docker image for amqp: https://hub.docker.com/r/bref/extra-amqp-php-74

Im honestly not 100% sure how you would use it to include it locally. I know there are some people running it with docker locally. I suggest asking on the slack channel. https://bref.sh/docs/community.html