Bref: Cannot read body content of POST request

[bertrandjamin]

v0.3

I try to retrieve the body content of a POST request with an HTTP application configuration

I use this simple handler :

# index.php
<?php 

echo file_get_contents('php://input');

---

When testing my lambda via Api Gateway everything seems to work fine (when I manually specify the content-length) :

But with a « real » HTTP call I cannot read any content from the body of my request...

---

this issue come from this comment : https://github.com/mnapoli/bref/issues/116#issuecomment-453745811

[bertrandjamin]

@mnapoli FYI I have activated detailed logs to retrieve more information.

We can see that body is emptied during the "conversions" process.

Unfortunately Gateway logs are truncated before being sent to cloudwatch (https://forums.aws.amazon.com/thread.jspa?threadID=221740) and I think that we lost important information during the step : Endpoint request body after transformations

Maybe you'll find something interesting...

(914bad5e-1840-11e9-83f1-efc9a0398ffc) Extended Request Id: TgwN2EJqCYcFnFQ=
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Verifying Usage Plan for request: 914bad5e-1840-11e9-83f1-efc9a0398ffc. API Key: API Stage: m2t7qi7916/Prod
(914bad5e-1840-11e9-83f1-efc9a0398ffc) API Key authorized because method 'ANY /' does not require API Key. Request will not contribute to throttle or quota limits
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Usage Plan check succeeded for API Key and API Stage m2t7qi7916/Prod
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Starting execution for request: 914bad5e-1840-11e9-83f1-efc9a0398ffc
(914bad5e-1840-11e9-83f1-efc9a0398ffc) HTTP Method: POST, Resource Path: /
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Method request path:
{}
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Method request query string:
{}
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Method request headers: {Accept=*/*, CloudFront-Viewer-Country=FR, CloudFront-Forwarded-Proto=https, CloudFront-Is-Tablet-Viewer=false, CloudFront-Is-Mobile-Viewer=false, User-Agent=insomnia/6.3.2, X-Forwarded-Proto=https, CloudFront-Is-SmartTV-Viewer=false, Host=m2t7qi7916.execute-api.us-east-2.amazonaws.com, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-5c3cfa58-7ebfb0ec7e431584f741130e, Via=1.1 17fbe2e6aa5682781b6ee23bfab1fd28.cloudfront.net (CloudFront), X-Amz-Cf-Id=C3SfmopdZJiqADduwSdrD78zLfFxbb5eom3kW3xcZg6d3qpk-PXNeQ==, X-Forwarded-For=82.241.228.91, 54.239.156.98, CloudFront-Is-Desktop-Viewer=true}
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Method request body before transformations: coucou=bertrand
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Endpoint request URI: https://lambda.us-east-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-2:894492105031:function:gitlab-webhook/invocations
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Endpoint request headers: {x-amzn-lambda-integration-tag=914bad5e-1840-11e9-83f1-efc9a0398ffc, Authorization=************************************************************************************************************************************************************************************************************************************************************************************************************************1013cf, X-Amz-Date=20190114T210840Z, x-amzn-apigateway-api-id=m2t7qi7916, X-Amz-Source-Arn=arn:aws:execute-api:us-east-2:894492105031:m2t7qi7916/Prod/ANY/, Accept=application/json, User-Agent=AmazonAPIGateway_m2t7qi7916, X-Amz-Security-Token=FQoGZXIvYXdzELb//////////wEaDOmfCIQpl4jVT/tz7yK3A3/hRqJEukqkfs9qCqjohnF5X6OM5YvFHAmJCsWVtDNAniIu5Y+/pf6d9OcEug6DUOx7p28PYiG5k4/Iz/Nj6fySHfGOCG7pSAnSSdyTm9yJYabd4LnIFPXu7/5yK65j/I6Sa9IYD8DbglijvOFG5/E0HGUmMp0aA79dzjaCQ5a0XMFJg88u10afeqt/pWIcq512RvuX0Ll+qlcauJqLK91XaigMez5MseLRMf/BhMu1i1AHhYCqR7mCld06vNQNjLhfZ2dFgQicdGuAUTWvxGTaG3wiLdb80H9depM+gsvQfIX3QoxBbWCjj [TRUNCATED]
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Endpoint request body after transformations: {"resource":"/","path":"/","httpMethod":"POST","headers":{"Accept":"*/*","CloudFront-Forwarded-Proto":"https","CloudFront-Is-Desktop-Viewer":"true","CloudFront-Is-Mobile-Viewer":"false","CloudFront-Is-SmartTV-Viewer":"false","CloudFront-Is-Tablet-Viewer":"false","CloudFront-Viewer-Country":"FR","Host":"m2t7qi7916.execute-api.us-east-2.amazonaws.com","User-Agent":"insomnia/6.3.2","Via":"1.1 17fbe2e6aa5682781b6ee23bfab1fd28.cloudfront.net (CloudFront)","X-Amz-Cf-Id":"C3SfmopdZJiqADduwSdrD78zLfFxbb5eom3kW3xcZg6d3qpk-PXNeQ==","X-Amzn-Trace-Id":"Root=1-5c3cfa58-7ebfb0ec7e431584f741130e","X-Forwarded-For":"82.241.228.91, 54.239.156.98","X-Forwarded-Port":"443","X-Forwarded-Proto":"https"},"multiValueHeaders":{"Accept":["*/*"],"CloudFront-Forwarded-Proto":["https"],"CloudFront-Is-Desktop-Viewer":["true"],"CloudFront-Is-Mobile-Viewer":["false"],"CloudFront-Is-SmartTV-Viewer":["false"],"CloudFront-Is-Tablet-Viewer":["false"],"CloudFront-Viewer-Country":["FR"],"Host":["m2t7q [TRUNCATED]
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Sending request to https://lambda.us-east-2.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-2:894492105031:function:gitlab-webhook/invocations
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Received response. Integration latency: 33 ms
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Endpoint response body before transformations:
{
    "isBase64Encoded": false,
    "statusCode": 200,
    "headers": {
        "x-powered-by": "PHP/7.2.12",
        "content-type": "text/html; charset=UTF-8",
        "status": "200 Ok"
    },
    "body": ""
}
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Endpoint response headers: {Date=Mon, 14 Jan 2019 21:08:40 GMT, Content-Type=application/json, Content-Length=154, Connection=keep-alive, x-amzn-RequestId=914c2240-1840-11e9-afcc-d3d312e75ea6, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-5c3cfa58-7ebfb0ec7e431584f741130e;sampled=0}
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Method response body after transformations:
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Method response headers: {x-powered-by=PHP/7.2.12, content-type=text/html; charset=UTF-8, status=200 Ok, X-Amzn-Trace-Id=Root=1-5c3cfa58-7ebfb0ec7e431584f741130e;Sampled=0}
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Successfully completed execution
(914bad5e-1840-11e9-83f1-efc9a0398ffc) Method completed with status: 200
(914bad5e-1840-11e9-83f1-efc9a0398ffc) AWS Integration Endpoint RequestId : 914c2240-1840-11e9-afcc-d3d312e75ea6
(914bad5e-1840-11e9-83f1-efc9a0398ffc) X-ray Tracing ID : 1-5c3cfa58-7ebfb0ec7e431584f741130e

[mnapoli]

Thanks I'll look into that.

Do you know if you get the same behavior when running locally? (using sam local start-api)

The goal will be to add tests that run with sam local, hopefully I can reproduce and cover that case.

[bertrandjamin]

@mnapoli the request works fine with in a local environment... 🤔

If I understand well the process I suppose that the problem occurs here :

// \Bref\Runtime\LambdaRuntime::waitNextInvocation

   $handler = curl_init("http://{$this->apiUrl}/2018-06-01/runtime/invocation/next");
        curl_setopt($handler, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($handler, CURLOPT_FAILONERROR, true);

        // Retrieve invocation ID
        $invocationId = '';
        curl_setopt($handler, CURLOPT_HEADERFUNCTION, function ($ch, $header) use (&$invocationId) {
            if (! preg_match('/:\s*/', $header)) {
                return strlen($header);
            }
            [$name, $value] = preg_split('/:\s*/', $header, 2);
            if (strtolower($name) === 'lambda-runtime-aws-request-id') {
                $invocationId = trim($value);
            }

            return strlen($header);
        });

        // Retrieve body
        $body = '';
        curl_setopt($handler, CURLOPT_WRITEFUNCTION, function ($ch, $chunk) use (&$body) {
            $body .= $chunk;

            return strlen($chunk);
        });
        ...
        $event = json_decode($body, true);
        ... 

I have made several assumption

• The body is already missing in the received event ?

  • Cannot valid this because of the truncated log :/

• The CURLOPT_WRITEFUNCTION doesn't retrieve correctly the body of http://{$this->apiUrl}/2018-06-01/runtime/invocation/next ?

  • Should the "$body" variable contains headers entries ( as CURLOPT_HEADER is not explicitly set at 0) ?
  • Should not we use mb_strlen rather than strlen in case of UTF-8 chars ?

* The $body is not a valid json object ?

It could really help if we could know the content of the request http://{$this->apiUrl}/2018-06-01/runtime/invocation/next and/or of the $body variable.

Maybe we should create a debug layer with additional log entries to trace more efficiently all the process ?

[bertrandjamin]

But I haven't realized that the LambdaRuntime is not in the layer but in my lambda function.

I will make several tests around my assumptions !

[mnapoli]

All that information is super helpful! FYI you can replace the bootstrap of the layer by creating a bootstrap file at the root of your project (and add debug stuff, e.g. by echoing stuff to stdout in the bootstrap).

Here is Bref's bootstrap for PHP-FPM: https://github.com/mnapoli/bref/blob/0.3/runtime/php/layers/fpm/bootstrap

[bertrandjamin]

Ok so after several checks I can confirm that the body is correctly sent by Aws Lambda and correctly handled all along the « Bref » process.
I lost it between the phpfpm call (via Hoa) and the ‘php://input’ in my script.

So I conclude that the issue occurs on the php side...

Does you’re self compiled php could break something or is missing a configuration ? Does we need to force directives setting in the php.ini (ex: post_max_size) ?

[julienmortuaire]

Hello,

I came across the same issue, and could resolved it by forcing content length header on this file : https://github.com/mnapoli/bref/blob/0.3/src/Runtime/PhpFpm.php#L181-L183

if (isset($headers['content-length'])) {
    $requestHeaders['CONTENT_LENGTH'] = $headers['content-length'];
}else if(!empty($requestBody)){
    $requestHeaders['CONTENT_LENGTH'] = mb_strlen($requestBody);
}

PHP binary seems to ignore POST content if content-length header is not set, and this header is forced with this little piece of code. The tricky thing is : why content-length is not forwarded to lambda function ? I suspect a misconfiguration on API gateway stack which block forwarding this header.

[mnapoli]

That sounds reasonable to add the missing Content-Length header automatically. I tried a simple PHP script with php -S 127.0.0.1:8000 and this request:

curl -d '{}' http://127.0.0.1:8000/request.php

I see that the CONTENT_LENGTH is set to 2 automatically.

What you suggest @julienmortuaire seems correct!

[mnapoli]

I've opened #178, @bertrandjamin could you try it out? (you need to require the dev-fix-162 version in your composer.json)

[bertrandjamin]

@mnapoli yes it works perfectly 🎉

Good catch @julienmortuaire 👍

why content-length is not forwarded to lambda function ? I suspect a misconfiguration on API gateway stack which block forwarding this header.

This mistery still annoy me but let's move on ;)

[bertrandjamin]

oh wait @mnapoli it seems to have an issue with multi-bytes chars.
I cannot test it right now but I will take a look this evening...

[mnapoli]

@bertrandjamin See the last commit I pushed in #178

I was using mb_strlen but it was incorrect (see https://mark.koli.ch/remember-kids-an-http-content-length-is-the-number-of-bytes-not-the-number-of-characters). I added a new test case to cover this.

We are slowly covering more and more advanced test cases, sorry you have to be the guinea pig ^^

[bertrandjamin]

@mnapoli perfect everything seems to work fine !
No worries I'm happy to help a little ;)

[ghost]

@mnapoli I ran into this issue earlier, but with a PATCH request (we're using straight application/json, so nothing fancy) and would've gotten the same issues with PUT, which makes it pretty hard to work on RESTful APIs on Lambda using Bref.

Could we apply the same for PUT and PATCH as well, not just for POST?

I'd be happy to contribute a patch - the simplest might be to invert the HTTP method check to !== 'GET', but if there's any argument against it I'd prefer to override our bootstrap file and pass the allowed methods with auto-content-length to Bref\Runtime\PhpFpm's Constructor.

[mnapoli]

@timo-schaefer-sm that sounds good!

the simplest might be to invert the HTTP method check to !== 'GET'

I like "simple" :p

Have a look at 4f781eb7b5cf832ef4564ee261f013ecec4b1a07 to see where to add the tests.

Thanks!

[Sh1d0w]

@mnapoli This problem still exists for PUT and PATCH requests.

The body of PUT and PATCH requests I am sending to my bref application is empty, even if I attempt to set the content length header in the request.

[mnapoli]

@Sh1d0w right, that is very probably what @timo-schaefer-sm reported above

[Sh1d0w]

@mnapoli Just wanted to make sure you are aware, as this issue was closed, but the problem still exists. I did not wanted to open a new issue.

Will you be able to implement the proposed fix @timo-schaefer-sm suggested?

[mnapoli]

@Sh1d0w that's now fixed thanks to @timo-schaefer-sm

[Sh1d0w]

@mnapoli Works like a charm. Thank you and also @timo-schaefer-sm