Pact-js: Access and modify req.body in request filters

[YOU54F]

What am I trying to do?

Trying to verify pacts against an AWS Lambda function, with an API gateway, that requires AWSv4 signed request headers

Problems I need to overcome

Signing Requests
To sign a request, you first calculate a hash (digest) of the request. Then you use the hash value, some other information from the request, and your secret access key to calculate another hash known as the signature. Then you add the signature to the request in one of the following ways:

Using the HTTP Authorization header.

Basically we need to pre-sign some headers based on information on our request

• For GET requests, Request query string is needed
• For POST requests, Request body is needed

We can acheive this with aws4 in a stateHandler for isAuthenticated, however I cannot get access to the pact under test's path or requestBody, which I need to pass to the pre-signed URL

What I've tried so far

Using the following Proposal: Allow customProviderHeaders to be dynamically added to different interactions , I have been able to get the verifier successfully working locally & in CI with a hardcoded request path, rather than the request path from the pact under test.

Examples in CI

CircleCI AWS Verification Step
AWS-Provider Pact
AWS-Provider Pact Verification Results

Steps taken in code

1. Generate temporary AWS credentials with a bash script and export to bash & run verify script

#!/bin/bash
set -o pipefail

  AWS_TEMP_CREDS=`aws sts assume-role --role-arn $ARN_ROLE --role-session-name api-gateway-access| jq -c '.Credentials'`
  export AWS_ACCESS_KEY_ID=`echo $AWS_TEMP_CREDS | jq -r '.AccessKeyId'`
  export AWS_SECRET_ACCESS_KEY=`echo $AWS_TEMP_CREDS | jq -r '.SecretAccessKey'`
  export AWS_SESSION_TOKEN=`echo $AWS_TEMP_CREDS | jq -r '.SessionToken'`

npx ts-node src/pact/verifier/verify.ts | grep -v Created

1. Pact is read, and state 'is authenticated' is met, passes over to the stateHandler.
   
   
   
   • Request host, path and body need to be ascertained
   
   
   • Request host comes from PROVIDER_BASE_URL which is set to https://3efkw1ju81.execute-api.us-east-2.amazonaws.com/default
   
   
   • For GET requests, Request path needs to come from pact under test, currently hardcoded to default/helloworld
   
   
   • For POST requests, Request body needs to come from pact under test
   
   
2. stateHandler for is Authenticated returns modified headers
   ```
   let signedHost: string;
   let signedXAmzSecurityToken: string;
   let signedXAmzDate: string;
   let signedAuthorization: string;
   let authHeaders: any;

const opts: VerifierOptions = {
stateHandlers: {
"Is authenticated": async () => {
const requestUrl = process.env.PACT_PROVIDER_URL;
const host = new url.URL(requestUrl).host;
const apiroute = new url.URL(requestUrl).pathname;
const pathname = ${apiroute}/helloworld;
const options = {
host,
path: pathname,
headers: {}
};
await aws4.sign(options);
aws4.sign(options, {
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
accessKeyId: process.env.AWS_ACCESS_KEY_ID,
sessionToken: process.env.AWS_SESSION_TOKEN
});
authHeaders = options.headers;
signedHost = authHeaders.Host;
signedXAmzSecurityToken = authHeaders["X-Amz-Security-Token"];
signedXAmzDate = authHeaders["X-Amz-Date"];
signedAuthorization = authHeaders.Authorization;
return Promise.resolve(AWS signed headers created);
},
"Is not authenticated": async () => {
signedHost = null;
signedXAmzSecurityToken = null;
signedXAmzDate = null;
signedAuthorization = null;
return Promise.resolve(Blank aws headers created);
}
},

5. requestFilter will set amazon signed headers if they have been set

requestFilter: (req, res, next) => {
// over-riding request headers with AWS credentials
if (signedHost != null) {
req.headers.Host = signedHost;
}
if (signedXAmzSecurityToken != null) {
req.headers["X-Amz-Security-Token"] = signedXAmzSecurityToken;
}
if (signedXAmzDate != null) {
req.headers["X-Amz-Date"] = signedXAmzDate;
}
if (signedAuthorization != null) {
req.headers.Authorization = signedAuthorization;
}
next();
},

## How can I get access to the path and body of the pact under test, in the stateHandler?

I logged out the `req.path` & `req.body` inside `requestFilter`

req.path /_pactSetup
req.body { consumer: 'consumer-service',
state: 'Is authenticated',
states: [ 'Is authenticated' ],
params: {} }
creating AWS signed headers
created AWS signed headers
req.path /path/that/the/pact/test/is/calling
req.body undefined

It looks like

1. Pact setup is called `req.path /_pactSetup` with the body 

{ consumer: 'consumer-service',
state: 'Is authenticated',
states: [ 'Is authenticated' ],
params: {} }

2. The `stateHandler` is called

creating AWS signed headers
created AWS signed headers

3. The pact under tests, request path is called

req.path /helloworld
req.body undefined

4. For a post request, it might look like

req.path /helloworld
req.body {message:"hello world")
```

So my real question is, can the stateHandlers access the req object?

Cheers for any advice and help in advance!

[vgrigoruk]

I guess this will require changes in pact-ruby & pact-provider-verifier

[bernardobridge]

Actually, @YOU54F, a similar use case to yours was what originated me asking about being able to modify them dynamically (I was also working with Lambda behind an API Gateway with AWSv4 signing). I (wrongly) assumed that the state handlers they implemented had access to the request object, but it appears not. It means you can test the happy path with what they've implemented, but not the negative scenario!

I can share with you the workaround I had at the time, which would still work for you now! I wrote a blog post about it here: https://medium.com/dazn-tech/pact-contract-testing-dealing-with-authentication-on-the-provider-51fd46fdaa78 . Hopefully that helps!

But TLDR: instead of pulling the pacts directly from the broker, we point it to a local file instead. in a before hook to the pact verification, we pull the pact ourselves from the pact broker into a file, and then use a function to parse through it and add the request headers based on the request information. Since you'll have access to the provider states there too, you should be able to add custom logic based on that too (i.e. if "Is authenticated" -> add the header; otherwise, don't)

[YOU54F]

I've got this working nicely inside the requestFilter with access to the path for get requests, but I can't get access to the body of the pact under test for POST requests.

I'm not too fussed about the unhappy paths at the moment.

here is my req filter that works for GET requests

    requestFilter:  (req, res, next) => {
      const requestUrl =  PACT_PROVIDER_URL;
      const host =  new url.URL(requestUrl).host;
      const options = {
        host,
        path: '/Test' + req.path
        headers: {}
      };
       aws4.sign(options, {
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        sessionToken: process.env.AWS_SESSION_TOKEN
      });
      authHeaders =  options.headers;
      req.headers["Host"] =  authHeaders["Host"];
      req.headers["X-Amz-Security-Token"] =  authHeaders["X-Amz-Security-Token"];
      req.headers["X-Amz-Date"] =  authHeaders["X-Amz-Date"];
      req.headers["Authorization"] = authHeaders["Authorization"];
      next();
    },

For POST requests, I need to something like

    requestFilter:  (req, res, next) => {
      const requestUrl =  PACT_PROVIDER_URL;
      const host =  new url.URL(requestUrl).host;
      const options = {
        host,
        path: '/Test' + req.path,
        body: JSON.stringify(req.body),
        headers: {}
      };
       aws4.sign(options, {
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
        accessKeyId: process.env.AWS_ACCESS_KEY_ID,
        sessionToken: process.env.AWS_SESSION_TOKEN
      });
      authHeaders =  options.headers;
      req.headers["Host"] =  authHeaders["Host"];
      req.headers["X-Amz-Security-Token"] =  authHeaders["X-Amz-Security-Token"];
      req.headers["X-Amz-Date"] =  authHeaders["X-Amz-Date"];
      req.headers["Authorization"] = authHeaders["Authorization"];
      next();
    },

but req.body is empty.

Reading up on Stack Overflow, as of Express v4.16, (https://stackoverflow.com/questions/11625519/how-to-access-the-request-body-when-posting-using-node-js-and-express#11631664)

we can use app.use(express.json())

which when placed above this line - https://github.com/pact-foundation/pact-js/blob/9d118a82ae2232448093589a98124491760a7f96/src/dsl/verifier.ts#L148

gives me access to the request body, in the requestFilter, with req.body, I can create the AWS header, with the body contents, but the verifier times out. This might be due to the changes, or due to my providers endpoint, which is painfully slow at times. (times out after 30 seconds, and tends to take a horrendous ~25 seconds to return)

Will do some more digging tomorrow

[YOU54F]

Well the data we have encoded in the pact is garbage (our term matchers generate the string "string" which throws a validation error if directly sent to the provider through postman, so I would expect the verifier to throw an error, and not time out)

[2019-06-12T22:33:11.879Z] DEBUG: [email protected]/54019 on YOU54FMAC: Proxing /decision
{ Error: Timeout waiting for verification process to complete (PID: 54029)
    at Timeout._onTimeout (/Users/you54f/dev/compassdev/compass/compass-pact-provider/node_modules/q/q.js:1846:21)
    at listOnTimeout (internal/timers.js:535:17)
    at processTimers (internal/timers.js:479:7) code: 'ETIMEDOUT' }
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
➜  compass-pact-provider git:(awsVerify) ✗ 

which probably means there is something else amiss.

Time for a beer anyhoo =D

[combmag]

@YOU54F any updates please? since i'm still having the same issues for req.body as undefined

[mefellows]

Not all requests are JSON bodies (could be any type) so that parser hasn't been added. I'll double check, but I'm hopeful it looks at the media type and only attempts to parse a body into req.body if that's true, in which case I'll add that in as it's clearly going to be useful.

In the mean time, you can just read in the body yourself as you would any other node http request. Consult the node docs on how to do that.

[mefellows]

See https://stackoverflow.com/questions/62662605/how-to-modify-a-graphql-variable-in-a-contract-during-the-provider-verification for an example. Albeit it seems it won't let you modify the body anyway

[mefellows]

Summary of current issue.

req.body isn't automatically parsed. I think at the least, we should parse the body. Looking at the problem before, it may be an issue with the way the http proxy wires in to the process - right now, changing the body makes it hang. I suspect this is due to an event not being fired/closed on the event loop.

In any case, there are two issues:

1. req.body is not prepopulated for JSON bodies
2. Even if you parse the body (see https://stackoverflow.com/questions/62662605/how-to-modify-a-graphql-variable-in-a-contract-during-the-provider-verification) you can't modify it.

[combmag]

for now what i have done as a workaround is using an http proxy similar to https://github.com/http-party/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js where i update the body based on some criteria, of course this is not the ideal solution but for now it's the best option i could think of

[zsotyooo]

Hi there,

You have an awesome product. I really enjoy working with PACT.

Do you have an ETA on this issue? The workaround from @combmag is something we can try as well, but we would rather avoid it because it adds a lot of complexity and hides the implementation from the tests.

[mefellows]

Hi Zsolt, thanks for the kind words!

Our v3 branch (the beta release you can find) has support for this. We hope to have it out of beta in the next couple of months, it is only lacking a few features (e.g. message support and Pact Web) and may have a few rough edges API wise.

[zsotyooo]

Hi @mefellows,

Thanks for the quick response. I'll give it a try.

Cheers

[m-radzikowski]

for now what i have done as a workaround is using an http proxy similar to https://github.com/http-party/node-http-proxy/blob/master/examples/middleware/bodyDecoder-middleware.js where i update the body based on some criteria, of course this is not the ideal solution but for now it's the best option i could think of

@combmag could you share a short snippet of how you set up the filter and the proxy? I'm having trouble getting it to work properly. I would appreciate it greatly.