Sentry-python: v0.18.0+ regression with batched lambda events

[stickystyle]

This line got introduced in v0.18.0 that assumes that the event object will always be a dict, but there are cases where the lambda event object can be a List[dict].

BatchInvoke events with AppSync [1] are the ones that I'm running up against right now after trying to upgrade the sdk. Anytime I invoke the lambda with a resolver configured with BatchInvoke, the lambda function fails right away (and without a sentry error, which is understandable...but made finding this a real pain)

[ERROR] AttributeError: 'list' object has no attribute 'get'
Traceback (most recent call last):
  File "/var/task/sentry_sdk/integrations/aws_lambda.py", line 106, in sentry_handler
    headers = event.get("headers", 
{}
)

[1] https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-lambda.html#request-mapping-template

[tomazberisa]

We have also observed this unexpected behavior.

As per AWS Lambda docs [1], event is _usually_ dict, but can also be list, str, int, float, or NoneType, therefore the existence of .get() should not be assumed in the line @stickystyle pointed out.

[1] https://docs.aws.amazon.com/lambda/latest/dg/python-handler.html

[untitaker]

We are working on a fix.

[lobsterkatie]

@stickystyle @tomazberisa - I am working on this, but don't have an AppSync set up, and am striking out in my googling today. Can you tell me:

If you are batch-invoking, and event is actually a list of events, which of the bits of data we use would you expect to be the same across events within the list? The entries we look for and use are "headers", "httpMethod", "queryStringParameters", "identity", ""body", and "path".

Thanks!

[tomazberisa]

FWIW, we are not batch processing, rather using AWS Lambda directly.

We've been able to replicate this issue by creating a Lambda function with the Sentry integration enabled and specifying a non-dictionary (e.g., string, number, or list) test event using the AWS Console.

Not sure what input shape from AppSync would be expected (defer to @stickystyle for that), but think there shouldn't be a hard expectation on event shape for Lambda in the general case.

[lobsterkatie]

there shouldn't be a hard expectation on event shape for Lambda in the general case.

Yup, I'm making it more general - if event is a string, int, etc it clearly doesn't have headers, a method, query params... so I'll just skip pulling that data if .get() fails. (Further, even when event is a dictionary, we do .get()s everywhere, rather than assuming a particular value will be there.)

But in the case @stickystyle mentioned, where event is a list of dictionaries, I'm trying to figure out whether it's safe to just grab the first one and pull data from it (which I can do for any data which is the same across all dictionaries) or if I need to somehow account for data being different across events.

[stickystyle]

@lobsterkatie In my case, yes the user generated data contained in each event member can be, and usually is different in the AppSync context. But stuff like headers (which I actually don't pass through into my lambdas) and the aws_request_id that are tied to the request would be the same for each event member (the payload in the example at the bottom), should I pass them in.
In my lambda I just grab the values from the first member in the list for setting the sentry tags, and have never had an issue with it, so that would seem like a viable solution.

For your reference, here is what a sanitized request object looks like coming into my graphql resolver lambda

{
  "aws_request_id": "37ebc22a-e1d4-493d-90b3-0c37d4dfb23f",
  "correlation_id": "54f6602d-6452-4756-9302-1c742d92a7a5",
  "user_agent": null,
  "release": "seed-907-40ad8e39",
  "event": [
    {
      "type": "Address",
      "field": "country",
      "source": {
        "id": "QWRkcmVzcy02MjA="
      },
      "arguments": {
      },
      "identity": {},
      "correlation_id": "54f6602d-6452-4756-9302-1c742d92a7a5",
      "user_agent": null,
      "selection_set_list": [
        "id",
        "name",
        "iso2",
        "iso3"
      ]
    },
    {
      "type": "Address",
      "field": "country",
      "source": {
        "id": "QWRkcmVzcy02MjI="
      },
      "arguments": {
      },
      "identity": {},
      "correlation_id": "54f6602d-6452-4756-9302-1c742d92a7a5",
      "user_agent": null,
      "selection_set_list": [
        "id",
        "name",
        "iso2",
        "iso3"
      ]
    },
    {
      "type": "Address",
      "field": "country",
      "source": {
        "id": "QWRkcmVzcy02MjU="
      },
      "arguments": {
      },
      "identity": { },
      "correlation_id": "54f6602d-6452-4756-9302-1c742d92a7a5",
      "user_agent": null,
      "selection_set_list": [
        "id",
        "name",
        "iso2",
        "iso3"
      ]
    },
    {
      "type": "Address",
      "field": "country",
      "source": {
        "id": "QWRkcmVzcy03OTc="
      },
      "arguments": {
      },
      "identity": {},
      "correlation_id": "54f6602d-6452-4756-9302-1c742d92a7a5",
      "user_agent": null,
      "selection_set_list": [
        "id",
        "name",
        "iso2",
        "iso3"
      ]
    },
    {
      "type": "Address",
      "field": "country",
      "source": {
        "id": "QWRkcmVzcy01MTg5"
      },
      "arguments": {
      },
      "identity": {},
      "correlation_id": "54f6602d-6452-4756-9302-1c742d92a7a5",
      "user_agent": null,
      "selection_set_list": [
        "id",
        "name",
        "iso2",
        "iso3"
      ]
    }
  ],
  "logger": "handler",
  "level": "info"
}

And for further context, the mapping template (https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-lambda.html#request-mapping-template) that generated the request

$util.qr($context.stash.put("correlation_id", $context.request.headers.get("x-correlation-id")))

{
    "version": "2018-05-29",
    "operation": "BatchInvoke",
    "payload": {
        "type": "${context.info.parentTypeName}",
        "field": "${context.info.fieldName}",
        "source":  $utils.toJson($context.source),
        "arguments":  $utils.toJson($context.arguments),
        "identity" : $utils.toJson($context.identity),
        "correlation_id": $utils.toJson($context.request.headers.get("x-correlation-id")),
        "user_agent": $utils.toJson($context.request.headers.get("x-user-agent")),
        "selection_set_list": $utils.toJson($context.info.selectionSetList)
    }
}

[lobsterkatie]

@stickystyle - thanks for that. A few more questions, if I may:

So the event entry is what gets passed to the handler as its first argument, I take it? Are the $context from your mapping template and the context that gets passed to your handler related? When event is a list, is context still just a single entity? (That's very much what it looks like, but I couldn't find anywhere in the docs which said specifically.)

Anyway, looking now at the Working with other services section of the lambda docs, I'm beginning to get a sense of how varied the shape of an event can be! Many of them do have headers, path, body, and the like, but certainly not all. In any case, it sounds like pulling data from the first event in the list will be sufficient in most cases, so I'll do that.

[stickystyle]

Yes, the content of payload in the mapping template is what gets put into the first arg (event) of the lambda, the $context in the mapping template is unrelated to the second arg context of the lambda handler. The context arg in the lambda is AFAIK always a single object [1].

[1] https://docs.aws.amazon.com/lambda/latest/dg/python-context.html

[lobsterkatie]

Great, thanks!

And yup - have seen that page, but wasn't sure if multiple entries inside of event meant there might be multiple context objects. But the two "contexts" being unrelated makes more sense now, as the one in your mapping template calls upon properties not listed on that spec page for the passed context object. Them being independent also explains how in your mapping template, there seemed to be one $context per record, whereas you've just confirmed that there's only one passed context object per batch request.

About to push a fix.