Serverless: AWS API Gateway: Lack of support for CF intrinsic functions at `functions[].events[].http.authorizer.arn`

[rochdev]

• What was the config you used?

functions:
  myFunction:
    handler: handler.myFunction
    events:
      - http:
          ...
          authorizer:
            arn: { "Fn::Join" : [ ":", [ "arn:aws:lambda", { "Ref" : "AWS::Region" }, { "Ref" : "AWS::AccountId" }, "function:myAuthorizer" ] ] }
            ...

• What stacktrace or error message from your provider did you see?

Serverless: Packaging service...

  Type Error ---------------------------------------------

     functionArn.split is not a function

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues

     Please report this error. We think it might be a bug.

  Your Environment Information -----------------------------
     OS:                 darwin
     Node Version:       4.3.2
     Serverless Version: 1.6.1

Similar or dependent issues:

• #3129

Additional Data

• Serverless Framework Version you're using: 1.6.1
• Operating System: macOS Sierra 10.12.3

---

Proposed solution:

This report signals two issues:

1. Messed up error reporting (functionArn.split is not a function)

_No need to do anything here, as such errors will be handled neatly, once schema is configured for this event (see: https://github.com/serverless/serverless/issues/8018)_

2. No support for CF intrinsic functions at authorizer.arn.

See: https://github.com/serverless/serverless/issues/3212#issuecomment-696637248

1. When arn is passed in object form. Do not attempt _name_ resolution. Simply do not set it on AWS::ApiGateway::Authorizer resource (it's optional there). Use arn as passed, in _uri_ configuration
2. When arn is passed in string form and no name is passed. deprecate _name_ resolution with message that it won't be auto-resolved in next major and to have authorizer named, name has to be explicitly set in config

[jatsrt]

Similar issue:
One project that is the global authorizer, and exports lambda function ARNs

Other project that want to use that authorizer

authorizer:
  apn:
    Fn::ImportValue: ${self:authService}:${opt:stage, self:provider.stage}:AuthLambdaARN

This yields the same problem as the naming cannot be determined from this. Only other option is to hardcode the ARN for now.

[HyperBrain]

@eahefnawy @pmuens This is a similar issue as the arn parsing in the stream arns. I think we need a general arn parser function somewhere that is capable to
(1) read string formatted arns
(2) read arbitrary dynamic arns (Fn::Join, Fn::GetAttr, Ref)
And return the reference resource logical id.
It makes no sense to fix the problem in various places separately with different implementations.

I suggest to have one parseArn() function in serverless.utils that would be used from everywhere.

In my plugin I used a very generic function for that:

    static findAllReferences(root) {
        const resourceRefs = [];
        const stack = [ { parent: null, value: root, path: '' } ];

        while (!_.isEmpty(stack)) {
            const property = stack.pop();

            _.forOwn(property.value, (value, key) => {
                if (key === 'Ref') {
                    resourceRefs.push({ ref: value, path: property.path });
                } else if (key === 'Fn::GetAtt') {
                    resourceRefs.push({ ref: value[0], path: property.path });
                } else if (_.isObject(value)) {
                    key = _.isArray(property.value) ? `[${key}]` : (_.isEmpty(property.path) ? `${key}` : `.${key}`);
                    stack.push({ parent: property, value, path: `${property.path}${key}` });
                }
            });
        }

        return resourceRefs;
    }

Given a root object (in this case the arn: object) it will return an array of all contained references (including those contained within GetAtt) with their path. You can use _.get(root, refPath) to access the ref object or just use the returned refs (which are the logical resource ids).

Can someone link the other issue as related here?

[pmuens]

@HyperBrain love the idea of having a central ARN parser!

What do you think about opening up a separate issue for this. Or should we rename this one?

This implementation will also be affected by the ARN parser: https://github.com/serverless/serverless/pull/3111#issuecomment-285158339

[HyperBrain]

@pmuens I think we should open a separate issue and link all existing arn parser exception issues there. Then it is clear for every single issue that it will be handled centrally.

[HyperBrain]

Link to https://github.com/serverless/serverless/issues/3212#issuecomment-285731535 as the s3 event bucket specification could also be a generic ARN as an alternative to create the bucket implicitly. This would make the s3 events much more flexible.

https://github.com/serverless/serverless/pull/3309

[pmuens]

Adding #3162 here (as brought up by @HyperBrain)

[HyperBrain]

Adding a good example brought up by @erikerikson in the context of step functions:

Resource: [lambda name | activity name | lambda ARN | activity ARN | Fn::GetAtt (lambda) | Fn::Ref (activity) | Fn::ImportValue]

So the Arn parser should support the following declarations and parse them correctly:

• lambda name
• Fully qualified ARN (arn::....)
• Fn::Ref (resource id)
• Fn::GetAtt (resource id),(identifier)
• Fn::Join (containing a split arn possibly with Fn::Ref, Fn::GetAtt or Fn::ImportValue as parts)
• Fn::ImportValue
• Fn::Sub

Any additions are welcome.

[pmuens]

Thanks for updating @HyperBrain

Here's the link to the comment mentioned above 🔝 --> https://github.com/serverless/serverless/issues/3024#issuecomment-291334434

[jliebrand]

Perhaps I'm missing something; but isn't this issue also about referencing resources from within the same stack?

I have a cognito user pool in my Resources section, and I would like to reference it as an authorizer using Fn::GetAtt but the way the code is written today, this is not possible, right?

ie this is no just for cross stack referencing, is it? Or is there a way of doing what i want that I'm missing?

[HyperBrain]

This task is for a general universal ARN parser that allows specifying references as ARN string, Fn::GetAtt, Ref or whatever notations there are to reference resources. It should unify the references, may it be event declarations or something else.
So you're completely right - the initial reason for the task was, that multiple places where in-stack resources had been referenced where/are implemented differently in the different locations with the effect that e.g. some notations work in one place but not in other places.

[jliebrand]

Ok; and to be clear; right now there's no way to define a Cognito UserPool in the Resources section and then have the functions use events that auth against that newly created user pool, since there's no way to get to the ARN of that created pool; correct?

[erikerikson]

As HyperBrain notes, ARN parser was the original thought this was created under (the stream attribute was used for both Kinesis and Dynamo streams) but it is definitely intended to be a generalized resource reference implementation.

[erikerikson]

The reason for this is current inconsistency in the implementation. That sometimes you can use Ref or ImportValue, et cetera but not others. For a specific case, try it and see.

[pmuens]

As HyperBrain notes, ARN parser was the original thought this was created under (the stream attribute was used for both Kinesis and Dynamo streams) but it is definitely intended to be a generalized resource reference implementation.

The reason for this is current inconsistency in the implementation. That sometimes you can use Ref or ImportValue, et cetera but not others. For a specific case, try it and see.

@erikerikson comments are a pretty good summary of the purpose for the arn parser.

@jliebrand have you used the new cognitoUserPool event source (released in v1.15) to achieve that? Or are you using another solution right now?

[jliebrand]

@pmuens no, I wasn't aware of any cognito event source... What I do now is first deploy my stack with the functions having simply no authorizer at all... Then once the stack is deployed for each of my stages, I add these custom vars to the top of my yml:

custom:
  stagingUserPool: arn:aws:cognito-idp:xxxOne
  masterUserPool: arn:aws:cognito-idp:xxxTwo
  devUserPool: arn:aws:cognito-idp:xxxThree

And then add these authorizers:

          authorizer:
            arn: ${self:custom.${opt:stage}UserPool}

This will obviously break if I ever remove and re-deploy a stack, so it's far from ideal

[pmuens]

Interesting solution. Thanks for sharing 👍

We've just added the cognitoUserPool event source. Only supports triggers right now though...

[hassankhan]

Just chiming in to say a global ARN parser would be incredibly useful.

With #3657 we can define user pools, but we can't reference their ARNs as Lambda authorizers. This unfortunately means either having to split a stack or deploy once without authorizers and then again after retrieving and setting the ARN 😞

[horike37]

I'm also happy it would implement this.
With my https://github.com/horike37/serverless-step-functions plugin, you need describe Lambda ARN directly, so not userfriendly..
Therefore I'd like to use a global ARN parser and would also like to reference Lambda functions ARN

[ProTip]

A general issue that we run into is that an intrinsic function of choice(GetAttr, Sub, Split, Join, Ref, Import, etc) is seemingly randomly unavailable. It feel like there may be a bit of unfortunate one-off logic for config parsing going. Will the "ARN parser" work allow the intrinsic functions to work all the places one would expect(everywhere)?

[erikerikson]

That is exactly the intent (as valid) @protip

[pmuens]

Will the "ARN parser" work allow the intrinsic functions to work all the places one would expect(everywhere)?

@ProTip yes, that's exactly the intention of this issue / proposal (as @erikerikson also pointed out above).

The title might be a little bit misleading. I'll update it.

[pmuens]

Just a quick update on this one.

The inital implementation for a global arn parser can be found in this PR: https://github.com/serverless/serverless/pull/3878. The function is accessible at this.provider.findAllCfReferences().

[scottsanchez]

Glad to see this is making progress. Having to hard-code ARN for cognito identity pools after sls creates them them (rather than being able to dynamically reference them) seems counter intuitive :)

See also:

http://forum.serverless.com/t/cognito-user-identity-pools-as-serverless-yml-resource-defs/2050/

and

http://forum.serverless.com/t/cognito-user-identity-pools-as-serverless-yml-resource-defs/2050/9

[pmuens]

Just a quick update on this one:

https://github.com/serverless/serverless/pull/3878 which included a global arn parser will closed in favor of https://github.com/serverless/serverless/pull/3934 which doesn't include an arn parser implementation.

That's why we labeled it from in-progress back to help-wanted.

(so any help here is highly welcomed! 👍)

[tobyhede]

Is anyone working on this one atm?

[pmuens]

Is anyone working on this one atm?

Hey @tobyhede thanks for asking and jumping in here 👍 💯!

Right now nobody is working on this feature, but it's a feature everyone here is pretty excited about since it opens up lots of new possibilities.

A first pass on this one was done in #3878 here. However this PR will be replaced with another one which doesn't rely on this feature.

Feel free to reach out here if you have any additional questions!

[rifhanakram]

Hi @pmuens
any idea on when this feature will be released ?
Thanks in advance

[alekbarszczewski]

Just started using serverless and referencing user pool arn was the first issue I ran into :(.

[iDVB]

@alekbarszczewski We overcame this issue by separating our "Auth" and "API" into two separate Stacks. Where the Auth stack provides CFN Outputs for the needed things.
Then referencing one from the other by...

auth-stack: my-auth-${self:custom.stage}
userPoolId: ${cf:${self:custom.auth-stack}.UserPoolId}
userPoolClientId: ${cf:${self:custom.auth-stack}.UserPoolClientId}

[alekbarszczewski]

@iDVB Thanks, I will try this!

[raphadko]

+1. Any updates?

[andersonjohnf]

@iDVB That worked great, thanks. Here is my example for posterity:

In my "user" service with the auth function:

functions:
  auth:
    name: "${self:provider.stage}-user-auth"
    handler: "handler.auth"

resources:
  Outputs:
    LambdaAuthorizer:
      Value: {"Fn::Join": [":", ["arn:aws:lambda", {Ref: "AWS::Region"}, {Ref: "AWS::AccountId"}, "function:${self:provider.stage}-user-auth"]]}

In my other services:

functions:
  myFunction:
    handler: "handler.myFunction"
    events:
      - http:
          path: 'path/to/function'
          method: "get"
          authorizer:
            arn: "${cf:user-${self:provider.stage}.LambdaAuthorizer}"
            resultTtlInSeconds: 120
            identitySource: "method.request.header.Authorization"
            type: "token"
  otherFunction:
    handler: "handler.otherFunction"
    events:
      - http:
          path: 'path/to/function'
          method: "get"
          authorizer:
            arn: "${cf:user-${self:provider.stage}.LambdaAuthorizer}"

[logemann]

just stopping by to vote for the arn parser feature cause otherwise life is harder than it should be :-)

[et304383]

1.5 years later. Not resolved. :(

I don't understand why the ability to reference a user pool created by serverless was added to the cognitoUserPool event type but not in general. This seems so counterintuitive. I want to use APIG + Cognito authorizer and I simply can't. Why isn't this higher priority?

[otbe]

You can! Its not so intuitive as it could be, but its definitly possible.
As always you can augment function definitions:

functions:
  hello:
    handler: src/index.handle
    events:
      - http:
          method: get
          path: /hello
          cors: true

resources:
  Resources:
    UserPool:
      Type: AWS::Cognito::UserPool
      Properties:
        UserPoolName: my-user-pool

   # augments the compiled function cf json
    ApiGatewayMethodHelloGet:
      Properties:
        AuthorizationType: COGNITO_USER_POOLS
        AuthorizerId:
          Ref: CognitoAuthorizer

    CognitoAuthorizer:
      Type: AWS::ApiGateway::Authorizer
      Properties:
        IdentitySource: method.request.header.Authorization
        Name: my-authorizer
        RestApiId:
          Ref: ApiGatewayRestApi
        Type: COGNITO_USER_POOLS
        ProviderARNs:
          - Fn::GetAtt: [UserPool, Arn]

It took me a while to figure this out, maybe this should be added to the docs in the meantime?

[et304383]

@otbe this is the first useful workaround in this entire thread. Thanks!

[jagthedrummer]

Thanks a ton for the workaround @otbe. This is way better than hard coding an ARN.

(I hope that the workaround will be unnecessary soon and that we'll be able to reference other parts of the cloudformation stack directly (or do I mean indirectly?) where the function and event is defined.

[sortegam]

@otbe I see the purpose of your example but I don't understand how you actually setup / assign the created Cognito Authorizer programatically into the hello function. Is this really possible?

[otbe]

@sortegam
In my example ApiGatewayMethodHelloGet is the generated Cloudfromation resource name of the hello method. Serverless merges all resources into one JSON. The content of cloudformation-template-update-stack.json looks like this:

{
"ApiGatewayMethodHelloGet": {
  "Type": "AWS::ApiGateway::Method",
  "Properties": {
    "HttpMethod": "GET",
    "RequestParameters": {},
    "ResourceId": {
      "Ref": "ApiGatewayResourceHello"
    },
    "RestApiId": {
      "Ref": "ApiGatewayRestApi"
    },
    "ApiKeyRequired": false,
    "Integration": {},
    "MethodResponses": [],
    // the next two lines come from our augmented method definition
    "AuthorizationType": "COGNITO_USER_POOLS",
    "AuthorizerId": {
      "Ref": "CognitoAuthorizer"
      }
    }
  }
}

No programatically extra work needed here 😃

[sortegam]

@otbe Aha! I didn't know that! Thank you very much!

[RobertBrewitz]

https://forum.serverless.com/t/dynamic-arn-to-cognito-authorizer/2161/6

If anyone's interested in a pointer for this issue.

[Curtis017]

Thank you @otbe, that solution is awesome!!!! To @sortegam 's point above, I believe you forgot to add the cognito authorizer to the function in your yaml example:

functions:
  hello:
    handler: src/index.handle
    events:
      - http:
          method: get
          path: /hello
          cors: true
          authorizer: # <--- declare authorizer for the function 
            type: COGNITO_USER_POOLS
            authorizerId: 
              Ref: CognitoAuthorizer

resources:
  Resources:
    UserPool:
      Type: AWS::Cognito::UserPool
      Properties:
        UserPoolName: my-user-pool

   # augments the compiled function cf json
    ApiGatewayMethodHelloGet:
      Properties:
        AuthorizationType: COGNITO_USER_POOLS
        AuthorizerId:
          Ref: CognitoAuthorizer

    CognitoAuthorizer:
      Type: AWS::ApiGateway::Authorizer
      Properties:
        IdentitySource: method.request.header.Authorization
        Name: my-authorizer
        RestApiId:
          Ref: ApiGatewayRestApi
        Type: COGNITO_USER_POOLS
        ProviderARNs:
          - Fn::GetAtt: [UserPool, Arn]

[maddymaster]

@otbe thank you for the solution.

[agroff]

@otbe That worked great for getting a UserPool set up but now when I run serverless remove I get the error "Stack with id UserPool does not exist." Is this expected with using custom resources. If so is there any work around?

[otbe]

Mh. Can't reproduce this on our stacks. Deployed and removed several stacks last week with current sls 🤔
Maybe it's related to something different?

[agroff]

Hmmm yea I definitely broke something. I get that error no matter what I do now. I'll keep digging. Thank you!

[et304383]

@Curtis017 that is the entire point of this thread - the authorizer section of function doesn't support Ref or any other intrinsic functions. That's the point of having to declare ApiGatewayMethodHelloGet which augments the function CloudFormation.

sortegam's question was even answered by otbe, explaining this. Did you miss these replies?

Did you even try your code? It won't work. You should delete it.

[OndeVai]

I hate the to be the guy that whines about this wonderful open source project or the folks that selflessly contribute great work to it, but this bug is a major hinderance. Hard coding the User Pool arn is a nasty solution and having to split your Cognito into a separate stack is equally nasty (for us). It's been almost a year and still no fix. My naive question is this: can a quick fix be put in just to getauth:{dynamicclounfrontarn} working for api's, then work on the grander solution later?

[orndorffgrant]

@OndeVai

@RobertBrewitz posted this a while ago in this thread, but one of his links has broken so I'll leave this here. This solution does not require hardcoding, separate stacks, or augmenting each function in CloudFormation.

...

functions:
  getStuff:
    handler: path/to/handler
    events:
      - http:
          path: /path/to/whatever
          method: get
          authorizer:
            type: COGNITO_USER_POOLS
            authorizerId:
              Ref: ApiGatewayAuthorizer

...

resources:
  Resources:
    CognitoUserPool:
      Type: AWS::Cognito::UserPool
      Properties:
        UserPoolName: whatever
    ApiGatewayAuthorizer:
      DependsOn:
        - ApiGatewayRestApi
      Type: AWS::ApiGateway::Authorizer
      Properties:
        Name: whatever
        IdentitySource: method.request.header.Authorization
        RestApiId:
          Ref: ApiGatewayRestApi
        Type: COGNITO_USER_POOLS
        ProviderARNs:
          - Fn::GetAtt: [CognitoUserPool, Arn]

...

This is working for me with serverless 1.32.0

[amitaymolko]

Works well!!
Thanks a lot!

[RobertBrewitz]

@orndorffgrant thanks, links fixed in the forums :+1:

[ghost]

could you help me please , how can i create an event for S3 bucket which already exists , i dont want to overwrite it

[OndeVai]

@orndorffgrant Thanks!

[tbabak]

could you help me please , how can i create an event for S3 bucket which already exists , i dont want to overwrite it

@michaelm88 this does not seem to be the right place for your question but use serverless-plugin-existing-s3 maybe?!

[andreanaya]

After a lot of try and error, I figured that if you add a name prop the whole thing works. I didn't try to find why it works, to be honest. I'll leave that for @rochdev ;)

hello-world:
  handler: hello-world.handler
  events:
    - http:
        path: /hello-world
        authorizer:
          name: SharedAuthorizer
          arn:
            'Fn::ImportValue': ${self:provider.stage}-SharedAuthorizer

[pmuens]

Another interesting suggestion / feedback can be found here.

[brettstack]

Any reason not to add checks for authorization.type === 'COGNITO_USER_POOLS' || in these places:

1. https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/apiGateway/lib/method/authorization.js#L39
2. https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/apiGateway/lib/authorizers.js#L28
3. https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/apiGateway/lib/permissions.js#L53

It works for my template, but not sure if there are edge cases that it may break. Happy to send a PR. @pmuens

Doing a search on cognitoIdpArnExpr https://github.com/serverless/serverless/search?q=cognitoIdpArnExpr&unscoped_q=cognitoIdpArnExpr I see there may be other places to patch.

[rodrigogs]

After a lot of try and error, I figured that if you add a name prop the whole thing works. I didn't try to find why it works, to be honest. I'll leave that for @rochdev ;)

hello-world:
  handler: hello-world.handler
  events:
    - http:
        path: /hello-world
        authorizer:
          name: SharedAuthorizer
          arn:
            'Fn::ImportValue': ${self:provider.stage}-SharedAuthorizer

It doesn't work if you have several services where you wish to use the same authorizer. It gives me Authorizer name must be unique. Authorizer authorizer already exists in this RestApi.

Is there a really simple way to share an authorizer? Cuz, for God's sake, it should be pretty basic!

[rodrigogs]

After a lot of try and error, I figured that if you add a name prop the whole thing works. I didn't try to find why it works, to be honest. I'll leave that for @rochdev ;)

hello-world:
  handler: hello-world.handler
  events:
    - http:
        path: /hello-world
        authorizer:
          name: SharedAuthorizer
          arn:
            'Fn::ImportValue': ${self:provider.stage}-SharedAuthorizer

It doesn't work if you have several services where you wish to use the same authorizer. It gives me Authorizer name must be unique. Authorizer authorizer already exists in this RestApi.

Is there a really simple way to share an authorizer? Cuz, for God's sake, it should be pretty basic!

It works if you give a unique authorizer name to each service... which looks strange.

[medikoo]

If I see correctly, this report signals two issues:

1. Messed up error reporting, functionArn.split is not a function signals lack of support for non-string format for authorizer.arn -> This will be handled neatly, once schema is configured for this event (see: https://github.com/serverless/serverless/issues/8018)

2. No support for CF intrinsic functions at authorizer.arn.

To solve latter, there's a call to introduce a _global arn parser_, and implenting such doesn't seem to be trivial. In other functionalities we handle any resolution from CF intrinsic functions case by case, and I think similar approach should be taken (if needed) here.

However, when I looked closer, I've noticed that the only reason we try to inspect arn is to resolve a referenced _resource name_, so it's used as a _name_ of an APIGW authorizer. This approach, appears to derive from initial implementation, where just authorizers that referenced functions in a same service were supported (then we accepted only name property, which was used for authorizer _name_, and to construct authorizer _uri_.

When support for custom authorizers was added, _name_ was unconditionally assumed from passed _arn_, and that seems problematic (later it was improved, so we do not attempt to resolve if authorizer.name is passed directly)

If I read correctly _name_ is not a required property on AWS side: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-authorizer.html#cfn-apigateway-authorizer-name

I believe right fix, is to drop the resolution of _name_ whenever arn is passed, and by doing that we can support any format on authorizer.arn.

While, if for some reason we want to have _name_ unconditional, then imo right fix would be to unconditionally require name with arn.

I've re-titled issue and updated top description. PR with a fix is welcome