Serverless: Serverless Error - Function not found...

[scott-martin]

As of today (1/2/2017), when I try serverless deploy I get this error.

Serverless Error ---------------------------------------

     An error occurred while provisioning your stack: XXXLambdaVersionPUKt3p41YmjAJBYRa6HS1i4GOUTJQpzTYRXOhP8524
     - Function not found: arn:aws:lambda:us-east-1:11xxxxx32:function:xxx-xxx-xxxxx.

Yesterday I was able to deploy my service without failure.

I'm running on a Mac using serverless version 1.4. I have about 4 functions in my template. One is a lambda proxy, 2 sns topic handlers, and 1 function without an event attached to it.

[scott-martin]

This appears to be somewhat similar to #3028. I will try rolling back my serverless version to 1.2 to see if it fixes the issue.

[scott-martin]

Rolling back to version 1.2 did fix the issue for me. So there must have been a regression somewhere between then and now.

[MarcAMartin]

I'm seeing this issue today on 1.3

EDIT: Reverting to 1.2 worked for me, even though I encountered the VPC issue again. Performed the workaround -- deploying twice

[bawdy]

I experienced the same thing on version 1.4 today, while everything was normal yesterday. Reverting back to 1.2 fixed it for me too.

[FLGMwt]

I'm getting a mix of "function not found" and "function already exists" which I'm guessing is just some bad cleanup or delayed function state coming from AWS.

Reverting to 1.2 also worked for me.

[paulSambolin]

+1 same error when deploying a service which worked before the holidays

{ [ServerlessError: An error occurred while provisioning your stack: PutFilterLambdaVersion46ZDPxgESjSd09J1jD7mxwqnjXpsupxWzzaFKKW4ahc - Function not found: arn:aws:lambda:us-east-1:acctid:function:putFilter.] name: 'ServerlessError', statusCode: undefined }

[konstantinvlasenko]

Yeah. It worked yesterday with 1.4. Seems like AWS made some changes in CloudFormation.

[GektorGrom]

Also, you can use different region. us-west-2 works fine with sls v1.4

[pmuens]

Damn! Thanks for reporting!

Sad to see that all of you face this issue. We'll look into it today and try to figure out what's wrong.
Has anyone tried the recent master to deploy the service?

For the record: I was also able to deploy with 1.4 yesterday.

[eahefnawy]

hmmm we're terribly sorry you guys are facing this issue. I've been trying to reproduce it all morning without success. Deployment works as expected for me with multiple versions & regions.

Is there anything specific about your configuration or environment? Also, did you face this issue right after updating to 1.3 or 1.4? Or is it something that just happened in the last few days during the holidays? Just trying to figure out if AWS made some changes. I am aware that they roll their changes gradually with some particular users first.

[pmuens]

I also tried to reproduce this today with a bunch of different setups (which was sadly unsuccessful) 👎

I used this serverless.yml:

service: service

provider:
  name: aws
  runtime: nodejs4.3
  cfLogs: true
  region: eu-west-1

functions:
  hello:
    handler: handler.hello
    events:
      - http: GET hello
  goodbye:
    handler: handler.goodbye
    events:
      - http:
          method: GET
          path: goodbye
          integration: LAMBDA

And deployed to us-east-1, us-west-2, eu-central-1, eu-west-1 and ap-southeast-2 without seeing the error popping up.

Has someone tried to re-deploy today? Might be really something on AWS end.

[paulSambolin]

I have been trying to redeploy from github but since I am deploying from codebuild I am encountering unrelated error due to some newer changes in the repository (due to npm scripts). Trying to figure that one out before opening an issue

[manwaring]

Was having this issue yesterday with 1.4 and 1.3 along with lambda sync issues (even removing a 1.2 stack sometimes left lambda functions deployed, couldn't delete them via the aws console or cli) but no longer.

Today both issues seem resolved - I've deployed and removed several different stacks with no problem in both AWS accounts that were erroring out yesterday. I wonder if the two were related.

[scott-martin]

It did seem like it was mostly an AWS issue. But even so, why would 1.2 work and 1.4 not?

[pmuens]

@scott-martin @ironmanwaring Great to hear that everything is working fine again!

I think it might have smth. to do with the versioning of Lambda functions (which might have failed at AWS end).

Lambda versioning was introduced in Serverless v1.3 --> https://serverless.com/blog/serverless-v1.3.0/

[konstantinvlasenko]

Still having the issue. There is more - after sls remove - I can see my functions were not removed. And I can't delete them, trough console

[pmuens]

A quick update...

We're also facing this right now in our integration tests.

Edit: Not sure if it's maybe a kind of race condition and we need to add a DependsOn somewhere here: https://github.com/serverless/serverless/blob/809a28a55d1feeb0aee141235937478084fde38a/lib/plugins/aws/deploy/compile/functions/index.js#L236-L248 ?

[scott-martin]

@konstantinvlasenko I also noticed the issue with not being able to delete them through the console. If you wait about 2 minutes and refresh the page, they will probably be gone.

[konstantinvlasenko]

@scott-martin yeah. They are gone...and they are back next day. My us-east-1 region is a mess now :) I am getting different kind of errors (function already exists, function not found). Trying with us-east-2 now...

[konstantinvlasenko]

Using @GektorGrom workaround for now. Everything works just fine in us-east-2 region.

[pmuens]

This error is really non-deterministic. Just re-run our integration test suite today (which failed yesterday) and it succeeded w/o having the Function not found error again...

We're on us-east-1.

[konstantinvlasenko]

I think it is something to do with AWS itself. It is better to ask AWS Lambda team to check their internal log when it will happen again.

[pmuens]

Yes @konstantinvlasenko haven't seen the error again.

We'll reach out once it re-occurs.

[aspacebar]

I have noticed I get the error when trying to deploy a lambda function that I have recently deleted. If i change the name of the function using --stage then it works, as long as it's not something I have deleted recently :)

[philltomlinson]

I see this issue also. For a certain function name it will not deploy and says function not found. If I change the function name it works and deploys.

[iNDicat0r]

Same error:

Serverless Error ---------------------------------------

     An error occurred while provisioning your stack: CreatexxxxLambdaVersioneg-------
     - Function not found: arn:aws:lambda:eu-west-1:----------:function:xxxxx-dev-createUser.

[philhardy]

@philltomlinson likewise (I hadn't tried changing the name yet) - I've not seen this issue before, but getting it while defining a fresh function. Nothing in my environment has changed since defining another fresh one a couple of days ago

[iNDicat0r]

@philhardy I tried to add a new function in my .yml file and after that 'sls deploy' gives the above error!!

[jonsharratt]

I am currently also experiencing this issue now. Will have a dig and see if I can find any more info on to why this has started to happen.

[philhardy]

Seems to have resolved itself

[iNDicat0r]

Hmm for me, it still gives that error!

[joeyvandijk]

Suddenly it came back, while the past 2 hours I did not have issues.

Function not found: arn:aws:lambda:eu-west-1:.......

Every time now when I create a new stack.

[philltomlinson]

As previous posts have pointed out try downgrade to v1.2. That fixed it for me for now.

However I'm using the aws ireland region and can not publish a lambda version and get the following error when doing this manually through the console (so there may be some issues with the AWS service itself):

An error occurred publishing a version: Lambda is unable to service your request at this time. Please try again later. (Service: AWSLambdaInternal; Status Code: 503; Error Code: ServiceUnavailableException; Request ID: ....)

[pmuens]

That's a really odd problem. We've also seen this in the past (see posts above) but it seems to be something on AWS end.

FYI: You can opt-out of function versioning in v1.6 (which will be released on Monday).
Here's the PR which implemented that: https://github.com/serverless/serverless/pull/3042

This can be used as a workaround for this quirk...

[mrosett]

I'm seeing this again after manually deleting a function through the AWS console and then redeploying it. I ended up fixing it by running sls remove.

[asplogic]

I ran into a similar issue trying to redploy and kept getting function not found... I had to remove the CloudFormation stack manually after removing the S3 bucket. Not sure if it's the same issue as others.

[iNDicat0r]

If you have removed everything manually and you still get the function not found message, you might want to go and manually delete the cloudwatch log groups. This will solve the issue!

[pmuens]

Closing this issue since it's something on AWS end and unfortunately we cannot do smth. about it yet.

Feel free to continue the discussion here with possible workarounds.

[xzer]

I got this error as same as mrosett commented, removing the function via aws console will cause this issue and sls remove can recover it.

[jesstelford]

5 months later, and this is still happening :(

@mrosett's workaround was successful: Run serverless remove && serverless deploy

[edorivai]

I had the same problem after killing a running sls deploy.

@mrosett's workaround worked for me as well: sls remove && sls deploy.

[railto]

Started getting this when deploying our production stage, doesn't happen on our testing or development stages tho. Really not prepared to run remove on production in order to get past this tho, anyone else had luck with a workaround that doesn't involve deleting the stack?

[nford]

Hitting this error today too, 1.18.1

[railto]

I got around it by manually creating the missing lambda and then doing the deploy, pain but it worked.

[PedroChagasGuerra]

Hey i've found a way to "fix" it manually.

This bug happen when you just delete the lambda function from AWS panel, then when you try to deploy it again the errors appers.

When you delete the lambda function you should delete the API Gateway path too and the CloudFormation Stack. So this way I can deploy the function again via console.

[edorivai]

I know very little of CloudFormation, and the serverless deploy process. But from a more high-level view, I would expect the system to be able to deploy my function + supporting services, even if the stack is currently only partly deployed.

I understand that this is most probably easier said than done, but perhaps this could be the end goal of where we'd like the deployment process to be.

[bradgreens]

Similar to @PedroChagasGuerra I solved my problem by mostly emptying my serverless.yml file, deleting the .serverless generated folder, and deployed mostly from scratch due to some activity that happened in the AWS Console.

[sid88in]

@edorivai thanks! that worked for me

[ro-savage]

@pmuens Is this really something that can't be addressed by serverless?

It seems like deleting a lambda from from the AWS web-interface shouldn't completely break serverless. That appears to me to be a rather large issue.

It's easy enough to say 'just don't do it'. But when you have a large team, its an easy thing for someone to do now or in the future. No one would expect deleting a function would completely kill your deployment process.

[HyperBrain]

@ro-savage This is IMO an integrity issue in CF that is created by deletion of the function. It's not a Serverless issue. The same should happen, if you construct a CF stack manually, so that the functions are owned by the CF stack and then delete a function. The next stack update should fail - until you remove the stack and redeploy it again.

The safe way to prevent such issues is, that you only allow access to lambda:DeleteFunction to users that know what the risks are and how things are glued together behind the scenes in CF.

Another idea would be to open a feature request at AWS to implement a mechanism that prevents deleting functions if they are under CF control.

[pmuens]

@ro-savage thanks for commenting and @HyperBrain thanks for replying (and providing ways to mitigate that) 👍

I agree with @HyperBrain. Unfortunately there's really nothing we can do about that right now since the Serverless Framework uses CloudFormation behind the scenes. If AWS fixes this on their end we could leverage that in the Framework.

[HerbertLim]

Thanks to @PedroChagasGuerra, I solved this issue.
After testing serverless functions, I just deleted Lambda functions at AWS Lambda console to clean up test functions. Next day, when I tried to deploy same test function, I faced this issue.

I removed CloudFormation stacks. Some stack remained undeleted. By investigating events, I had to delete contents of S3 bucket, which contains a zip file and cloudformation json file. After that, I was able to remove unnescessary CloudFormation stacks and redeploy successfully.