Aws-sam-cli: Too many open files error when using a .jar

[pfandrade]

Staring a local API using SAM Local and a java8 project is apparently hitting a open file limit when decompressing the jar.

This always happens when processing the first request.

$ sam local start-api                                                                                                      ⏎
2017/09/08 14:30:54 Connected to Docker 1.30
2017/09/08 14:30:54 Successfully parsed template.yaml
2017/09/08 14:30:54 Fetching lambci/lambda:java8 image for java8 runtime...
java8: Pulling from lambci/lambda
Digest: sha256:7c7de2243109afc521e5414bb71a0b674a6c42019d5d2c92b6a4c88a17de1270
Status: Image is up to date for lambci/lambda:java8
2017/09/08 14:30:56 Fetching lambci/lambda:java8 image for java8 runtime...
java8: Pulling from lambci/lambda
Digest: sha256:7c7de2243109afc521e5414bb71a0b674a6c42019d5d2c92b6a4c88a17de1270
Status: Image is up to date for lambci/lambda:java8

Mounting static files from /Users/pfandrade/Repos/api.outercorner.com/Diagnostics at /
Mounting com.outercorner.api.diagnostics.DiagnosticsHandler (java8) at http://127.0.0.1:3000/diagnostics [POST]
Mounting com.outercorner.api.diagnostics.DiagnosticsHandler (java8) at http://127.0.0.1:3000/diagnostics/{product} [GET]

You can now browse to the above endpoints to invoke your functions.
You do not need to restart/reload SAM CLI while working on your functions,
changes will be reflected instantly/automatically. You only need to restart
SAM CLI if you update your AWS SAM template.

2017/09/08 14:31:17 Invoking com.outercorner.api.diagnostics.DiagnosticsHandler (java8)
2017/09/08 14:31:17 Decompressing diagnostics-1.0.0.jar into runtime container...
2017/09/08 14:31:17 ERROR: Failed to decompress archive: open /private/var/folders/4q/6gh24k9d0q78z_6l2483q2y00000gn/T/aws-sam-local-1504877477416305000/com/amazonaws/services/s3/model/GetBucketMetricsConfigurationResult.class: too many open files

[PaulMaddox]

Can you confirm which version of SAM Local you're running?

[pfandrade]

That would be sam version 0.2.0.

Also, as a side note, if I decompress the .jar and point the CodeUri to that folder (bypassing the decompression step) everything works as expected.

[PaulMaddox]

I believe this is now fixed in our development branch (commit: https://github.com/awslabs/aws-sam-local/commit/ae96c1c419d6424d64691d6c3fa74e8cdab4b3ef) and is just awaiting our next release.

We don't have a fixed date on our next release, it's more just balancing enough new features to be meaningful and not annoy people with having to update to frequently...

Is this something that's particularly frustrating, or are you ok with the workaround you mentioned above for another week or two?

[pfandrade]

The workaround is fine until the release. Thanks for the quick reply on this.

[cvuijst]

Question: is it always required to decompress the jar file? I noticed for multiple invocations, it is doing from start. Especially somewhat larger jar files (containing additional AWS SDK clients) take quite a while to test local in this way.

[PaulMaddox]

It's decompressed on each invoke as the normal development workflow would be:

1. Run sam local start-api
2. Make some changes
3. Run build process (mvn etc)
4. Test in browser
   ... repeat steps 2,3 & 4 until happy...
5. Commit/Deploy

As such, the decompress needs to happen on each invoke in order to reflect any changes that have been made since the last invoke.

[pfandrade]

Perhaps a check to see if the .jar has changed would mitigate the issue? Either using modified time or the hash value.

[PaulMaddox]

I did think of that, and it would work for sam local start-api where we could keep the hash or modified time state in memory, but it wouldn't work for sam local invoke due to separate processes. The inconsistency here feels wrong to me.

[cvuijst]

Thanks, seems reasonable. However, I noticed that the development cycle with creating a shaded jar and unpacking it in the Docker container takes quite a while. I'm now decompression of about 30 seconds with the example that uses SpringBoot (not even containing the AWS client jars that I want). Those jars are pretty stable, only my code changes. So maybe following the unpacked approach as suggested above would speed up things a little.

[pfandrade]

The unpacked approach works fine.

But like you mentioned, waiting 30s for a jar to be decompressed after each request is hindering the developer's productivity.

The inconsistency between the start-api and the invoke methods doesn't sound so bad to me. I'm not sure I would even call it an inconsistency, since the start-api method would also decompress again if it detects any changes. I would just call this an optimisation.

Also, maybe we can have the invoke method do the same. If we can reliably tell, between process launches, where the temporary folder for the .jar decompression is. We can simply save the hash of it side by side, and check it before decompressing again.

Anyway, I think we're getting out of topic. Perhaps a new Issue is in order?

[cvuijst]

:thumbsup:

[jamesbyars]

Interested in the release date when you know it

[woodcockjosh]

@PaulMaddox The workaround to decompress into folder, worked for me but can you please let us know when the planned release is?

[PaulMaddox]

Within the next 24-48hrs.

[pacey]

Hey how is the release going? I ran into this issue with a python build decompressing my virtualenv. I installed sam local today from npm and got version 0.2.0 which looks like the latest.

[sanathkr]

v0.2.2 is out - https://github.com/awslabs/aws-sam-local/releases/tag/v0.2.2