Serverless-python-requirements: Skip package/deployment steps if nothing changed

[AdrianSanchezLopez]

Would it be possible to completely skip the package and deploy steps if nothing actually changed? equal to what the vanilla _Serveless_ does, if no code, dependencies and _serverless.yml_ files changed it skips all the package, upload to provider and deployment steps.

At the moment, if you run sls deploy with this plugin, and re-run it again without any changes at all, it will re-package and re-deploy everything again (as a new version).

This improvement would help a lot in cases like mine, where we have a CD/CI based on a Monorepo, that will automatically deploy and skip when there were no changes.

[AndrewFarley]

So... this feature sounds like it needs to be a serverless feature itself, not a serverless-python-requirements feature.

Given the scope of this plugin, however, I implemented a proof-of-concept that does something similar to this feature and might do enough of what you want. It doesn't re-gather the requirements if requirements.txt hasn't changed. Please see #162 . This helps us cut down on build times greatly, but it of course still re-generates the zip file.

Just my 2c, but I'd say that the scope of this issue is too large, and should be "won't fix" because it needs to be implemented at the serverless level, not at the python requirements plugin level.

[AdrianSanchezLopez]

Hi @AndrewFarley thanks for replying, but that's my point, it is already a Serveless feature.
What I'm saying is that this plugin somehow overrides/breaks that feature.
As I said on the Issue topic: currently in Serveless, if no code, dependencies and serverless.yml files changed it skips all the package, upload and deployment steps.

[AndrewFarley]

@Adriayala I honestly had no idea that serverless even did this until you mentioned it (I've always had to be using this plugin). I just did a brief test of that feature, and you're right... it says Serverless: Service files not changed. Skipping deployment... when not using this plugin, lol. Amazing!

I just peeked at the serverless code that detects and does this feature. It basically still generates the .zip file locally, and then compares the hash of the local zip file against the hash of the file that was deployed. So, what that says to me, is the process of re-generating the pip dependencies doesn't have consistent output. I believe a future version of my proposed merge, re-using the same exact dependencies, will actually make this feature work right.

I'm working out the kinks, but from what I can see, I believe it will.

This really sounds like a wonderful feature, and I'd love to see it as well. I'm going to do some initial tests with some of my ideas about my PR (moving the data into a temp folder) to see if I can get this state to trigger at least once. If I can, then we're on the right path. Thanks for bringing this to light. Keep you posted...

[dschep]

I think https://github.com/UnitedIncome/serverless-python-requirements/pull/147/files#diff-a3004642ed64a6bb2b6651e426d41fb0R32 might handle this issue

[dschep]

(to try it, install the v4 beta with npm i serverless-python-requirements@beta)

[AndrewFarley]

@dschep You're on the right track. I was just digging into this and saw Date(0) is the trick that serverless uses to achieve this which I saw you using in that PR, however this patch doesn't quite fix it. After extracting the .zip with the above beta I still see current modification times of the FOLDERS, but the file times are okay (although far in the future, instead of in the past, but same end-result). Maybe just one more flag to also set the folders modification times?

$ ls -la
drwxr-xr-x   9 farley  staff   288 Mar 14 13:48 PyYAML-3.12.dist-info
-rw-r--r--   1 farley  staff   500 Jan  1  1980 handler.py
drwxr-xr-x  65 farley  staff  2080 Mar 14 13:48 node_modules
-rw-r--r--   1 farley  staff     7 Jan  1  1980 requirements.txt
drwxr-xr-x  36 farley  staff  1152 Mar 14 13:48 yaml

$ ls -la PyYAML-3.12.dist-info/
-rw-r--r--  1 farley  staff   534 Jan  1  2098 DESCRIPTION.rst
-rw-r--r--  1 farley  staff     4 Jan  1  2098 INSTALLER

[AndrewFarley]

Let me know if you have another beta to try, I've a few projects I can test them on.

[dschep]

Will do @AndrewFarley

[AndrewFarley]

While workin' on my merge... this problem piqued my interest. I noticed we're using zip-local module which is basically abandonware. I recommend modifying this plugin's zipping to use the same module that serverless does, which is a well supported and widely used module node-archiver. Anyone interested feel free to pick this up. Please see:

https://github.com/serverless/serverless/blob/master/lib/plugins/package/lib/zipService.js#L83

Which is a reference from the serverless framework of how they do it. I believe that archiver module properly sets the modification/creation times based on your input of all folder paths for a file you add to a zip.

[dschep]

I like the idea of using the same zip code. I initially did exactly that: https://github.com/UnitedIncome/serverless-python-requirements/blob/072915b546f53a914cae9c0b4ac01c0d6dffcd9b/lib/zipService.js

[AdrianSanchezLopez]

Hey, I'm glad to see you guys are making progress in here. So what's the plan, using that zip code instead of the current one? and is there an ETA?
Just asking for adjusting our team development, to see if it is possible for us to keep using this plugin our find an alternative in case this issue could take too much time.
Thanks in advance!

[dschep]

It's not on my(@UnitedIncome's) roadmap, but I'll merge it as soon as it's done. You can follow @AndrewFarley's progress on #165

[AndrewFarley]

I would love to work on this next I know I could use it, but I’m neck deep with my other merge hoping to finish that in a week or two. But if I can I’d say maybe a month and I’ll have some code for this then will just need some reviewing.

[jacintoArias]

Hi, is someone still looking into this?

I don't know if the addition of lambda layers changes how this is behaving. Running serverless 1.38.0 under macos I still have to upload both lambda and layers .zips every time we deploy a service, even if anything changes.

[levFarkas]

Hi, is someone working on this?

The issue is still up, and uploading even nothing has changed takes too much time.

[AndrewFarley]

No one is working on this, but if anyone wants to pick this up, please do.

[grvhi]

At the risk of crossing requirements here, it would seem there's an opportunity here to add a "skip layer building" option as well...? The idea being that if this plugin has been used to deploy python requirements as an independent layer, it should also support options for the user to choose to independently build (and deploy) either the requirements layer or the function code?

[xv-alex-t]

Really really would appreciate this, its a major pain point for us

[tvb]

Where are we with this feature request? My functions are being rebuild each time I deploy even if they or the requirements haven't changed at all.