Serverless-python-requirements: noDeploy defaults lost when defining overrides

[JBartlett86]

Hi,

On setting some additional noDeploy overrides within my configuration I noticed my deployed package instantly began to bloat in size. It turns out that by setting an override all the core default are lost.

Defaults:

noDeploy: [
          'boto3',
          'botocore',
          'docutils',
          'jmespath',
          'python-dateutil',
          's3transfer',
          'six',
          'pip',
          'setuptools'
        ]

This means you begin to include things like boto3 within your deployment package unless you replicate the overrides defined above in your serverless yaml (in addition to your specific overrides).

This seems a bit wrong as you will never want to deploy the above libraries and it would seem to make more sense to include the noDeploy overrides specified within the serverless.yaml as an override to this list not as a replacement for it?

Thanks,
John

[petergaultney]

+1

Also, I am noticing that noDeploy doesn't appear to do anything for me (e.g. boto and botocore are in my .requirements.zip. Is this because I'm also using zip: true, or could it be related to dockerizePip: 'non-linux'?

[dschep]

Could you both share your relevant sections of serverless.yml? noDeploy has defaults and setting it in your config sets it, it doesn't add to it. This is because otherwise, you wouldn't be able to remove anything from it. So, if you want to not include any of the faults AND say, flask, your config should have:

custom:
  pythonRequirements:
    noDeploy: 
      - boto3
      - botocore
      - docutils
      - jmespath
      - python-dateutil
      - s3transfer
      - six
      - pip
      - setuptools
      - flask

[Laurian]

@petergaultney I have the same issue, zip: true looks like it isn't affected by noDeploy :(

[Laurian]

OK, based on https://github.com/UnitedIncome/serverless-python-requirements/issues/138#issuecomment-366936615 I managed to remove things from .requirements.zip with serverless-scriptable-plugin like this:

custom:
  pythonRequirements:
    dockerizePip: true
    zip: true
    noDeploy:
      - boto3
      - botocore
  scriptHooks:
    before:package:createDeploymentArtifacts:
      - "zip -d .requirements.zip boto3/* botocore/* || echo ERR: UNABLE TO REMOVE FROM ZIP"

[mrtj]

@dschep indeed I use this plugin to prepare my greengrass lambda functions and i DO need boto3, botocore, etc. embedded in my package as they are not available by default in the greengrass lambda environment

[dschep]

Thanks for the reports & work around. I'll take a look at this tonight.

@mrtj, interesting. I'd never heard of greengrass.

[johnbartlettapak]

Hi,

I was just wondering whether there had been any movement on;

1. Keeping the default set of exclusions even after setting the noDeploy parameter
2. Removing dependencies that are in the noDeploy but are brought in by other requirements - For the moment I could just use the serverless-scriptable-plugin solution above but am interested to know if you are planning on implementing a solution?

Thanks,
John

[dschep]

@Laurian, I just added a test.. AFAICT zip & noDeploy work fine together. See 4447b28

@johnbartlettapak,

1. does https://github.com/UnitedIncome/serverless-python-requirements/issues/186#issuecomment-430615661 not work for you?
2. hadn't even considered that. It's not on the road map. It goes counter to what I know about python packaging, so I'm not super comfortable doing something like that. If package A you depend on doesn't actually depend on package B then package B shouldn't be a dependency of package A! If it's an optional dependency, that's what extras are for. I had a somewhat similar issue with records depending on pandas even tho it isn't actually a hard dependency. I fixed this by making a PR upstream to fix the problem instead of packaging hacks: https://github.com/kennethreitz/records/pull/113

[johnbartlettapak]

@dschep

1. To fix the issue with the overrides I didn't need to use the plugin as I just had to re-enter them - so include boto3 within my noDeploy list worked.
   This just feels a bit wrong as why should I reenter boto3 just because I now want to exclude pytest?

2. I think this issue specifically relates to boto/boto3 libraries - I am using smart_open which depends on boto3 so when you trim boto3 out of the requirements.txt it gets brought in anyway when smart_open is resolved.
   Now the serverless-scriptable-plugin does seem to fix the problem by allowing me to strip those plugins out but is this something you envision being part of your plugin?

Thanks,
John

[dschep]

Ah, I see your point for the 2nd one now. makes sense for things like boto3.

[DenisStad]

I don't think zip: true and noDeploy work together. I see you added a test @dschep, but if I have zip: true the removed requirements are in the package, if I set it to false or comment it out they are gone. I'm checking by hand.

I believe it's because injectRequirements(https://github.com/UnitedIncome/serverless-python-requirements/blob/master/lib/inject.js#L37) is filtering the requirements if zip is false (https://github.com/UnitedIncome/serverless-python-requirements/blob/master/lib/inject.js#L119), but for the zipped requirements if zip is true, nothing is filtered, which should happen here https://github.com/UnitedIncome/serverless-python-requirements/blob/master/lib/zip.js#L94.

Update:
I think I found out why your test works: If the requirement is explicitly listed in requirements.txt, it works as intended and the removed requirement is not in the final package. In my case, I tried to remove a requirement that was not explicitly listed, but installed by pip because another requirement depended on it.

[radove]

Hello, I had a similar issue but I was trying to achieve the opposite. When I have zip: true and I have a noDeploy of [], a newer boto3 version would get rolled up in the requirements.zip, but when the Lambda ran, it was running the "default" boto3 version that comes with the Lambda. In order to fix this issue, when I comment out the zip: true line, it allowed me to utilize my local boto3 version from Pipfile (and from the requirements.zip/txt). This issue gave me the hint of commenting out the zip: true line to have success with overriding AWS's boto3 default.

[blackmamo]

I just added this PR https://github.com/UnitedIncome/serverless-python-requirements/pull/354. But I also notice this issue https://github.com/UnitedIncome/serverless-python-requirements/issues/304. Not sure what way the maintainers want to go, but please consider it

[zbyte64]

I am still having issues with this. What I have seen is that the tests cover removal of an optional package: flask and the inclusion of a core package boto3. Is there a test of noDeploy on boto3 or botocore (or other core packages)?

I have tried combinations of zip & dockerizePip

serverless: 1.48.4
serverless-python-requirements: 4.3.0

[dschep]

closing since in v5 there are no defaults, per aws recomendations for pckaging