Serverless-python-requirements: Problem with pip cache permissions

[RasterBurn]

I'm getting some issues with the docker based build and file permissions inside of the container. Any ideas?

Here are the configuration options:

custom:
  pythonRequirements:
    zip: true
    dockerizePip: true

And here is the error message I'm getting:

Serverless: Removing Python requirements helper...
Serverless: Adding Python requirements helper...
Serverless: Parsing Python requirements.txt
Serverless: Installing required Python packages for runtime python2.7...
Serverless: Docker Image: lambci/lambda:build-python2.7

  Error --------------------------------------------------

  The directory '/root/.cache/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/root/.cache' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 281, in populate_link
    self.link = self._wheel_cache.cached_wheel(self.link, self.name)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 68, in cached_wheel
    self._cache_dir, link, self._format_control, package_name)
  File "/usr/local/lib/python2.7/site-packages/pip/wheel.py", line 129, in cached_wheel
    wheel_names = os.listdir(root)
OSError: [Errno 13] Permission denied: '/root/.cache/wheels/44/56/6f/d5ac6dce9c887d96df2cd7b27ed0a6dd9357ef82b951ebb777'


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.4.0
     Serverless Version:     1.20.2

[RasterBurn]

If I take the -u uid:gid option off of the docker call it works. It seems my uid (501) is not in the docker image, so it barfed when trying to write to /tmp.

Also I had to upgrade go because I was getting a "split stack overflow", but that may just be a docker issue.

Is it permissible to remove the -u option from the docker command?

[dschep]

Is it permissible to remove the -u option from the docker command?

Yeah, that's reasonable. Maybe a noDockerUser option? or dockerUID which if set to a string uses it as the -u option instead of the default and if set to null omits the option.

[AndrewFarley]

+1 been having this problem months ago on OS-X. I started manually packaging that project, but a new project I wanted to use this and found this bug report just now. Removing the -u option fixed this for me. I wouldn't mind creating a MR to help fix this... but I'm not sure the right way to fix this. I can do a noDockerUser as suggested above, or perhaps detect the host OS and skip the -u option altogether if on OS-X.

Thoughts/ideas/feedback from other contributors?

[dschep]

I'm ok with detecting OS. If you check for win32 also you'd fix #21 too.

[dschep]

I've released this in version 3.0.0-beta. It's a major version because the no deploy change is a breaking change and it's a beta because @UnitedIncome is nearing a public launch so I don't wanna pull a bunch of changes into our release.

[AndrewFarley]

Thanks guys... just tried the patch locally also, works great on OS-X. Cheers!

[fernando-mc]

@AndrewFarley and @dschep I'm seeing this issue on Ubuntu. Is the fix here supposed to work on other OSes? Or is there an alternative workaround suggested?

[dschep]

@fernando-mc checkout the v4 beta release, I just merged #145 which may help with this. (npm install serverless-python-requirements@beta)

[fernando-mc]

@dschep this still seems to be failing for me (I turned on the debug logs if it helps):

```Serverless: Load command run
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command wsgi
Serverless: Load command wsgi:serve
Serverless: Load command wsgi:clean
Serverless: Load command create_domain
Serverless: Load command delete_domain
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6

Error --------------------------------------------------

The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Could not find a version that satisfies the requirement pkg-resources (from -r .serverless/requirements.txt (line 14)) (from versions: )
No matching distribution found for pkg-resources (from -r .serverless/requirements.txt (line 14))
You are using pip version 9.0.1, however version 9.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Stack Trace --------------------------------------------

Error: The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Could not find a version that satisfies the requirement pkg-resources (from -r .serverless/requirements.txt (line 14)) (from versions: )
No matching distribution found for pkg-resources (from -r .serverless/requirements.txt (line 14))
You are using pip version 9.0.1, however version 9.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Here's the stack trace:

at installRequirements (/home/fernando/Documents/code/myappdir/node_modules/serverless-python-requirements/lib/pip.js:114:11)
at ServerlessPythonRequirements.installAllRequirements (/home/fernando/Documents/code/myappdir/node_modules/serverless-python-requirements/lib/pip.js:202:5)

From previous event:
at PluginManager.invoke (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
at PluginManager.spawn (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:390:17)
at Deploy.BbPromise.bind.then.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
at Object.before:deploy:deploy [as hook] (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
at BbPromise.reduce (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:55)
From previous event:
at PluginManager.invoke (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
at PluginManager.run (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:403:17)
at variables.populateService.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:102:33)
at runCallback (timers.js:672:20)
at tryOnImmediate (timers.js:645:5)
at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
at Serverless.run (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:89:74)
at serverless.init.then (/home/fernando/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/bin/serverless:42:50)

And my environment info:

Your Environment Information -----------------------------
OS: linux
Node Version: 6.10.3
Serverless Version: 1.26.1
```

Any suggestions?

[fernando-mc]

@dschep I think this might actually be related to either #158 or a bug with Ubuntu. https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463

When I removed the pkg-resources line in my requirements.txt this worked fine.

[dschep]

Thanks for the report @fernando-mc. Was the pkg-resources line version 0.0.0 like that bug in the ubuntu launchpad?

[fernando-mc]

@dschep yep! Just removing the line resolved this. Not sure it will break anything on other OSes doing this but it resolved the issue on Ubuntu 16.04.

[danbovey]

I think I'm seeing the same issue on Ubuntu 16.04, running [email protected].

The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-HoBoR1/psutil/

I don't have pkg-resources in my requirements.txt, so it's a slightly different permissions error from what @fernando-mc had.

Is this the same issue of docker not running pip as the correct user, or an error with this wheels package?

[dschep]

Hmm. are you using a custom docker image?

[danbovey]

@dschep No I'm using the default lambci/lambda image.

[simlu]

Same error/problem here. @dschep How are you starting the docker image?

[AndrewFarley]

Hey @simlu and @danbovey and anyone else with this caching issue still, can you guys do me/us a favor and try my PR to improve caching? See if this solves the problem? In the PR we are making caching an actual feature, not something you "kinda hack in poorly" with a pip option. Additionally, the PR significantly changes (and limits) the scope of what docker does, which should eliminate all permissions issues. To do this...

rm -Rf node_modules/serverless-python-requirements 
npm i UnitedIncome/serverless-python-requirements#pull/165/head

Then in your serverless.yml enable the cache with...

custom: 
  pythonRequirements:
    dockerizePip: true
    useDownloadCache: true

Then try a few full deploys... see if that works. The first time might be slow (normal speed, filling the download cache), but the second should be faster.

If that works for you, please also then try the following to enable the "static" cache (which caches the outputs of pip, not just the downloads). To do this...

custom: 
  pythonRequirements:
    dockerizePip: true
    useDownloadCache: true
    useStaticCache: true

And if that works, you'll see your builds go super-fast after the first time as long as your requirements.txt doesn't change (which it rarely does). Let me know how that works, and if that solves your problem. Thanks!

[simlu]

For now I've fixed this by using

    pipCmdExtraArgs:
      - --no-cache-dir

I'll try your branch next

[simlu]

Getting an error when trying to install as you described:

npm ERR! code 1
npm ERR! Command failed: /usr/bin/git checkout pull/165/head
npm ERR! error: pathspec 'pull/165/head' did not match any file(s) known to git.
npm ERR! 

[simlu]

Tried with npm i github:andrewfarley/serverless-python-requirements#master as per https://github.com/UnitedIncome/serverless-python-requirements/pull/165

but still getting the same permission error. I should highlight that we use our own docker container and then run sls deploy inside it. So we have dockerizePip set to false.

[AndrewFarley]

@simlu Ahh... alright, well, your issue is a bit different than most of the rest above whose problem revolves around this plugin using docker. You've got a bit of a custom thing going... that sounds like you just need to add a chown to one of your steps in your Dockerfile to fix that. Or if your Docker container mounts volumes containing the parent code and that's where you're running serverless from, then that would explain things.

[simlu]

I tried that initially and couldn't get it to work. So I thought this might be a nice and easy workaround. Why does your approach not work for our use case then?

[AndrewFarley]

Because my fork still tries to use and pip cache inside the users home directory (sort of) but I bet your dockerfile and the files in your docker file are mounted via a volume which gives them an invalid/null owner which freaks pip out. If you don’t mind sharing your dockerfile and the script and or command you use to deploy your serverless stacks I’d gladly look at it and either advise you what to change or submit a patch to this software. The problem you are facing is specific to pip, not this plugin. :)

[simlu]

Well, not really. It's using the user coming from the OS so files are accessed/written with the same permissions.

For sure, it's from py-gardener which is published under MIT:

We run . manage.sh to start the docker container (Ubuntu 16.04 / 18.04)

• manage.sh
• Dockerfile

Thank you very much for helping!