Serverless-python-requirements: pip install exception on sls deployment

[helveticafire]

Hi,

Thanks the plugin. I am currently having a problem with deploying my small service which is open source luckily. See: https://github.com/helveticafire/serverless-pynamodb-rest-example

Is there something I am missing? Or doing wrong?

It was previously working unfortunately I have not got far enough with CI for auto-deployment to prevent this from happening.

$ sls deploy

Result:

Serverless: Parsing Python requirements.txt
Serverless: Installing required Python packages for runtime python3.6...

  Error --------------------------------------------------

  Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/local/lib/python3.6/site-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/local/lib/python3.6/site-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

    at module.exports.logError (/Users/benfitzgerald/.nvm/versions/node/v7.5.0/lib/node_modules/serverless/lib/classes/Error.js:94:11)
    at initializeErrorReporter.then.catch.e (/Users/benfitzgerald/.nvm/versions/node/v7.5.0/lib/node_modules/serverless/bin/serverless:42:3)
    at runCallback (timers.js:651:20)
    at tryOnImmediate (timers.js:624:5)
    at processImmediate [as _immediateCallback] (timers.js:596:5)
From previous event:
    at __dirname (/Users/benfitzgerald/.nvm/versions/node/v7.5.0/lib/node_modules/serverless/bin/serverless:40:9)
    at Object.<anonymous> (/Users/benfitzgerald/.nvm/versions/node/v7.5.0/lib/node_modules/serverless/bin/serverless:43:4)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:418:7)
    at startup (bootstrap_node.js:139:9)
    at bootstrap_node.js:533:3

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           7.5.0
     Serverless Version:     1.19.0

[srizzling]

I'm seeing this as well.

[dschep]

Are you two running this on macOS without docker? If so I suspected this might be an issue (but haven't confirmed it or been affected because I use Linux and the rest of the team at @UnitedIncome uses macOS but with the docker option). I think a temporary work around is to run the deploy or package step with a virtualenv activated. (ie: virtualenv -p $(which python3.6) venv; source ./venv/bin/activate)

[dschep]

Oh. I see Cellar in your path, so it is a mac with a homebrew version of python. Try the work around I mentioned.

[helveticafire]

@dschep Yes, I am running it without docker. I have solved this issue!

I removed the virtualenv directory and then install again with:
$ python3 -m venv env
Re-ran sls deploy unfortunately that fails

Then I tried:
$ virtualenv -p python3 env
and success I can deploy again.

[dschep]

Ah yes, it needs to be a py3.6 venv. good catch. I'm gonna mark this as a bug since you shouldn't need a work around for this.

[helveticafire]

Awesome! Thanks again for the plugin. Keep up the good work.

[srizzling]

Looks like its because of this btw: https://github.com/Homebrew/brew/blob/master/docs/Homebrew-and-Python.md#note-on-pip-install---user

[dschep]

My current thought is to vendorize pip, it would also avoid an issue on Debian I've worked around: https://github.com/UnitedIncome/serverless-python-requirements/blob/2388e55b7589984865be1a4c800ba746ed2162a2/index.js#L71-L81

[srizzling]

Would it be over-complication to design if the plugin force an isolated environment where the packages are bundled too? Alot of python build tooling (i.e tox) does this, and would make it OS agnostic, but I understand that this isn't easy.

[dschep]

Meaning a virtualenv? That's definitely a possibility, but I do appreciate the simplicity of pip --target

[woozyking]

To @srizzling 's point, if pipenv is available, we could attempt to activate $ pipenv shell and then perform subsequent serverless actions. This may or may not need to be worked into serverless-python-requirements workflow, but definitely an option/workaround.

[dschep]

I'd rather not have a solution that only works with pipenv.

[woozyking]

Fair enough. Would it be a good usage tip to be documented here though?

[dschep]

That's already in the readme: https://github.com/UnitedIncome/serverless-python-requirements#applebeersnake-mac-brew-installed-python-notes

[radius314]

pyenv is a good workaround too.

brew install pyenv
pyenv version 3.6
sls deploy

[radius314]

You also need to prepend to your PATH $(pyenv root)/shims

[bsamuel-ui]

I think if it's worth it to vendorize pip, we can reopen this.