Serverless-python-requirements: example fails on osx due to known bug in brewed pip install --target

Created on 13 Dec 2017  路  8Comments  路  Source: UnitedIncome/serverless-python-requirements

Reproduce

On a mac:

  • get python 3.6: brew install python3
  • make it default interpreter: brew unlink python && brew link python3
  • git clone https://github.com/UnitedIncome/serverless-python-requirements
  • cd serverless-python-requirements/example
  • npm install -g serverless
  • [... config aws credentials]
  • npm install serverless-python-requirements
  • SLS_DEBUG=* sls deploy -v

This is the output:

Serverless: Load command run
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Installing required Python packages with python3.6...

  Error --------------------------------------------------

  Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/local/lib/python3.6/site-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/usr/local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/usr/local/lib/python3.6/site-packages/pip/wheel.py", line 247, in move_wheel_files
    prefix=prefix,
  File "/usr/local/lib/python3.6/site-packages/pip/locations.py", line 153, in distutils_scheme
    i.finalize_options()
  File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

    at ServerlessPythonRequirements.installRequirements (/Users/me/serverless-python-requirements/example/node_modules/serverless-python-requirements/lib/pip.js:76:11)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:366:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:384:17)
    at Deploy.BbPromise.bind.then.then (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:120:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:110:10)
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:366:55)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:366:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:397:17)
    at variables.populateService.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:104:33)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:91:74)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:42:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.9.3
     Serverless Version:     1.24.1

Resolutions

  • Avoiding --target flag would be best. If not possible:
  • Add setup.cfg to example root with this content:
[install]
prefix=
  • Use the dockerImage: lambci/lambda:build-python3.6 directive in example?
bug help wanted
Source
picture avishayp

All 8 comments

Thanks for filing this issue, I've known about it but not really tracked it (I use Linux and those at @UnitedIncome working on macs are also using the dockerizePip directive)

The easiest work around is to create & activate a virtualenv.

re: possible resolutions:

  • avoiding --target - but its so... clean & simple (ignoring the fact brew breaks it, that's brews fault)

    • a setup.cfg - oh? does that work? I'll have to play around with that (tricky tho, since I don't have a mac.. again back to the main reason this hasn't been fixed. If a mac user would contribute a fix I'd love that since I can't test anything)

    • the directive - i assume you meant dockerizePip, but I'd rather that default usage of this pulgin not require having docker installed.

My own ideas:

  • document this issue (definitely should this do for now)
  • detect this particular error and document the work around (aforementioned virtualenv trick)
  • vendorize virtualenv and implement the work around automatically (either detect macs or detect the error and try again with workaround)

I'm open to avoiding --target but I'm not sure what that would entail, since it' does exactly what i want :wink:

dschep picture dschep on 13 Dec 2017

See issue with brew, they have no intention of fixing it AFAICT: Homebrew/brew#821

And their recommended fix doesn't work:

$ /usr/local/Cellar/python3/3.6.3/bin/python3 -m pip install --user --install-option='--prefix=' pip
/usr/local/lib/python3.6/site-packages/pip/commands/install.py:194: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
 cmdoptions.check_install_build_global(options)
Requirement already satisfied: pip in /usr/local/lib/python3.6/site-packages
Tibors-MacBook-Pro:~ tibor$ /usr/local/Cellar/python3/3.6.3/bin/python3 -m pip install --target=target --install-option='--prefix=' requests
/usr/local/lib/python3.6/site-packages/pip/commands/install.py:194: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
 cmdoptions.check_install_build_global(options)
Collecting requests
 Downloading requests-2.18.4.tar.gz (126kB)
   100% |鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 133kB 3.6MB/s
Collecting chardet<3.1.0,>=3.0.2 (from requests)
 Downloading chardet-3.0.4.tar.gz (1.9MB)
   100% |鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 1.9MB 756kB/s
Collecting idna<2.7,>=2.5 (from requests)
 Downloading idna-2.6.tar.gz (135kB)
   100% |鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 143kB 6.0MB/s
Collecting urllib3<1.23,>=1.21.1 (from requests)
 Downloading urllib3-1.22.tar.gz (226kB)
   100% |鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 235kB 5.1MB/s
Collecting certifi>=2017.4.17 (from requests)
 Downloading certifi-2017.11.5.tar.gz (329kB)
   100% |鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 337kB 4.2MB/s
Skipping bdist_wheel for requests, due to binaries being disabled for it.
Skipping bdist_wheel for chardet, due to binaries being disabled for it.
Skipping bdist_wheel for idna, due to binaries being disabled for it.
Skipping bdist_wheel for urllib3, due to binaries being disabled for it.
Skipping bdist_wheel for certifi, due to binaries being disabled for it.
Installing collected packages: chardet, idna, urllib3, certifi, requests
 Running setup.py install for chardet ... done
 Running setup.py install for idna ... done
 Running setup.py install for urllib3 ... done
 Running setup.py install for certifi ... done
 Running setup.py install for requests ... done
Exception:
Traceback (most recent call last):
 File "/usr/local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
   status = self.run(options, args)
 File "/usr/local/lib/python3.6/site-packages/pip/commands/install.py", line 350, in run
   isolated=options.isolated_mode,
 File "/usr/local/lib/python3.6/site-packages/pip/commands/install.py", line 436, in get_lib_location_guesses
   scheme = distutils_scheme('', *args, **kwargs)
 File "/usr/local/lib/python3.6/site-packages/pip/locations.py", line 153, in distutils_scheme
   i.finalize_options()
 File "/usr/local/lib/python3.6/site-packages/setuptools/command/install.py", line 38, in finalize_options
   orig.install.finalize_options(self)
 File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
   "must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both
dschep picture dschep on 13 Dec 2017

The setup.cfg "fix" works, but it's an ugly workaround that should be avoided. Once the deployment I'm working on works I'll submit PR with it as an example and call it mac_example.

avishayp picture avishayp on 13 Dec 2017
👍1

Cool. Thanks. Yeah, seems like a bit of a hack. This plugin could safely use it if there doesn't exists a setup.cfg in the project, but it would fall apart or have unexpected behavior of it already exists, especially if it already includes a prefix directive (tho I imagine in that case, it probably breaks this plugin already)

dschep picture dschep on 13 Dec 2017

@dschep see #108. I did not touch the main README, maybe add a note there to point angry osx mob to the native example. Thanks!

avishayp picture avishayp on 14 Dec 2017

Just a note, using the setup.cfg workaround causes some python wheels to not work, meaning pip will recompile wheels from source tar. Thus whole recompilation process is expensive and might take a long time to complete.

heri16 picture heri16 on 14 Dec 2017
👍1

Good to know @heri16, but if it's compiling anything without using dockerizePip on a mac it won't work on lambda.

dschep picture dschep on 14 Dec 2017

Valid point indeed.

heri16 picture heri16 on 14 Dec 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

cdimitroulas picture cdimitroulas  路  4Comments
gutyril picture gutyril  路  5Comments
amitm02 picture amitm02  路  6Comments
nknotts picture nknotts  路  5Comments
bsamuel-ui picture bsamuel-ui  路  3Comments