Serverless-python-requirements: dockerizePip: "git clone failed with error code 128 in None" error

Created on 4 Sep 2018  路  10Comments  路  Source: UnitedIncome/serverless-python-requirements

serverless-python-requirements version

4.1.1

Description

I'm trying to deploy my code using serverless-python-requirements, but this error is ocurring when I execute sls deploy. I've changed pip.js a little bit to print the docker options to stdout.

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Installing requirements of requirements.txt in .serverless...
Serverless: Docker Image: lambci/lambda:build-python3.6
[ 'run',
  '-v',
  '"/home/myuser/mycode:/var/task:z"',
  '-v',
  '/home/myuser/.ssh/id_rsa:/root/.ssh/id_rsa:z',
  '-v',
  '/home/myuser/.ssh/known_hosts:/root/.ssh/known_hosts:z',
  '-v',
  '/tmp/ssh-WhmLrY0zrGTs/agent.31137:/tmp/ssh_sock:z',
  '-e',
  'SSH_AUTH_SOCK=/tmp/ssh_sock',
  '-u',
  '1000',
  'lambci/lambda:build-python3.6',
  'python3.6',
  '-m',
  'pip',
  'install',
  '-t',
  '.serverless/requirements',
  '-r',
  '.serverless/requirements.txt' ]

  Error --------------------------------------------------

  The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Command "git clone -q ssh://[email protected]/my-repo/my_private_repo.git /tmp/pip-req-build-i1pzenek" failed with error code 128 in None


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Command "git clone -q ssh://[email protected]/my-repo/my_private_repo.git /tmp/pip-req-build-i1pzenek" failed with error code 128 in None

    at installRequirements (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:148:11)
    at targetFuncs.filter.map.f (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:225:11)
    at Array.map (native)
    at ServerlessPythonRequirements.installAllRequirements (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:220:8)
From previous event:
    at PluginManager.invoke (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.spawn (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:408:17)
    at Deploy.BbPromise.bind.then.then (/usr/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/usr/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Serverless.run (/usr/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/lib/node_modules/serverless/bin/serverless:43:50)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           6.12.0
     Serverless Version:     1.29.2

If I remove the -u 1000 param from docker command line, the code is successfully built. But, another error is displayed when serverless tries to remove the .serverless/requirements/* content because the files inside it are created by root user from within container.

I've tried to run the docker command standalone, but the error persists. Also, more logs are displayed:

docker run -v "/home/myuser/mycode:/var/task:z" -v /home/myuser/.ssh/id_rsa:/root/.ssh/id_rsa:z -v /home/myuser/.ssh/known_hosts:/root/.ssh/known_hosts:z -v /tmp/ssh-WhmLrY0zrGTs/agent.31137:/tmp/ssh_sock:z -e SSH_AUTH_SOCK=/tmp/ssh_sock -u 1000 lambci/lambda:build-python3.6 python3.6 -m pip install -t .serverless/requirements -r .serverless/requirements.txt 
The directory '/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting git+ssh://[email protected]/my-repo/[email protected] (from -r .serverless/requirements.txt (line 3))
  Cloning ssh://[email protected]/my-repo/my_provate_repo.git (to revision 1.0.0) to /tmp/pip-req-build-xkb3_7kq
No user exists for uid 1000
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Command "git clone -q ssh://[email protected]/my-repo/my_provate_repo.git /tmp/pip-req-build-xkb3_7kq" failed with error code 128 in None

How could I solve this???

picture galindro

All 10 comments

@galindro If this problem is caching related, you might try an upcoming pull request that refactors a lot of how caching works, and fixes (for me) problems like this...

Please see: https://github.com/UnitedIncome/serverless-python-requirements/pull/165

Specifically, from inside your codebase...

rm -Rf node_modules/serverless-python-requirements
npm i github:andrewfarley/serverless-python-requirements#master

Then re-run your sls package or sls deploy and let me know if it works.

If it doesn't work, then your problem must lie with the requirements.txt contents (which you didn't paste yet), with git being unable to checkout your code, or with your file/folder permissions/ownership. All three errors are there mentioned above.

  • If it's the permissions thing, my branch above will probably fix it.
  • If it's git, then you might consider pre-checking out that code and telling pip where to find it or pushing your code to a Pip-compatible storage (Artifactory/Nexus/etc) and pulling from there instead of from git. To debug/test this, you might manually exec into the same docker container eg: docker run -ti lambci/lambda:build-python3.6 /bin/sh and "try" to manually do what this plugin is doing to figure out and debug your error further. We're probably not going to be able to solve this without you digging in. :)
  • If git clone's work fine from that container then it's your requirements.txt, I've noticed that pip is a little bit funny and works differently I've had better luck with git+ssh://git@... sometimes, and other times depending on the server and the environment using git+https://... works better. There are actually a slew of other prefixes as well with various combinations of git/ssh/https/etc. Google around and trial/error until you make one work.
AndrewFarley picture AndrewFarley on 4 Sep 2018
👍1

I had the exact same issue (and I initially removed the -u 1000 from the command when I worked on the project, since it makes no sense...), but I have switched to @AndrewFarley version of the plugin a couple of months ago and it fixes it, on top of improving the performance of the plugin overall.

Besides, its PR will probably be merged very soon :crossed_fingers: so you should definitely try it!

cgrimal picture cgrimal on 4 Sep 2018

@AndrewFarley @cgrimal. This is my requirements.txt file content:

requests==2.18.1
pymssql==2.1.3
git+ssh://[email protected]/my-repo/[email protected]
git+ssh://[email protected]/my-repo/[email protected]

I've installed the serverless-python-requirements version that you have mentioned, but it still not working. But now, the error is related to a missing lib inside lambci/lambda:build-python3.6 image. Again, I've put a simple console.log(cmdOptions); at line 225 of pip.js to print the docker run parameters:

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /home/myuser/mycode/requirements.txt in /home/myuser/mycode/.serverless/requirements.txt...
Serverless: Installing requirements from /home/myuser/.cache/serverless-python-requirements/b3221caecae33c28316456045abeb0ab_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Using download cache directory /home/myuser/.cache/serverless-python-requirements/downloadCacheslspyc
[ 'run',
  '--rm',
  '-v',
  '"/home/myuser/.cache/serverless-python-requirements/b3221caecae33c28316456045abeb0ab_slspyc:/var/task:z"',
  '-v',
  '"/home/myuser/.ssh/id_rsa:/root/.ssh/id_rsa:z"',
  '-v',
  '"/home/myuser/.ssh/known_hosts:/root/.ssh/known_hosts:z"',
  '-v',
  '"/tmp/ssh-7QSB5sUaaZov/agent.7861:/tmp/ssh_sock:z"',
  '-e',
  'SSH_AUTH_SOCK=/tmp/ssh_sock',
  '-v',
  '"/home/myuser/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z"',
  'lambci/lambda:build-python3.6',
  '/bin/bash',
  '-c',
  '"chown -R 0\\:0 /var/useDownloadCache && python3.6 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1000\\:1000 /var/task && chown -R 1000\\:1000 /var/useDownloadCache"' ]
Collecting git+ssh://[email protected]/my-repo/[email protected] (from -r /var/task/requirements.txt (line 1))
  Cloning ssh://[email protected]/my-repo/my_lib1.git (to revision 1.0.0) to /tmp/pip-req-build-kek2k7ki
Collecting git+ssh://[email protected]/my-repo/[email protected] (from -r /var/task/requirements.txt (line 2))
  Cloning ssh://[email protected]/my-repo/my_lib2.git (to revision 1.2.2) to /tmp/pip-req-build-ppejnev8
Collecting pymssql==2.1.3 (from -r /var/task/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/4c/c8/5ad36d8d3c304ab4f310c89d0593ab7b6229568dd8e9cde927311b2f0c00/pymssql-2.1.3.tar.gz (897kB)
Collecting requests==2.18.1 (from -r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/5a/58/671011e3ff4a06e2969322267d78dcfda1bf4d1576551df1cce93cd7239d/requests-2.18.1-py2.py3-none-any.whl (88kB)
Collecting jsonschema==2.6.0 (from fx-lib-common==1.0.0->-r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting SQLAlchemy~=1.2.6 (from fx-lib-sqldb==1.2.1->-r /var/task/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/aa/cc/348eec885d81f7260b07d961b3ececfc0aa82f7d4a8f45ff997e0d3f44ba/SQLAlchemy-1.2.11.tar.gz (5.6MB)
Collecting pymysql~=0.8.0 (from fx-lib-sqldb==1.2.1->-r /var/task/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/32/e8/222d9e1c7821f935d6dba8d4c60b9985124149b35a9f93a84f0b98afc219/PyMySQL-0.8.1-py2.py3-none-any.whl (81kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting urllib3<1.22,>=1.21.1 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/24/53/f397db567de0aa0e81b211d81c13c41a779f14893e42189cf5bdb97611b2/urllib3-1.21.1-py2.py3-none-any.whl (131kB)
Collecting certifi>=2017.4.17 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB)
Collecting idna<2.6,>=2.5 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/11/7d/9bbbd7bb35f34b0169542487d2a8859e44306bb2e6a4455d491800a5621f/idna-2.5-py2.py3-none-any.whl (55kB)
Installing collected packages: pymssql, chardet, urllib3, certifi, idna, requests, jsonschema, fx-lib-common, SQLAlchemy, pymysql, fx-lib-sqldb
  Running setup.py install for pymssql: started
    Running setup.py install for pymssql: finished with status 'error'
    Complete output from command /var/lang/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-6rr75k0l/pymssql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-4w96ygzp/install-record.txt --single-version-externally-managed --compile --home=/tmp/pip-target-a_aycsjo:
    setup.py: platform.system() => 'Linux'
    setup.py: platform.architecture() => ('64bit', 'ELF')
    setup.py: platform.linux_distribution() => ('', '', '')
    setup.py: platform.libc_ver() => ('glibc', '2.2.5')
    setup.py: Not using bundled FreeTDS
    setup.py: include_dirs = ['/usr/local/include']
    setup.py: library_dirs = ['/usr/local/lib']
    running install
    running build
    running build_ext
    building '_mssql' extension
    creating build
    creating build/temp.linux-x86_64-3.6
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -I/var/lang/include/python3.6m -c _mssql.c -o build/temp.linux-x86_64-3.6/_mssql.o -DMSDBLIB
    _mssql.c:266:22: fatal error: sqlfront.h: No such file or directory
     #include "sqlfront.h"
                          ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/var/lang/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-6rr75k0l/pymssql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-4w96ygzp/install-record.txt --single-version-externally-managed --compile --home=/tmp/pip-target-a_aycsjo" failed with error code 1 in /tmp/pip-install-6rr75k0l/pymssql/

  Error --------------------------------------------------

  null

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: null
    at installRequirements (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:249:11)
    at installRequirementsIfNeeded (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:413:3)
    at targetFuncs.filter.map.f (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:456:35)
    at Array.map (<anonymous>)
    at ServerlessPythonRequirements.installAllRequirements (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:450:8)
From previous event:
    at PluginManager.invoke (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/usr/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/usr/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           8.11.4
     Serverless Version:     1.29.2

Because the pip install failed, the docker container could not execute these two commands: chown -R 1000:1000 /var/task && chown -R 1000:1000 /var/useDownloadCache. Therefore, if I try to execute sls deploy or sls package again, a permission denied issue occurs, forcing me to do a rm -rf /home/myuser/.cache/serverless-python-requirements/ to fix it. @AndrewFarley, maybe you could change your code to use a semi-colon instead of && between the shell commands executed inside docker container. This will execute all commands regardless of the others commands execution.

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /home/myuser/mycode/requirements.txt in /home/myuser/mycode/.serverless/requirements.txt...
Serverless: Installing requirements from /home/myuser/.cache/serverless-python-requirements/b3221caecae33c28316456045abeb0ab_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Using download cache directory /home/myuser/.cache/serverless-python-requirements/downloadCacheslspyc

  Error --------------------------------------------------

  EACCES: permission denied, open '/home/myuser/.cache/serverless-python-requirements/downloadCacheslspyc/requirements.txt'

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: EACCES: permission denied, open '/home/myuser/.cache/serverless-python-requirements/downloadCacheslspyc/requirements.txt'
    at Object.fs.openSync (fs.js:646:18)
    at installRequirements (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:176:13)
    at installRequirementsIfNeeded (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:413:3)
    at targetFuncs.filter.map.f (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:456:35)
    at Array.map (<anonymous>)
    at ServerlessPythonRequirements.installAllRequirements (/home/myuser/mycode/node_modules/serverless-python-requirements/lib/pip.js:450:8)
From previous event:
    at PluginManager.invoke (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/usr/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/usr/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/usr/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     linux
     Node Version:           8.11.4
     Serverless Version:     1.29.2
galindro picture galindro on 4 Sep 2018

@AndrewFarley I've found a solution for the above problem (related to the missing lib): gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -I/var/lang/include/python3.6m -c _mssql.c -o build/temp.linux-x86_64-3.6/_mssql.o -DMSDBLIB _mssql.c:266:22: fatal error: sqlfront.h: No such file or directory.

What I did was add two more environment variables to the docker run command. I've did this editing again pip.js and adding these two lines right bellow line 160:

      cmdOptions.push('-e', 'PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1');
      cmdOptions.push('-e', 'CFLAGS=-fPIC');

This was the resulting sls package output:

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /home/myuser/mycode/requirements.txt in /home/myuser/mycode/.serverless/requirements.txt...
Serverless: Installing requirements from /home/myuser/.cache/serverless-python-requirements/b3221caecae33c28316456045abeb0ab_slspyc/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Serverless: Using download cache directory /home/myuser/.cache/serverless-python-requirements/downloadCacheslspyc
[ 'run',
  '--rm',
  '-v',
  '"/home/myuser/.cache/serverless-python-requirements/b3221caecae33c28316456045abeb0ab_slspyc:/var/task:z"',
  '-v',
  '"/home/myuser/.ssh/id_rsa:/root/.ssh/id_rsa:z"',
  '-v',
  '"/home/myuser/.ssh/known_hosts:/root/.ssh/known_hosts:z"',
  '-v',
  '"/tmp/ssh-7QSB5sUaaZov/agent.7861:/tmp/ssh_sock:z"',
  '-e',
  'SSH_AUTH_SOCK=/tmp/ssh_sock',
  '-e',
  'PYMSSQL_BUILD_WITH_BUNDLED_FREETDS=1',
  '-e',
  'CFLAGS=-fPIC',
  '-v',
  '"/home/myuser/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z"',
  'lambci/lambda:build-python3.6',
  '/bin/bash',
  '-c',
  '"chown -R 0\\:0 /var/useDownloadCache && python3.6 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 1000\\:1000 /var/task && chown -R 1000\\:1000 /var/useDownloadCache"' ]
Collecting git+ssh://[email protected]/my-repo/[email protected] (from -r /var/task/requirements.txt (line 1))
  Cloning ssh://[email protected]/my-repo/my_lib1.git (to revision 1.0.0) to /tmp/pip-req-build-sqec5kdr
Collecting git+ssh://[email protected]/my-repo/[email protected] (from -r /var/task/requirements.txt (line 2))
  Cloning ssh://[email protected]/my-repo/my_lib2.git (to revision 1.2.2) to /tmp/pip-req-build-vjark09w
Collecting pymssql==2.1.3 (from -r /var/task/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/4c/c8/5ad36d8d3c304ab4f310c89d0593ab7b6229568dd8e9cde927311b2f0c00/pymssql-2.1.3.tar.gz (897kB)
Collecting requests==2.18.1 (from -r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/5a/58/671011e3ff4a06e2969322267d78dcfda1bf4d1576551df1cce93cd7239d/requests-2.18.1-py2.py3-none-any.whl (88kB)
Collecting jsonschema==2.6.0 (from fx-lib-common==1.0.0->-r /var/task/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting SQLAlchemy~=1.2.6 (from fx-lib-sqldb==1.2.1->-r /var/task/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/aa/cc/348eec885d81f7260b07d961b3ececfc0aa82f7d4a8f45ff997e0d3f44ba/SQLAlchemy-1.2.11.tar.gz (5.6MB)
Collecting pymysql~=0.8.0 (from fx-lib-sqldb==1.2.1->-r /var/task/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/32/e8/222d9e1c7821f935d6dba8d4c60b9985124149b35a9f93a84f0b98afc219/PyMySQL-0.8.1-py2.py3-none-any.whl (81kB)
Collecting certifi>=2017.4.17 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB)
Collecting urllib3<1.22,>=1.21.1 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/24/53/f397db567de0aa0e81b211d81c13c41a779f14893e42189cf5bdb97611b2/urllib3-1.21.1-py2.py3-none-any.whl (131kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting idna<2.6,>=2.5 (from requests==2.18.1->-r /var/task/requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/11/7d/9bbbd7bb35f34b0169542487d2a8859e44306bb2e6a4455d491800a5621f/idna-2.5-py2.py3-none-any.whl (55kB)
Installing collected packages: pymssql, certifi, urllib3, chardet, idna, requests, jsonschema, fx-lib-common, SQLAlchemy, pymysql, fx-lib-sqldb
  Running setup.py install for pymssql: started
    Running setup.py install for pymssql: finished with status 'done'
  Running setup.py install for fx-lib-common: started
    Running setup.py install for fx-lib-common: finished with status 'done'
  Running setup.py install for SQLAlchemy: started
    Running setup.py install for SQLAlchemy: finished with status 'done'
  Running setup.py install for fx-lib-sqldb: started
    Running setup.py install for fx-lib-sqldb: finished with status 'done'
Successfully installed SQLAlchemy-1.2.11 certifi-2018.8.24 chardet-3.0.4 fx-lib-common-1.0.0 fx-lib-sqldb-1.2.1 idna-2.5 jsonschema-2.6.0 pymssql-2.1.3 pymysql-0.8.1 requests-2.18.1 urllib3-1.21.1
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage

Isn't there any options to add environment variables to docker container? If no, I could refactor my pip.js to add an option for make it possible. After that I'll open a PR

galindro picture galindro on 4 Sep 2018

@galindro This is already possible with pipCmdExtraArgs, please see: https://github.com/UnitedIncome/serverless-python-requirements#extra-pip-arguments

Go try adding those with that command instead of modifying this plugin code. :) Should do the trick!

AndrewFarley picture AndrewFarley on 4 Sep 2018

@AndrewFarley I don't think that it will work using pipCmdExtraArgs because PYMSSQL_BUILD_WITH_BUNDLED_FREETDS and CFLAGS are environment variables and not a pip command line argument.

galindro picture galindro on 4 Sep 2018

Ahh good point missed that. Then that would be a great MR! You could probably take that pip args as an template for this feature.

AndrewFarley picture AndrewFarley on 4 Sep 2018
👍1

Hello,

I've encountered the same issue but the fix I've used was quite different. I'll put my solution here in case it could help somebody else.

serverless-python-requirements

version 4.3.0

Environment

Windows 10 pro (Docker for Windows)

Description

I wanted to deploy a lambda function with private directory dependencies. Thus, I'm using the option 'dockerSsh: true' to mount my ssh key in the container.

When deploying, I received the error "git clone failed with error code 128 in None"

Issue

As the plugin was mounting the ssh key in the container, the container was using it to git clone the dependencies. The source of the error was that the SSH key has too broad permission (755 instead of 600).
After checking, I've found that docker for Windows has some issues with the permissions definition when mounting files in containers. https://github.com/docker/for-win/issues/2042

First thing, I've tried to modify the plugin to 'chmod' the key before using it. I guess it was a collateral effect that being mounted but the command wasn't able to modify the permissions properly.
Finally, my workaround was to mount the key on another file then copying and applying the proper permission 600 on the copy.

Here is an extract of the modified code in the pip.js file:

 if (options.dockerSsh) {

  // Mount necessary ssh files to work with private repos
  dockerCmd.push(
    '-v',
    `${process.env.HOME}/.ssh/id_rsa:/root/tmp/id_rsa:z`,
    '-v',
    `${process.env.HOME}/.ssh/known_hosts:/root/.ssh/known_hosts:z`,
    '-v',
    `${process.env.SSH_AUTH_SOCK}:/tmp/ssh_sock:z`,
    '-e',
    'SSH_AUTH_SOCK=/tmp/ssh_sock'
  );
  copy_cmd = ['cp', '/root/tmp/id_rsa', '/root/.ssh/id_rsa'];
  chmod_cmd = ['chmod', '600', '/root/.ssh/id_rsa'];
  pipCmds.unshift(chmod_cmd);
  pipCmds.unshift(copy_cmd);
}

It seems quite a hack so could some of you tell me what you are thinking about it and if you have a better way of fixing it?

AJHealey picture AJHealey on 25 Mar 2019
👍1

I'm getting the same failed with error code 128 in None error when it tries to clone my private repository.

environment: macOS 10.14.4

serverless-python-requirements: version 4.3.0

  • dockerizePip: true
  • dockerSsh: true

More specifically, Command "git clone -q ssh://[email protected]/username/project.git /tmp/pip-req-build-uc61gelx" failed with error code 128 in None. Within docker, its a "permission denied (publickey)" error, though I can clone the repo from my machine.

By modifying (replacing) the pipCmds, I cam confirm its mounting my local id_rsa and known_hosts files, but even doing a simple test with ssh -T [email protected] (as bitbucket suggests) gives a permission denied error. Any suggestions?

chirhotec picture chirhotec on 5 Jun 2019

This appears to have been resolved in #271. Closing!

miketheman picture miketheman on 22 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tmclaugh picture tmclaugh  路  4Comments
miketheman picture miketheman  路  3Comments
calclavia picture calclavia  路  5Comments
IanTayler picture IanTayler  路  4Comments
xedef picture xedef  路  3Comments