Serverless-python-requirements: Can i use the private git repository?

[SteveRoland]

Can i use the private repository like bitbucket in requirements.txt?

is there any config sets a ssh key or basic authentication?

thanks.

[dschep]

Do you mean when using docker? Because, when not using docker, yes pip should just find your ssh keys in your home directory. For Docker I could add -v $HOME/.ssh:/root/.ssh, probably optionally.

[SteveRoland]

sls deploy -v $HOME/.ssh:/root/.ssh

Error --------------------------------------------------

Command "git clone -q ${private_repository_url} /tmp/pip-s2e56Y-build" failed with error code 128 in None

 For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

Stack Trace --------------------------------------------

Error: Command "git clone -q ${private_repository_url} /tmp/pip-s2e56Y-build" failed with error code 128 in None

at module.exports.logError.errorHandlingError (/usr/local/lib/node_modules/serverless/lib/classes/Error.js:94:11)
at initializeErrorReporter.then.catch.e (/usr/local/lib/node_modules/serverless/bin/serverless:42:3)
at runCallback (timers.js:672:20)
at tryOnImmediate (timers.js:645:5)
at processImmediate [as _immediateCallback] (timers.js:617:5)

From previous event:
at __dirname (/usr/local/lib/node_modules/serverless/bin/serverless:40:9)
at Object. (/usr/local/lib/node_modules/serverless/bin/serverless:43:4)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:389:7)
at startup (bootstrap_node.js:149:9)
at bootstrap_node.js:504:3

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Forums: forum.serverless.com
Chat: gitter.im/serverless/serverless

Your Environment Information -----------------------------
OS: darwin
Node Version: 6.11.0
Serverless Version: 1.20.2

[dschep]

Sorry, by add that option I meant a change to this plugin's codebase.

[SteveRoland]

Yes, that happen when i'm using docker. I'll be happy if you add them for me. Thank you.

[ConorMcGee]

I tried getting around this by taking the default docker image, commiting the .ssh files to it and setting the new image as dockerImage in serverless.yml, but I'm still seeing the same error. Is there a reason that wouldn't work?

[SteveRoland]

Hi, @mcgeeco .
use the lambci/lambda:build-python2.7 docker image.
You can get the image info in the serverless-python-requirements code. (index.js:87)

[ConorMcGee]

Yeah that's what I tried (lambci/lambda:build-python3.6). Thought it should work all right - I'll double check that I didn't make an error somewhere.

[stickystyle]

@SteveRoland @mcgeeco Did either of you figure out a way to accomplish accessing private git repos from docker?

[ConorMcGee]

@stickystyle No, sorry, I was going to come back to it when the proposed fix above was implemented.

[stickystyle]

I managed to work around it in my project by vendoring the repo into my project. Not ideal, but surely works.

$ mkdir vendor
$ pip download --no-deps --dest ./vendor -e git+https://github.com/myrepo/myproject.git#egg=0.4.2

then add to requirements.txt ./vendor/myproject-0.4.2.zip

Since everything gets downloaded ahead of time with your existing github keys and sent to docker with the rest of the src, there are no issues trying to do it in the container.

[cgrimal]

Hi @dschep (and the others!),

I have the same issue (a private repos with a Numpy dependency), so I really need the dockerizePip option to be set to true.

I debugged by printing the docker command ran by the script, and adjusting it as follows:

• added the -v $HOME/.ssh:/root/.ssh (as per your suggestion)
• removed the -u ... option (don't really understand why it helped)
• added the -it flags to be able to type my ssh passphrase

This is the resulting command:

docker run --rm -it -v /path/to/my/project:/var/task:z -v /home/username/.ssh:/root/.ssh lambci/lambda:build-python3.6 python3.6 -m pip --isolated install -t .serverless/requirements -r .serverless/requirements.txt

If I run it in my terminal, it works!
If I edit the pip.js file to propagate my options to the docker command, I get a the input device is not a TTY error.

One workaround could be to remove my passphrase, but I don't find it very satisfying...

I noticed you added the Help wanted tag to this issue, and I'll be glad to help, by making a PR for instance, but I'd like your advices beforehand. What do you think?

Many thanks,
Clément

EDIT: I tried @stickystyle's solution after posting my comment (which is quite stupid, I admit) and it works quite well indeed! That said, I need an easy solution to automatically do that every time I deploy, as my private repos will be changing very often. Maybe an option to this plugin could work, I'll try to think this through, but any input is welcome!
First idea: a requirements_private.txt file with URL of privates repos, run the pip command on it, update the requirements.txt file with zip file paths, run the "standard" dockerize pip process.

[dschep]

Good to know the docker command itself works. And yes, I'd love a PR (even if it doesn't work with password protected keys, better than nothing)!

Did you add the -it flag when you edited pip.js too? Even if you did, I'm not entirely sure node's child_process.spawn can set a TTY.

As an alternative I wonder if you can -v mount the SSH_AUTH_SOCK (and set that environment variable) to allow keys with passphrases to work if you're using ssh-agent and have run ssh-add. IE:

docker run --rm -v $PWD:/var/task:z -v $SSH_AUTH_SOCK:/var/task/ssh:z -e SSH_AUTH_SOCK=/var/task/ssh

[dschep]

I just thought of another workaround. If the repo referencing the private repo is also private (most likely, i imagine) add your dependency as git+https://[email protected]/user/repo where $TOKEN is a github personal token.

[cgrimal]

Thanks for your help @dschep :+1:

I made a PR with a solution working with password protected keys, using the SSH_AUTH_SOCK as per your suggestion, thanks!

I tried adding the -it flag when I edited pip.js as well, but indeed, I had a the input device is not a TTY error because I think the process is not able to set a TTY.

I haven't tried your last suggestion yet, but I am using bitbucket (Atlassian product) and I have seen we can generate access keys on a per repo level, which might be a solution. Don't know if it is a good/safe solution though, especially as we are multiple developers working on the project.

Thanks again!