Pip: Error when trying to install package from github repo with submodule

[0xf0f]

Environment

• pip version: 19.1.1
• Python version: 3.7
• OS: Windows 10 x64

Description
Pip is unable to install the package at https://github.com/pyminimp3/pyminimp3.
There's a permission error while cloning its submodule. Judging by Host key verification failed, it looks like git tries to use SSH to clone it when the initial command is requesting https.

Expected behavior
Package is cloned and installed.

How to Reproduce
Run the command pip install git+https://github.com/pyminimp3/pyminimp3

Output

C:\Users\admin>pip install git+https://github.com/pyminimp3/pyminimp3
Collecting git+https://github.com/pyminimp3/pyminimp3
  Cloning https://github.com/pyminimp3/pyminimp3 to c:\users\admin\appdata\local\temp\pip-req-build-dw1g94wp
  Running command git clone -q https://github.com/pyminimp3/pyminimp3 'C:\Users\admin\AppData\Local\Temp\pip-req-build-dw1g94wp'
  Running command git submodule update --init --recursive -q
  Host key verification failed.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  fatal: clone of '[email protected]:lieff/minimp3.git' into submodule path 'C:/Users/admin/AppData/Local/Temp/pip-req-build-dw1g94wp/minimp3' failed
  Failed to clone 'minimp3'. Retry scheduled
  Host key verification failed.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
  fatal: clone of '[email protected]:lieff/minimp3.git' into submodule path 'C:/Users/admin/AppData/Local/Temp/pip-req-build-dw1g94wp/minimp3' failed
  Failed to clone 'minimp3' a second time, aborting
ERROR: Command "git submodule update --init --recursive -q" failed with error code 1 in C:\Users\admin\AppData\Local\Temp\pip-req-build-dw1g94wp

[xavfernandez]

Hello, like the error says, pip simply calls git submodule update --init --recursive -q then git uses the submodule url provided by https://github.com/pyminimp3/pyminimp3/blob/7d0b3695a2ff768f7bfcb881b5e493faf76a1cfd/.gitmodules#L3

I don't think pip can avoid that ?

[cjerdonek]

The reporter also reported it in the package's tracker: https://github.com/pyminimp3/pyminimp3/issues/1

[pganssle]

What is the recommended solution here? Change the submodule to use the HTTPS link?

[cjerdonek]

That would be the easiest way to solve the reporter's issue I'm sure. GitHub's docs suggest here that https urls are more widely supported:

Cloning with HTTPS URLs (recommended)
The https:// clone URLs are available on all repositories, public and private. These URLs work everywhere--even if you are behind a firewall or proxy. In certain cases, if you'd rather use SSH, you might be able to use SSH over the HTTPS port.

I do wonder now though if there's anything pip can do to make this work. For example, I see this open issue ("Pip - doesn't pass proxy settings through to git/hg etc."): https://github.com/pypa/pip/issues/227

@0xf0f If you try to clone the ssh url coming from the submodule on the command-line manually yourself, what happens for you? Does it work -- are you prompted for anything?

You want to try this:

$ git clone [email protected]:lieff/minimp3.git

[0xf0f]

Cloning the submodule manually works with that command. Only thing I'm prompted for is my passphrase.
One thing I noticed is if you follow the link to the submodule from the repo's master branch you arrive at this page: https://github.com/lieff/minimp3/tree/66bf4ad047b471b6e3288f8c76a20737c078b9a3 which is an old version of the master branch. I'm not experienced enough to know why this is the case. Perhaps the submodule needs to be removed and readded with a different command?

[pradyunsg]

Only thing I'm prompted for is my passphrase.

This might be why the command fails -- pip does not forward stdin to the VCS backends (for a bunch of reasons including that it is non-deterministic and automation usually hangs if waiting on stdin input). That means prompt for the passphrase would not work.

---

Switching to https is a good workaround for this.

[cjerdonek]

Out of curiosity, now that you've typed your password, if you try again from scratch using pip, does it work?

which is an old version of the master branch.

This is because the parent repo is pointing to particular older commit. That's up to whoever maintains the parent repo.

@pradyunsg It might be possible to add support for interactive mode if some kind of --interactive option is present.

[pradyunsg]

@pradyunsg It might be possible to add support for interactive mode if some kind of --interactive option is present.

I'm not opposed to it but not super warm to the idea. I don't know if that's useful enough and I'm weary of adding more knobs to pip (#6221, #5204).

All that said, it'd be nice to have a dedicated issue for discussing this specific idea (if we don't already).

[0xf0f]

Out of curiosity, now that you've typed your password, if you try again from scratch using pip, does it work?

Unfortunately not. Though I do get different results between command prompt and git bash. In command prompt I get the same output as the original comment. In git bash, a small window titled 'OpenSSH' that looks like it was made with Tk (has the feather icon) pops up several times asking me to enter the passphrase. After that this happens:

admin@DESKTOP-K910G6S MINGW64 ~/Documents/Projects/clone_test
$ pip install git+https://github.com/pyminimp3/pyminimp3
Collecting git+https://github.com/pyminimp3/pyminimp3
  Cloning https://github.com/pyminimp3/pyminimp3 to c:\users\admin\appdata\local\temp\pip-req-build-8ofjjsva
  Running command git clone -q https://github.com/pyminimp3/pyminimp3 'C:\Users\admin\AppData\Local\Temp\pip-req-build-8ofjjsva'
  Running command git submodule update --init --recursive -q
  error: Server does not allow request for unadvertised object 66bf4ad047b471b6e3288f8c76a20737c078b9a3
  Fetched in submodule path 'minimp3', but it did not contain 66bf4ad047b471b6e3288f8c76a20737c078b9a3. Direct fetching of that commit failed.
ERROR: Command "git submodule update --init --recursive -q" failed with error code 1 in C:\Users\admin\AppData\Local\Temp\pip-req-build-8ofjjsva

[chrahunt]

Trying to pass-through settings to git for all variations of submodules (https with auth, https without auth, https with custom certificate, ssh) would be a lot of work. I'm inclined to say that if someone actually wants this implemented they can open a feature request for it.

For the simple case here - if a user doesn't want submodules requested with ssh, it is possible to use URL rewriting to force one way or another for all requests: git config --global url.https://github.com/.insteadOf [email protected]:

[pradyunsg]

Coming back around to this, if anyone still cares about this specific problem -- stdin for subprocesses is closed by pip, which results in VCS not showing prompts for passwords -- please file a new issue for discussion, linking to #6623 (this issue).

[pradyunsg]

I'm going to go ahead and close this issue due to a lack of activity/interest.