pip doesn't check out submodules when .gitmodules is present

[etc0de]

Environment

• pip version: 19.1.1
• Python version: 3.7.3
• OS: Fedora Linux

Description
pip doesn't appear to be checking out submodules when .gitmodules is present (git submodule init) and when installing from a git repo like this: pip3 install --user -U "sfxscan @ https://github.com/JonasT/sfxscan/archive/master.zip#". However, I have seen mentions that it is supposed to be initializing the submodules properly, so I'm wondering whether this is a bug? Is it related to this being a https URL maybe? Should this really make such a difference to a git url, when clearly .git and .gitmodules are easily present & detectable in that archive?

Expected behavior
pip initializes submodules and proceeds with install successfully

How to Reproduce

1. pip3 install --user -U "sfxscan @ https://github.com/JonasT/sfxscan/archive/master.zip#

Output

Collecting sfxscan@ https://github.com/JonasT/sfxscan/archive/master.zip# from https://github.com/JonasT/sfxscan/archive/master.zip
  Downloading https://github.com/JonasT/sfxscan/archive/master.zip
     - 133kB 7.8MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: sfxscan
  Building wheel for sfxscan (PEP 517) ... error
  ERROR: Complete output from command /usr/bin/python3 /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpnjnw8edv:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/sfxscan
  copying sfxscan/__init__.py -> build/lib.linux-x86_64-3.7/sfxscan
  running build_ext
  Traceback (most recent call last):
    File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
      main()
    File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 141, in build_wheel
      metadata_directory)
    File "/tmp/pip-build-env-ip56u3xo/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 192, in build_wheel
      self.run_setup()
    File "/tmp/pip-build-env-ip56u3xo/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 141, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 147, in <module>
      ] + cython_extensions(),
    File "/tmp/pip-build-env-ip56u3xo/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib64/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib64/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-ip56u3xo/overlay/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/lib64/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib64/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib64/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "setup.py", line 29, in run
      str(os.path.join(vendor_taglib_dir, "CMakeLists.txt"))
  RuntimeError: did you use git submodule init? got missing taglib CMakeLists.txt at: /tmp/pip-install-t6h5li8e/sfxscan/vendor/taglib/CMakeLists.txt
  ----------------------------------------
  ERROR: Failed building wheel for sfxscan
  Running setup.py clean for sfxscan
Failed to build sfxscan
ERROR: Could not build wheels for sfxscan which use PEP 517 and cannot be installed directl

[cjerdonek]

You're downloading the repo contents as a zip file though, rather than as a clone, right? Have you tried prefixing the URL with git+ as the docs say here? https://pip.pypa.io/en/stable/reference/pip_install/#git

[etc0de]

@cjerdonek I just tried the git prefix and that seems to work. However, just to give some feedback:

1. Some URL formats (pep 518 if I recall) only work with install_requires but not requirements.txt
2. Some URL formats aren't automatically downloaded unless put as dependency links, others are (this does affect git urls if I remember although I might have done something wrong back then, anyway that is why I avoided them so far)
3. Some URL formats have git submodule initialized, others don't
4. Some URLs can be put either in setup.py directly or requirements.txt via parsing (good!) or must be in pyproject.toml (Cython .pxd import dependencies, bad) since setup_requires is deprecated and broken

The URL situation is frustrating and messy. Wouldn't it be possible to also process zip URLs the same way (that is, initialize submodules after extraction when there's .git and .gitmodules around) to reduce this mess a little? I really think it would be a great idea to remove all these weird, as an outside user seemingly nonsensical differences, I can't tell you how much time I've wasted trying yet another URL format just to magically get some things to work and others to break...

[cjerdonek]

Wouldn't it be possible to also process zip URLs the same way (that is, initialize submodules after extraction when there's .git and .gitmodules around) to reduce this mess a little?

I wouldn't say it's "nonsensical" not to treat downloading a zip file like a Git repository. At least for me (and I think others), that would be pretty surprising behavior, as they are totally different animals. Like, this is what happens if I try to Git clone a URL like the one you posted. It doesn't work:

$ git clone https://github.com/pypa/pip/archive/master.zip
Cloning into 'master.zip'...
remote: Not Found
fatal: repository 'https://github.com/pypa/pip/archive/master.zip/' not found

Also, when I tried downloading the zip URL you gave, the extracted directory doesn't actually contain a .git directory, so I don't think one could treat it like a Git repo even if you wanted to.

1. Some URL formats have git submodule initialized, others don't

The rule is that if it's downloaded as a Git repository (e.g. with git clone), it will update the submodules. Since a zip download is treated as a simple directory and not as a Git repository, it won't do anything Git related.

[cjerdonek]

Should this really make such a difference to a git url, when clearly .git and .gitmodules are easily present & detectable in that archive?

Is the .git directory really present for you?

[etc0de]

Is the .git directory really present for you?

It's not (thought it was but I was wrong :open_mouth: ) making this admittedly a nonsensical ticket :grimacing: but I'm thinking, could pip maybe emit a warning when encountering a zip with .gitsubmodules in it? Because that file is present in it, and I think that could be a quite good general indicator that the current chosen way of installing won't work right

For what it's worth I'm sure all of these URL differences have a rule, just all of it coming together has required me to switch through lots of formats over the months. This ticket has basically been the one of the latest switch where I'd thought maybe it'd be worth filing something

[cjerdonek]

but I'm thinking, could pip maybe emit a warning when encountering a zip with .gitsubmodules in it?

Well, submodules aren't necessarily required even if they were present in the original repo. So it could result in warnings that are false alarms and would need to be ignored every time. You did wind up getting an error, so it's not like a warning was needed. :)

[etc0de]

yes because I happened to have put one in, but I'm not sure other projects will handle it as well. Personally I think false positives might be worth the trade-off but in any case, I should probably close this ticket since what I initially reported is not really fixable. thanks again for the help!

[cjerdonek]

Okay, thanks, and good luck with your project.