Stack: extra-deps doesn't work with git and submodules

[neil-da]

Using stack-2.1.1 on Mac, if I create a stack file:

resolver: nightly-2019-05-20
packages:
- .

extra-deps:
- git: https://github.com/alanz/haskell-ide-engine.git
  commit: 9bc0b0146c5775d36cb6fb036fa529c71ca1df8e

I get:

Cloning 9bc0b0146c5775d36cb6fb036fa529c71ca1df8e from https://github.com/alanz/haskell-ide-engine.git
Received ExitFailure 128 when running
Raw command: /usr/local/bin/git submodule foreach --recursive "git -c core.autocrlf=false archive --prefix=$displaypath/ -o bar.tar HEAD && if [ -f bar.tar ]; then tar --force-local -Af /private/var/folders/mz/5qkm8w1x5w79fykl7_s9v14m0000gn/T/with-repo-archive50160/foo.tar bar.tar ; fi"
Run from: /private/var/folders/mz/5qkm8w1x5w79fykl7_s9v14m0000gn/T/with-repo50160/cloned
Standard output:

Entering 'submodules/HaRe'
Standard error:

tar: Option --force-local is not supported
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
fatal: run_command returned non-zero status for submodules/HaRe
.

It seems the extra-deps fail because it has submodules - see the submodules directory.

[neil-da]

For people who reach this issue, the workaround I found is:

- url: https://github.com/alanz/haskell-ide-engine/archive/9bc0b0146c5775d36cb6fb036fa529c71ca1df8e.tar.gz

[incertia]

The incriminating line appears to be here. Maybe we can turn this off on OSX/Linux as the commit introducing it seems to be targeted towards Windows?

[dbaynard]

@qrilka would be the best person to talk to — I suspect we may need to improve our submodules integration tests. Thanks, @incertia — would you be willing to implement/test a PR?

[qrilka]

I think @psibi probably could give a bit more help here and better tests are always a good thing to have for sure

[psibi]

@incertia Let us know if you want to implement a PR, it should be easy to do. I went ahead and implemented it.

Another workaround without changing the stack configuration is to have GNU Tar in your $PATH. It can be installed via brew using:

brew install gnu-tar

[esjmb]

I think there is a design issue/team process here that needs resolution before applying a PR.

1. Is the stack team committed to introducing a dependency on gnu-tar? If not, then reverse the changes and find another way. If yes, then...

2. Are dependencies global or platform specific? If dependencies are to be platform specific, then the question becomes one of how these are documented and how they can be introduced and tested so that they do not break other platforms. If global then again, documentation, alongside global change to installation scripts etc.

I see no reason why a global dependency on gnu-tar, the case in point, should not be decided on, but the team seems to be operating in this case on a platform specific basis. At the very least, the introduction of a new dependency of this kind should have red flag team level visibility so that members with platform specific expertise can verify changes do not break platforms.

Please take my comments in the spirit given, which is one of huge respect for the work the stack team does.

[esjmb]

ps.

@incertia Let us know if you want to implement a PR, it should be easy to do.

Another workaround without changing the stack configuration is to have GNU Tar in your $PATH. It can be installed via brew using:

brew install gnu-tar

an addendum to this. One needs to add export PATH="/usr/local/opt/gnu-tar/libexec/gnubin:$PATH" in order to give gtar preference over standard mac tar.

[neil-da]

Given stack is beautifully hermetic and doesn't rely on much external state, I think that adding a dependency on something that doesn't come out of the box, and which stack doesn't grab on its own, would be most unfortunate. It feels like against the ethos of Stack.

[psibi]

@neil-da I agree. It was an oversight on my part. I have opened a PR to address this: #4874

[yairchu]

For people who reach this issue, the workaround I found is:

- url: https://github.com/alanz/haskell-ide-engine/archive/9bc0b0146c5775d36cb6fb036fa529c71ca1df8e.tar.gz

Thanks for this work-around! I was bummed when this problem hit me but finding this saved my mood :)

[yairchu]

@ndmitchell's work around (using url for commit .tar.gz from GitHub) doesn't fully solve the problem if your dependency actually needs its submodules.

The missing piece for the workaround in this case is to make the Setup.py bring the submodule if it isn't there (which it is with nix and should be future versions of stack).

An example for this part of the workaround can be found here:

• https://github.com/lamdu/bindings-freetype-gl/blob/master/Setup.hs
• https://github.com/lamdu/bindings-freetype-gl/blob/master/prepare_submodule.sh

[yairchu]

For people who reach this issue, the workaround I found is:

- url: https://github.com/alanz/haskell-ide-engine/archive/9bc0b0146c5775d36cb6fb036fa529c71ca1df8e.tar.gz

I found an alternative workaround, which is also compatible with Stack 1 (url: doesn't parse there) -

But this one is only usable if the repo is on github.

Instead of:

git: https://github.com/alanz/haskell-ide-engine.git

Use:

github: alanz/haskell-ide-engine

[snoyberg]

Fixed by #4879, closing.