Stack: hakyll dependency on nightly is not cached

[st3ll1s]

With the resolver set to resolver: nightly-2018-03-01, a package depending on hakyll always rebuilds haddock-library, pandoc, pandoc-citeproc and hakyll. This does not happen if I set the resolver to resolver: lts-10.7. The rebuilds cause significant slowdowns.

Both pandoc and pandoc-citeproc seem to be extra-deps of hakyll. Maybe this is related?

Steps to reproduce

1. In stack.yaml, set resolver: nightly-2018-03-01.
2. Run command stack build.
3. Run command stack build again.

package.yaml:

name: repro

dependencies:
- base >= 4.7 && < 5
- hakyll

library:
  source-dirs: src

Expected

The second stack build should not build anything again.

Actual

haddock-library, pandoc, pandoc-citeproc and hakyll are rebuilt.

Output of the second stack build --verbose: https://gist.github.com/st3ll1s/3ef40d7d3e5513a31ef031bab129deff

Stack version

root@9aa4322411fe ~# stack --version
Version 1.6.5, Git revision 24ab0d6ff07f28276e082c3ce74dfdeb1a2ca9e9 (5514 commits) x86_64 hpack-0.20.0

Method of installation

curl -sSL https://get.haskellstack.org/ | sh

[st3ll1s]

I think this may be caused by haddock-library which is a transitive dependency of hakyll. Just depending on haddock-library suffices to cause rebuilds on every stack build invocation.

[st3ll1s]

I found a single commit in Haddock that introduces the change: https://github.com/haskell/haddock/commit/3bf3d4c95c3b32ff350e127ee087c85d5bbb24c6.
Before to this commit the rebuilds do not happen.
The latest LTS does not include the above commit which explains why this problem did not occur before.

Steps to reproduce

Run stack build and then stack build again.

package.yaml as before.

name: repro

dependencies:
- base >= 4.7 && < 5
- haddock-library

library:
  source-dirs: src

Build is cached

stack.yaml:

resolver: nightly-2018-03-07

extra-deps:
  - git: https://github.com/haskell/haddock.git
    commit: 8064353244842c8a5b04d99cd5e6bb6430a9b51c
    subdirs:
      - haddock-library

Build is not cached

stack.yaml:

resolver: nightly-2018-03-07

extra-deps:
  - git: https://github.com/haskell/haddock.git
    commit: 3bf3d4c95c3b32ff350e127ee087c85d5bbb24c6
    subdirs:
      - haddock-library

[mgsloan]

@st3ll1s Thanks for digging into this. Looks like a bug at some level indeed.

My guess is that there is an assumption somewhere that these sub libraries do not alias existing library names like attoparsec. Not sure.

I encourage anyone affected by this to dig into stack's code and see if they can work out exactly what's going on and ideally fix it in the form of a PR.

[bergus]

I experience the same issue with Hakyll. It causes build times of about 15 min, which is unbearable. How can I help to get this fixed? I tried running stack with the verbose flag, but the output didn't indicate anything special.

[k-bx]

@bergus seems like you were right in pinpointing the commit, it defines library attoparsec which probably somehow causes the rebuilds. I guess there are two options: ask haddock-library maintainers to change this back, or to improve stack to be able to handle this use-case. Not sure if I'll have time to look into this in detail anytime soon, unfortunately.

[st3ll1s]

I've tried changing haddock-library's cabal file (https://github.com/st3ll1s/haddock/commit/b3912d70f74b0693f1ea8cffb8f547b1303ef325) as @gbaz suggested, but when building haddock-library with stack I still see rebuilds.

stack build --verbose prints Ignoring package z-haddock-library-z-attoparsec-vendored, from (InstalledTo Local, [...]), due to it being unknown to the resolver / extra-deps a few times.

[k-bx]

@st3ll1s can you do something similar to undoing the https://github.com/haskell/haddock/commit/3bf3d4c95c3b32ff350e127ee087c85d5bbb24c6 , e.g. don't introduce any named libraries at all please?

[st3ll1s]

@k-bx See https://github.com/st3ll1s/haddock/commit/b29547eb7feb757e1dadb03fbedd506c3cdaba3c. This seems to work, no unnecessary rebuilds anymore!

[gbaz]

It appears that internal library support was added to stack already (cf https://github.com/commercialhaskell/stack/issues/3361 and the resolution in https://github.com/commercialhaskell/stack/pull/3430). So I think the problem here is just the rebuild logic, and was likely present from the start. I could get a minimal repro going with the internal-libraries repo given in https://github.com/commercialhaskell/stack/issues/3926

Indeed I see a similar message in the verbose logs:

2018-03-24 20:14:30.161561: [debug] Ignoring package z-internal-libraries-z-internal, from (InstalledTo Local,"/..."), due to it being unknown to the resolver / extra-deps.

So it appears that the package is discarded from the check for known/needed packages, and thus forced to always be rebuilt. Since it is rebuilt, this in turn precipitates everything downstream of it being rebuilt.

Since this affects all packages with internal libraries, then it seems like it would be good to fix. Until then, all such packages will have forced rebuilds.

By the way, if someone wants to tackle it, from reviewing 3430, my best guess at where the problem is would be: https://github.com/commercialhaskell/stack/blob/d1c40aef50f1100cad78615d6f12cfdc58c52f7e/src/Stack/Package.hs#L624

The comment seems relevant. (edit: just checked master and confirmed the comment is still there)

[mihaimaruseac]

I think I'm going to give it a try and see if I can submit a PR fixing this sometimes this week.

[mihaimaruseac]

I've made a candidate PR but it's still work in progress, I'll revisit it after the weekend. At the moment, the internal-libraries above passes, but a hakyll-based project still recompiles haddock-library due to

2018-04-05 21:18:26.547698: [debug] Ignoring package z-haddock-library-z-attoparsec, from (InstalledTo Snap,"/home/mihai/.stack/snapshots/x86_64-linux-tinfo6/lts-11.2/8.2.2/pkgdb/"), due to it being unknown to the resolver / extra-deps.

Most likely, I'll need a similar change to the one for InstalledTo Local, but that seems to conflict with changes for #292 so I have to think more about this.

I'll add more to this after the weekend, don't think I can get some more work done until then.

[mihaimaruseac]

Got some time today, I think all is done now and #3955 solves this.

[snoyberg]

Solved by #3955, thanks @mihaimaruseac!