stack build does unnecessary rebuild if more than one stack file

[alanz]

General summary/comments (optional)

Performing stack build twice in a row should be a noop the second time.

See also https://github.com/haskell/haskell-ide-engine/issues/1289#issuecomment-503025524

Steps to reproduce

$ git clone https://github.com/haskell/haskell-ide-engine.git
$ cd haskell-ide-engine
$ git submodule update --init --recursive
$ stack --stack-yaml=stack-8.6.5.yaml build

A build happens, which (eventually) finishes

$ stack --stack-yaml=stack-8.6.5.yaml build
$ stack --stack-yaml=stack-8.6.5.yaml build
$

No further build activity takes place.

Now build for GHC 8.6.4

$ stack --stack-yaml=stack-8.6.4.yaml build

The build eventually ends

Invoke the build again

$ stack --stack-yaml=stack-8.6.4.yaml build

Expected

The build is idempotent, i.e. no-ops the second and subsequent times, with each separate stack-xxx.yaml.

Actual

It rebuilds the submodules

And oddly enough, doing

$ stack --stack-yaml=stack-8.6.4.yaml bulld --verbose

generates a lot of output, but does not rebuild anything.

Stack version

$ stack --version
Version 2.1.1, Git revision f612ea85316bbc327a64e4ad8d9f0b150dc12d4b (7648 commits) x86_64 hpack-0.31.2

Method of installation

• Official binary, downloaded from stackage.org or fpcomplete's package repository, then stack upgrade

[qrilka]

@alanz why do you expect builds to be idemponent when you change GHC? Stack 2 uses so called implicit snapshots with reproducibility as one of the main ideas, some of the design is described in Build overview see especially Determine snapshot hash section

[alanz]

@qrilka I expect them to be idempotent because

a) they were so with stack 1.9.3
b) the .stack-work directory has separate areas under it for each compiler/snapshot combination.

I guess a possible workaround would be to pass in a different directory to be used as the stack workdir for each stack file we build against.

[qrilka]

Oh, probably I didn't read your report properly - the point is that after having done a build with GHC 8.6.5 subsequent build should be no-op disregarding intermediate builds with other compiler? For project packages the logic is a bit different - I'll try to refresh my understanding of it later.

[alanz]

Yes. I would expect

stack --stack-yaml=stack-8.6.4.yaml build
stack --stack-yaml=stack-8.6.5.yaml build
stack --stack-yaml=stack-8.6.4.yaml build

to be a no-op on the third one. It's not.

And as expected, each stack file uses a different GHC (via the appropriate resolver).

[snoyberg]

Can you include some of the messages which explain why it's rebuilding? My guess is this is related to moving the config cache into a user-global location, and the PR I just opened (#4898) is a step in fixing this, but would need some more modifications.

[alanz]

Here is the output, after first getting rid of .stack-work

$ stack --stack-yaml=stack-8.6.5.yaml build
[builds as expected]
$ stack --stack-yaml=stack-8.6.5.yaml build
[no-op]
$ stack --stack-yaml=stack-8.6.4.yaml build
[builds as expected]
$ stack --stack-yaml=stack-8.6.4.yaml build
HaRe-0.8.4.1: unregistering (missing dependencies: cabal-helper, hie-plugin-api)
cabal-helper-0.9.0.0: unregistering (local file changes: /home/alanz/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.5/lib/ghc-8.6.5/include/ghcversion.h)
ghc-mod-5.9.0.0: unregistering (missing dependencies: cabal-helper, ghc-mod-core)
ghc-mod-core-5.9.0.0: unregistering (missing dependencies: cabal-helper, ghc-project-types)
ghc-project-types-5.9.0.0: unregistering (missing dependencies: cabal-helper)
haskell-ide-engine-0.10.0.0: unregistering (missing dependencies: HaRe, cabal-helper, ghc-mod, ghc-mod-core, hie-plugin-api)
z-haskell-ide-engine-z-hie-test-utils-0.10.0.0: unregistering (Dependency being unregistered: ghc-mod-core-5.9.0.0)
hie-plugin-api-0.10.0.0: unregistering (missing dependencies: ghc-mod-core, ghc-project-types)
cabal-helper      > build (lib + exe)
cabal-helper      > copy/register
ghc-project-types > build (lib)
Progress 1/7: ghc-project-types^C

[snoyberg]

Thanks. One further request: would it be possible to work this down to a more minimal repro, instead of needing to build all of haskell-ide-engine?

[qrilka]

@alanz it looks like the module https://github.com/alanz/cabal-helper/blob/eafed5e8c1d82b8daa35775b52361132f2e70261/src/CabalHelper/Compiletime/Compat/Environment.hs adds ghcversion.h from GHC as a dependent file and as it depends on GHC version it triggers the recompilation.
I'm not quite sure why that header file is needed though...

[power-fungus]

I did some minifications today: 5 files, <30 LOC, https://github.com/power-fungus/haskell-ide-engine/tree/stack-hie-issues

To reproduce:

$ stack --version
Version 2.1.3, Git revision 636e3a759d51127df2b62f90772def126cdf6d1f (7735 commits) x86_64 hpack-0.31.2
$ git clean -Xdf
$ stack --stack-yaml=stack-8.6.5.yaml build
[builds as expected]
$ stack --stack-yaml=stack-8.6.4.yaml build
[builds as expected]
$ stack --stack-yaml=stack-8.6.5.yaml build
[unexpected rebuild]

This issue has nothing to do with cabal-helper. The issue arises when CPP is used inside a module that is used in package with custom-setup

I would be pleased if you could fix this issue :sweat_smile: