Stack: `stack clean` fails due to missing ghc-pkg (on latest from master)

[silky]

looks suspiciously related to https://github.com/commercialhaskell/stack/issues/4466

to repro

1. use stack from https://github.com/commercialhaskell/stack/commit/5144dd57bd78553f5051601295a2eb215d1db5a6
2. go into a project
3. run stack clean

error:

> stack clean
Executable named ghc-pkg not found on path: [...]

version

11:21 AM noon ∈ Moon-Tile (master*) ♫ stack --version
Version 1.10.0, Git revision 5144dd57bd78553f5051601295a2eb215d1db5a6 x86_64 hpack-0.31.1

[mihaimaruseac]

Confirmed that I can reproduce this

[qrilka]

It appears to be a bit of a clash between #4412 and #4268
To fix this properly I guess we need some good way to solve #4390 as after #4412 the most of the commands became more dependent on GHC being available

[dbaynard]

Fix is probably to change the behaviour of stack clean when GHC is not present. #4466 is probably a coincidence.

We don't have EnvConfig without GHC present, and stack clean currently relies on EnvConfig in order to identify the Cabal library present (there's a documentation PR about these precise details following a conversation with @snoyberg).

The sensible thing would be:

• if there's only one Cabal directory in dist then use that as the target (as now)
• if there are multiple Cabal directories, either
  
  
  
  • prompt the user for which to delete, or
  
  
  • prompt the user to install ghc, or
  
  
  • select a dist directory to clean (unexpected behaviour), or
  
  
  • download GHC (undesirable and unexpected behaviour), or
  
  
  • delete multiple dist directories (probably undesirable behaviour), or
  
  
  • present an error, and introduce flags to control behaviour (seems unnecessarily complicated).
  
  

I'd favour the prompts but that leads to issues in scripts.

For now, does running stack setup work around the issue?

[mihaimaruseac]

Prompts with a default and flags for the other would help the scripts.

[qrilka]

@dbaynard I don't understand why do you say that deleting the only one Cabal directory in dist is the way stack clean currently works, see e.g.:

qrilka@qdesktop ~/ws/h/stack-build-test $ ls -l $(dirname `stack path --dist-dir`)
итого 8
drwxr-xr-x 5 qrilka qrilka 4096 янв  7 21:18 Cabal-2.0.1.0
drwxr-xr-x 5 qrilka qrilka 4096 янв  7 21:17 Cabal-2.4.0.1
qrilka@qdesktop ~/ws/h/stack-build-test $ stack --resolver=lts-12.13 clean
qrilka@qdesktop ~/ws/h/stack-build-test $ ls -l $(dirname `stack path --dist-dir`)
итого 8
drwxr-xr-x 5 qrilka qrilka 4096 янв  7 21:18 Cabal-2.0.1.0
drwxr-xr-x 5 qrilka qrilka 4096 янв  7 21:17 Cabal-2.4.0.1

Currently stack clean uses for cleaning only Cabal version shipped with GHC, for that we could use global-hints.yaml and that's what I was asking you about in https://github.com/commercialhaskell/stack/pull/4268#issuecomment-451656732 - with that we won't need any user prompts or defaults (thought you could name values in global-hints.yaml as defaults of course). My main concern here is about allowing what is described in #4488 but after reading Michael's words in #stack-collaborators discussion it looks like a proper way to go. I've started a refactoring which will split setupEnv into 2 stages and the 1st one will result in SMActual getting constructed and that should be enough for at least stack clean

[dbaynard]

Also see #4385 — I thought that I had already merged it (hence some of my confusion). It changes how stack clean works. And I think it gets the documentation correct.

[dbaynard]

I don't understand why do you say that deleting the only one Cabal directory in dist is the way stack clean currently works

Hold on, I'm confused by your example — lts-12.13 uses Cabal 2.2 so that is exactly the output I'd expect from that command.

EDIT:

stack clean only deletes the dist dir corresponding to the selected resolver. But I've no idea whether anybody uses this functionality — in particular as it isn't documented — and so I think we can change the functionality to do something sensible.

> ls $(stack path --dist-dir)(:h)
Cabal-2.2.0.1  Cabal-2.4.0.1
> stack clean --resolver lts-12
Selected resolver: lts-12.26
> ls $(stack path --dist-dir)(:h)
Cabal-2.4.0.1
> stack build --resolver lts-12
Selected resolver: lts-12.26
…
> ls $(stack path --dist-dir)(:h)
Cabal-2.2.0.1  Cabal-2.4.0.1
> stack clean
> ls $(stack --resolver lts-12 path --dist-dir)(:h)
Selected resolver: lts-12.26
Cabal-2.2.0.1

[qrilka]

Regarding #4385 - I don't quite see what does it change, I only see it adding stack purge as a synonym for stack clean --full.
Regarding Cabal directories - my concern was about your first formulation of stack clean which appears a bit vague to me: you were saying that it deletes a Cabal directory in dist if there exist one such directory and my argument was more about what you are saying in this last comment - it deletes only directory corresponding to the selected GHC and it doesn't depend on number of Cabal directories in dist.
I'm OK with changing it to something sensible though I don't have good proposals what that could be.

[dbaynard]

my concern was about your first formulation of stack clean which appears a bit vague to me: you were saying that it deletes a Cabal directory in dist if there exist one such directory

Ah, I see my ambiguity, now.

if there's only one Cabal directory in dist then use that as the target (as now)

I was incorrect to say stack clean deletes a Cabal directory if there is only one. It only targets that single directory if the global (compiler) Cabal version matches — otherwise it does nothing.

Yes, I thought that if we can't know which Cabal directory to delete, and there is only one directory, we may wish to assume the user wishes to delete whichever directory is there.

it doesn't depend on number of Cabal directories in dist

Yes — this is the current behaviour, where we use ghc-pkg to identify the correct Cabal directory. Without ghc-pkg, this is not possible in general, but we can get close.

---

In light of this, it may be better not to assume stack clean by default should delete whatever is present, and we should implement a single solution.

I've started a refactoring which will split setupEnv into 2 stages and the 1st one will result in SMActual getting constructed and that should be enough for at least stack clean

Sounds good 👍

[snoyberg]

But I've no idea whether anybody uses this functionality — in particular as it isn't documented — and so I think we can change the functionality to do something sensible.

I think this is a great point @dbaynard. I'd be fine with massively simplifying the approach to clean to just getting rid of all of the dist directories, or perhaps even just wiping out the .stack-work directories entirely. I'm not aware of any reason why this should cause breakage. Worst case scenario: some corner case depending on undocumented behavior in Stack ends up having a longer recompilation period.

If someone sees something I don't, please let me know.

[dbaynard]

This is odd — I'm writing an integration test for #4385 (I've rebased on to master) and stack clean has changed. Specifically, it no longer identifies the same GHC variant as stack build.

In my test, when I run stack build it creates x86_64-linux-tinfo6 directories inside the .stack-work/dist and .stack-work/install directory. stack clean ought to delete the corresponding directory inside .stack-work/dist. However, instead it _adds_ a directory: .stack-work/install/x86_64-linux.

The clean command outputs this, at the end. It is using the different install directory.

2019-02-10 22:55:57.995962: [debug] Run process: $HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg init stack/test/integration/tests/3863-purge-command/files/.stack-work/install/x86_64-linux/2038573ca7245a0c03b75f394d0498bd6cd9ca70bb5b1d5833750d0911e12d48/8.6.3/pkgdb/
2019-02-10 22:55:58.026367: [debug] Process finished in 30ms: $HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg init stack/test/integration/tests/3863-purge-command/files/.stack-work/install/x86_64-linux/2038573ca7245a0c03b75f394d0498bd6cd9ca70bb5b1d5833750d0911e12d48/8.6.3/pkgdb/

I'm going to update that PR with the broken integration test.

[vanceism7]

Hey so just to clarify with everyone, should we actually change stack clean to do a full clean by default (i.e just delete the .stack-work dir), or are we preferring to keep the purge synonym and have stack clean do the "shallow" clean like it currently does?

[qrilka]

@dbaynard it looks like a side-effect of SkipDownloadCompiler. At the same time Michael's proposal is to just delete dist without bothering with Cabal/GHC versions. It's a bit of a breaking change but it's not clear what's the point of the current "fine-grained" cleaning and how any code could depend on that. And the question looks to be more about what's @vanceism7 is asking - what will be the default behavior of stack clean - only delete dist or just .stack-work altogether?

[dbaynard]

Keep the split functionality — it is useful to be able to delete artefacts (in dist) and keep outputs (in install).

So: delete the build artefacts (dist) with stack clean, I guess for all the cabal subdirectories there. Delete all .stack-work directories with stack purge.

How does that sound?

[snoyberg]

That sounds good to me. To confirm: that would mean that the clean command would never need to know anything about GHC or Cabal version, right?

[dbaynard]

Yes. It would need to know the .stack-work/dist directory paths for the project (for when people override .stack-work).

[qrilka]

It looks fine to me too, the main point was about not depending on GHC/Cabal.