Stack: Support: universal stack configuration for use with nix

[Fresheyeball]

Some Haskellers don't use stack for various reasons, in my case it's because my work (and myself) prefer managing dependencies with custom Nix code to Stackage. That's all well and good, except there is a growing tool chain built around stack, that do not involve Stackage. As such I can't take advantage of these new tools, simply because they expect stack and stackwork to be present. For example

• https://github.com/ndmitchell/weeder
• https://github.com/haskell-tools/haskell-tools
• https://github.com/phoityne/phoityne-vscode
• https://github.com/commercialhaskell/intero

I see no reason these tools and Stackage should be coupled. But they are, simply because they use _other_ nice things that stack provides that are unrelated to Stackage. IE no stack.yml, as the deps are managed elseware. If cabal can find the deps, I believe stack could as well.

Therefore, it would be very very nice, if stack offered some support for plain old cabal projects, and cabal + nix projects. Enough to permit the generation of stackwork and other api's used by tooling, while perhaps not supporting api's that are Stackage specific.

[Fresheyeball]

This may be close to doable with https://www.google.com/amp/s/www.fpcomplete.com/blog/2017/07/stacks-new-extensible-snapshots%3fhs_amp=true
If it can resolve to the same packages as cabal, when stack.yml is not found.

[mgsloan]

Interesting idea! One approach might be to write a converter from cabal project files to stack.yaml.

Note that you can already create a stack.yaml when you have cabal files, by doing stack init

It is highly unlikely that stack will have an operating mode that does not use stack.yaml, but you are free to have stack.yaml be a generated file.

[Fresheyeball]

@mgsloan just to be clear, a converter to convert a cabal project to a stack project is not what I am after. There are reasons to use other dependencies management systems (like nix), which can have their own build logic, incompatible with become a project based on Stackage, and extra-deps. Do you think it would be possible to have a stack.yml for this use case along these lines?

resolver: local-nix-shell

or

resolver: cabal-sandbox

Basically a near empty file, that tells stack, that this is a nix + cabal project, and to use whatever it finds in the nix-shell as the resolved dependencies.

Then tools built on stack, should work for a cabal + nix project, with minimal changes. Specifically without having to do a big conversion, and can use custom nix code to manage dependencies.

[snoyberg]

This may sound like something simple, but the devil's in the details. As a simple example: what does it mean to run stack build in a project that says resolver: cabal-sandbox if one of the dependencies is unavailable? Is Stack responsible for building that dependency, or does it error out? Where does Stack register local packages to?

As I understand it, the reason tools would elect to support only Stack is because it avoids the complexity of including many different possible code paths in the tool itself. This request will shift that burden into Stack itself, and almost certainly result in something that breaks often and makes no one happy.

If there's a well defined set of behaviors you're looking for, that's easier to start with. But "should work" doesn't say nearly enough.

(As a side point: you refer to Stack and Stackage almost interchangeably in this issue, though they are quite distinct projects with different purposes. It may help any discussion here if you clarify terminology.)

[23Skidoo]

resolver: cabal-sandbox

Just a quick note: I'd recommend against developing new features based on cabal-install's sandboxes, since they're going to be deprecated relatively soon and eventually removed in favour of new-build.

[Fresheyeball]

@snoyberg Sure, the details matter. Since I'm not a collaborator on the stack project, I was hoping for creative insights about how this might be accomplished. I had not used stack and Stackage interchangeably, in-fact, my whole point rests on that stack and Stackage are different projects. If there is a point in my previous writing where you felt I was confusing the two, would you mind pointing it out? I don't see it.

@23Skidoo Agreed. Was just posting some possibilities.
@snoyberg @23Skidoo For the moment I will stick to the local-nix-shell use case since it is the one I am most familiar with.

Right now I work in a project that is based on nix + cabal. We use non-trivial nix code to manage Haskell dependencies, and those dependencies are made available to cabal inside a nix-shell. I would like to use the tools _based on_ stack with this project, even if I don't use stack directly. Since stack is separate from Stackage, I was thinking stack could have a resolver that picks up the dependencies provided by the nix-shell.

I agree "should work" doesn't say enough. But that is largely because I am not intimately involved with the stack project, and don't know what kind of approach you might be amenable to, or what caveats there might be. So in response to the caveats you did bring:

what does it mean to run stack build in a project that says resolver: cabal-sandbox if one of the dependencies is unavailable? Is Stack responsible for building that dependency, or does it error out?

I would expect it to throw an error. But that is only my expectation, if you have a better idea, I am comfortable.

Where does Stack register local packages to?

I don't know what you mean by this. Or why it's a problem for this use-case. I think we could just use where it registers normally no? If not, we could ask the user somehow where they would like them placed.

[snoyberg]

You can probably already achieve something like what you're looking for with:

• resolver: ghc-X.Y.Z
• system-ghc: true
• And some playing around with the extra package database flags if your packages aren't showing up in the global database

Perhaps if you get something working you'd be able to send a doc PR so that others looking for the same thing could get a quick idea of how to do it.

[Fresheyeball]

@snoyberg can you give more guidance as to what you mean by

extra package database flags if your packages aren't showing up in the global database

Remember, I have limited stack exposure. What is a package database in this context?

[snoyberg]

A package database is not a Stack concept. It's how GHC keeps track of installed packages. You'll need some way to tell Stack about your packages. The easiest is if the packages are already in your global database, which is what I believe Nix is doing.

[Fresheyeball]

@snoyberg how do I tell stack to use the global package database? Is there some docs you can point at me?

[snoyberg]

It uses the global package database by default.

I'm not familiar enough with Nix to do any of this myself. Could you give me an example of how you would, for example, start a Nix shell that has GHC 8.2.2 (or 8.0.2, doesn't really matter) and some non-standard package like async available, and I can perhaps demonstrate how to use that with Stack?

[Fresheyeball]

@snoyberg absolutely! I will put up a repo as soon as I can.

[Fresheyeball]

@snoyberg sorry for the delay, was busy with holidays. Here is a repo: https://gitlab.com/fresheyeball/nix-stack/tree/master

git clone https://gitlab.com/fresheyeball/nix-stack/tree/master
cd nix-stack
nix-shell
cabal build

[snoyberg]

That repo doesn't work with Stack _or_ cabal, since you have the wrong version of base and no version of compactable. In fact, they give exactly the same error (in their own languages):

$ cabal build
Resolving dependencies...
Warning: solver failed to find a solution:
Could not resolve dependencies:
trying: nix-stack-0.1.0.0 (user goal)
next goal: base (dependency of nix-stack-0.1.0.0)
rejecting: base-4.10.1.0/installed-4.1... (conflict: nix-stack => base>=4.9 &&
<4.10)
fail (backjumping, conflict set: base, nix-stack)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: nix-stack, base
Trying configure anyway.
Configuring nix-stack-0.1.0.0...
cabal: Encountered missing dependencies:
base ==4.9.*, compactable -any

$ stack build

Error: While constructing the build plan, the following exceptions were encountered:

In the dependencies for nix-stack-0.1.0.0:
    base-4.10.1.0 from stack configuration does not match >=4.9 && <4.10 (latest matching version is 4.9.1.0)
    compactable must match -any, but the stack configuration has no specified version (latest matching version is 0.1.0.2)
needed since nix-stack is a build target.

Some potential ways to resolve this:

  * Recommended action: try adding the following to your extra-deps in /Users/michael/Desktop/nix-stack/stack.yaml:

- base-4.9.1.0
- compactable-0.1.0.2

  * Set 'allow-newer: true' to ignore all version constraints and build anyway.

  * You may also want to try using the 'stack solver' command.

Plan construction failed.

So I can't really test anything at all here. But in theory, this stack.yaml might work:

resolver: ghc-8.2.2
system-ghc: true
install-ghc: false

[Fresheyeball]

Right. It doesn't build outside the nix-shell you have to follow the commands I posted. It builds fine with cabal.

I put in that stack.yaml file and got the following error:

$ stack build
Error: While constructing the build plan, the following exceptions were encountered:

In the dependencies for nix-stack-0.1.0.0:
    compactable must match -any, but the stack configuration has no specified version (latest applicable is 0.1.0.2)
needed since nix-stack-0.1.0.0 is a build target.

Recommended action: try adding the following to your extra-deps in /home/isaac/Projects/nix-stack/stack.yaml:
- compactable-0.1.0.2

You may also want to try the 'stack solver' command
Plan construction failed.

[Fresheyeball]

After running stack solver --update-config it builds fine in the nix-shell this is an awesome start! Is there a way this can work without this step?

[mgsloan]

Did you try this?

Recommended action: try adding the following to your extra-deps in /home/isaac/Projects/nix-stack/stack.yaml:
- compactable-0.1.0.2

[Fresheyeball]

@mgsloan that seems to fix it too, but kind of defeats the whole: can work with any cabal + nix project thing. Would it be reasonable to add a stack.yaml field like:

auto-solve: true

that does the stack solver --update-config with an in memory stack.yaml and then builds with that? Something functionally similar, but non-ugly?

[mgsloan]

Running the solver each time would be really slow. Not sure what the best design for this would be. Maybe resolver: installed-packages? Or perhaps installed or pkg-db

The installed-packages resolver would disable installing dependencies and instead use the versions from the packages on GHC_PACKAGE_PATH. Targets which are not in the DB would be installed from latest hackage version (like when you ask for a hackage package that is not in a snapshot).

So, in this configuration, there would be no snapshot DB, just the user's DB stack + the local project DB

[snoyberg]

you have to follow the commands I posted

That's what I'm reporting to you: your commands fail the way I described. Also, the URL you include is not valid. I had to clone from https://gitlab.com/fresheyeball/nix-stack.

@mgsloan I suspect that, with a working nix-shell configuration, this may work, so I don't want to get into deeper discussions on workaround in Stack until we've got a working test case.

[Fresheyeball]

@snoyberg hu. You are right, my url was wrong. But I just retested. It works on my end. The error you got, looks like you are running cabal outside the nix-shell. This test case works for me so far. Not sure what the issue is on your end. Let me get a screen cap of it working.

@mgsloan that sounds good to me so far.

[Fresheyeball]

I did need to make a change to the repo. You where right @snoyberg. Still I got it working. Also it turns out that stack solver is not needed, once everything is right in nix.

[gbaz]

What do people think about letting an _arbitrary_ packagedb (as specified by a path) serve as a resolver? (Think of it as an alternate format for custom snapshots). This could let users specify e.g. sandbox dbs as well.

[gbaz]

It might also be worth teaching stack about plan.json files that come from new-build, not least because they'll be a good stable way to back out solver results as a longer-term solution to #3453

[snoyberg]

For arbitrary package DBs: I think the (quite underdocumented) extra-package-dbs setting may address this already. I won't have a chance to test this myself with sandboxes any time this week, but if someone else can take a stab at it and report it, that would be awesome.

I haven't heard about plan.json yet, that's interesting. Do you have a link by any chance?

I know you didn't say it explicitly, but on my mind now: I'm guessing that while supporting sandboxes will be easy, supporting nix-style databases will be non-trivial due to intentionally breaking Stack's "one package one version" policy. Is that a fair assessment?

[gbaz]

I didn't know about extra-package-dbs before. This is very useful -- just tested it both with a local packagedb and a sandbox packagedb and it works correctly.

So if a package builds with cabal, there's an essentially "generic" stack.yaml that means it can also build with stack, with no specification of dependencies (or even solving) necessary.

system-ghc: true
install-ghc: false
resolver: ghc-8.2.1
extra-package-dbs: 
# use this for the user's local packagedb
# - /Users/user/.ghc/x86_64-darwin-8.2.1/package.conf.d
# use this for a sandbox db
 - .cabal-sandbox/x86_64-osx-ghc-8.2.1-packages.conf.d/

This should let people integrate with tools that require stack like intero or weeder without having to maintain a full stack.yaml.

[gbaz]

plan.json isn't fully documented yet (the current docs have a TODO for it) but it is generated by new-build commands and lives in dist-newstyle/cache/plan.json as a json serialization of the build plan. there's a recent blog post describing some new tools that are starting to make use of it: http://oleg.fi/gists/posts/2018-01-08-haskell-package-qa.html

(and yes, the above doesn't necessarily work with the nix-like store of new-build -- one stab at a general approach [that could be leveraged more broadly] might be to have a little tool that leveraged the data from a plan.json to create a derived "slice" or "view" packagedb registered out of just those components in the transitive graph as picked out by the plan).

[nh2]

I can confirm that this works.

I have managed to build stack itself (commit 94d302c) using stack2nix on this commit to turn stack.yaml into default.nix, and the following stack-nix-package-db.yaml:

resolver: ghc-8.2.2 # Change this to your GHC version here!
system-ghc: true
install-ghc: false

I could build using

nix-shell -A stack.env --run 'stack --no-nix --stack-yaml stack-nix-package-db.yaml build'

stack.env makes nix build a package DB containing stack's dependencies; with the given arguments, stack (I tried 1.7.1) can use that without any modifications needed.

This approach makes it possible to use stack's CLI instead of cabal or Setup.hs.

Further explanations:

• The --no-nix is just to make sure that if you're on NixOS, where stack detects that and automaticallys starts nix-shell; we don't want that here because it'd be starting a nix-shell from nix-shell, which doesn't work (it would result in not picking up the package DB we've provided in the outer nix-shell). We should ~probably~ maybe change stack's auto-detection of NixOS to not do that when the IN_NIX_SHELL env var is set (but not sure we should do it though because stack uses STACK_IN_NIX_SHELL, which probably means that whoever wrote it wanted to _not_ use IN_NIX_SHELL).
  
  
  
  • The reason it would rebuild on NixOS without --no-nix is that in https://github.com/commercialhaskell/stack/blob/47d73f3ec6271884fd713ed447250c4449d5839e/src/Stack/Nix.hs#L71-L74 we're passing some --extra-lib-dirs flags and similar, which makes stack think that it needs to rebuild from source with those flags added instead of using the prepared package DB we provided.
  
  

[nh2]

I believe that this satisfies one of the two things asked for in the issue description:

it would be very very nice, if stack offered some support for [...], and cabal + nix projects

@Fresheyeball Do you agree?

[snoyberg]

No response to last question, and original question seems addressed, closing