Stack: Backpack for Stack

[ezyang]

Hello all,

I want to open up the discussion about how Backpack can be integrated with Stack; in particular, I'd be willing to work on and submit a patchset adding support for it in Stack. But before I do that, I want to hash out some general architectural concerns. The specification for the GHC and Cabal support for Backpack is here: https://github.com/ezyang/ghc-proposals/blob/backpack/proposals/0000-backpack.rst ; please read it for more motivation on why you might like to support it.

The design process over two years lead us to a design where Backpack requires the package manager (Stack) to know how to _instantiate_ packages that have required signatures in them (I spent the better part of six months trying to figure out how to do it all in GHC; if you are interested in the sordid details I can let you know). Things seem to work most smoothly if:

1. The package manager has a _component_ oriented build model; that is, the unit of work in the install plan is a component rather than a package. (I don't believe Stack is architected in this way, although issues like #1406 suggest it might be a good architecture for Stack anyway.) See also https://github.com/ezyang/ghc-proposals/blob/master/proposals/0000-componentized-cabal.rst for the proposal that added per-component configure to Cabal
2. The package manager knows enough to be able to finalize the package description (i.e., resolve all of its conditionals, getting a PackageDescription); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.

I'll note that _private_ use of Backpack (i.e., the public library does not have any signatures) should be supported out-of-the-box, as there is no change to the outward-facing API of Setup scripts in this case (Stack will need to understand how to register multiple libraries produced by a Setup script, however). In the meantime, public libraries which have signatures would not be supported.

One of the major applications of Backpack I would like to see in the near future is the Backpack-ification of GHC's boot libraries to solve the string problem. Backpack would make it easy to write modules which are parametric over a particular desired string representation; the original package names would continue to export String-based interfaces, but versions using ByteString etc could be made available, and the availability of a package abstracted over holes means that an end-user could specify their own. Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.

Please let me know if you have any questions or concerns; I'd be happy to answer them.

CC @mgsloan, @snoyberg, @Blaisorblade , @borsboom , @harendra-kumar

[aljce]

This would be _awesome_.

[Blaisorblade]

@ezyang First let me thank you for all your work on this.

TL;DR. Does stack need this shipped tomorrow, in a month or in a year?

Before I can look into details and ask more substantive questions: is there a timeline for Backpack being merged/released/used, and an estimate how much work was required on cabal-install?

Absent full support for Backpack, Stack would at least have to be able to handle traditional-looking packages which internally depended on libraries that made use of Backpack. Maybe this would be easy.

I agree this is probably essential. Having feature parity also matters—having to choose between stack and Backpack, or stack and sane strings, would be pretty bad.

[borsboom]

@mgsloan, I think you know the most about the parts of Stack that @ezyang is talking about. Once you're back from BM, can you weigh in?

@ezyang: how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code.

[ezyang]

Does stack need this shipped tomorrow, in a month or in a year?

Definitely not tomorrow or a month. If all goes well Backpack will get merged to GHC and Cabal, and people can start experimenting with it, but there were certainly be bugs and I won't claim that anyone should start using it for real immediately. Backpack'ing base is at least blocked on teaching GHC's build system how to deal, which will take some time and I am not planning for 8.2. On the other hand, base is probably the most plausible, widely used package that will get Backpack'ed, because (1) the potential upside is so great, and (2) it closely tracks GHC versions, so we don't have to worry about supporting old GHCs.

how is cabal-install going to handle Backpack? Both cabal-install and Stack use the Cabal library to actually build packages, so there may be opportunity to share much of the code.

There are some key library functions for configuring and instantiating packages, and cabal-install (new-build only; the legacy codepath doesn't support it) just calls out to them. At present, there are some things that are duplicated because I couldn't figure out an API that could be reused in both cases, but this is definitely a case where I can try harder.

The two key functions have these signatures:

type ConfiguredComponentMap =
        (Map PackageName (ComponentId, PackageId), -- libraries
         Map String ComponentId)                   -- executables
toConfiguredComponent
    :: PackageDescription
    -> ComponentId
    -> Map PackageName (ComponentId, PackageId) -- external
    -> ConfiguredComponentMap
    -> Component
    -> ConfiguredComponent

type LinkedComponentMap = Map ComponentId (UnitId, ModuleShape)
toLinkedComponent
    :: Verbosity
    -> PackageId
    -> LinkedComponentMap
    -> ConfiguredComponent
    -> LogProgress LinkedComponent

and a LinkedComponent has all the information to actually build with Backpack (the model is that every source package produces a configured component, which is mix-in linked into a linked component; then you go ahead and instantiate all the linked components getting the final graph of build products you need to build.)

[chrisdone]

The package manager knows enough to be able to finalize the package description (i.e., resolve all of its conditionals, getting a PackageDescription); we can't do Backpack instantiation until this finalization has occurred. I don't know where in the Stack codebase one would have to engineer this.

Stack does this. We use Cabal to parse the .cabal file but then resolve the GenericPackageDescription manually. Here is an example: https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L765 The whole of Stack.Package is pretty much dedicated to resolving package info. See also https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L198

[ezyang]

Great. Another question I have (since I am looking into this) is how difficult it would be to get Stack to support a per-component build model. I'm hoping that I can take anywhere we talk about 'Package' and actually it's talking about a 'Component', but I am not familiar enough with the code to say if this is the case or not.

[mgsloan]

@ezyang Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:

1) Dependencies vary based on whether --enable-tests or --enable-benchmarks is passed to Cabal, so this needs to be figured out on a package level
2) Avoiding package reconfiguration - best to configure all the components we want at once

And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'.

[ezyang]

OK. So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package (another thing where I'm not sure how well Stack will deal with this change.)

[ezyang]

I've been fishing around the codebase, and here are some initial architectural thoughts on what Backpack needs.

• Stack defines a simplified Package data type, which it reads out the contents of PackageDescription into. Backpack introduces some new fields, most notably mixins, which need to be preserved so that Backpack can figure out how packages are being instantiated. At the moment, Package doesn't really preserve any information like exposed modules which we would need to do this. It seems the most convenient way of keeping all the information Backpack needs is to keep PackageDescription verbatim in Package? This will also make it easier to call into Cabal to do mix-in linking.

• Once we have all the packages, we need to do a "mix-in" linking step to figure out exactly how packages have been instantiated. For example, if I have build-depends: regex-indef, str-string, I need to figure out that regex-indef's signature named Str was instantiated with the Str from str-string.

  It is an interesting question on how to integrate this with Stack. Stack's model is that it goes ahead and reads in all of the local source packages, but then lazily reads in the external package information using the accessor function created by withLoadPackage. The idea is that if a package is already installed, then we can avoid traversing its dependency graph. This should integrate reasonably well with Backpack: if a package is installed, we can read off the module shape (that's the metadata you need to mix-in link) from the installed package info; if it is not, we can compute it when we installPackage.

• Mix-in linking is not enough: after we figure out how everything is linked, we have to instantiate, creating a build task for every configuration by which a package was instantiated. In Stack, a build Plan consists of some number of Tasks to be done. I'm not sure how dependencies between tasks are computed, but each task is uniquely identified by the PackageIdentifier it provides (taskProvides). A Task roughly corresponds to a configure/build pair on Cabal. So, we need to modify taskProvides to be a bit more flexible: rather than only build a PackageIdentifier, it builds both a package identifier and an instantiation (to be defined shortly). Let's call such an identifier a UnitIdentifier, following the terminology I've used in Cabal and my thesis. Roughly speaking, these data structures should look like this:

  data UnitIdentifier = UnitIdentifier PackageIdentifier [(ModuleName, Module)]
  data Module = Module UnitIdentifier ModuleName
  

If Stack was modified in this way, would you take the patch? How would you like the design to be different? What is unclear?

[mgsloan]

Hey @ezyang , sorry for letting this languish, got caught up in other things. Passing this over to @snoyberg

[snoyberg]

I'm just coming up to speed on this now Edward. It seems relevant to work I'm already doing upgrading to Cabal 2.0. I'm also in favor of pushing Stack towards a component base build system. My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version.

I realize this issue is a bit dated now, and perhaps you have different thoughts. Maybe we could jump on a text chat and synch up?

[ezyang]

@snoyberg Sure thing; ping me on IRC some time? As far as I am concerned, the thinking on this ticket hasn't changed much (although maybe Stack's internals have!)

My only concern with this is making sure we retain support for building packages with older versions of Cabal, as all existing snapshots include such an older library version.

Yeah, so in cabal new-build, this is done by treating these packages as non-per-component, single packages in the plan. It's a little bit of complexity but not too bad, and keeps BC.

[snoyberg]

Just chatted with @ezyang on IRC about this. Here's my summary:

1. We first need to get Cabal 2.0 support merged into Stack, which is on me (ETA: 2 weeks)
2. Once that's done, we will switch Stack to a per-component build plan. This is something that I'm very excited about in fact, see https://github.com/haskell/cabal/issues/2802
3. The biggest wrinkle is continuing support for pre-2.0 Cabals, which will be necessary for any LTS or Nightly release before GHC 8.2. Basic plan of attack is:
   
   
   
   1. Every build plan is calculated on a per-component basis
   
   
   2. If we're using Cabal 2.0 or later, we will also build on a per-component basis (separate configure with a separate dist dir for each and every component)
   
   
   3. If we're using pre-2.0 and there are no cycles in dependencies (which can happen in test suites and benchmarks), then we convert the per-component plan to a simple per-package plan and build
   
   
   4. If we're using pre-2.0 and there _are_ cycles, we will convert into a staged per-package plan: any packages that have cycles will be split into lib/exe builds and full builds (including the tests and benchmarks). This is essentially the same as we have today, but can hopefully be streamlined by relying on the per-component plan logic. (Example use case: running stack test inside the bytestring package.)
   
   

I'll ping this issue with updates when Cabal 2.0 is merged in

[snoyberg]

Pinging appropriately: Cabal 2.0 support has just been merged to master :)

[mgsloan]

One thing for switching stack to a per-component build plan will also be adding support for internal libraries - https://github.com/commercialhaskell/stack/issues/3361

Also support for foreign libraries, not sure if that's as relevant - https://github.com/commercialhaskell/stack/issues/3364

[centromere]

What work needs to be done to finish adding Backpack support to stack?

[snoyberg]

There's still quite a bit of work to be done, see my comment above https://github.com/commercialhaskell/stack/issues/2540#issuecomment-319570811. All that we've done so far is get support for Cabal 2.0 into Stack.

[ezyang]

I'm working on per-component build for Stack in https://github.com/ezyang/stack/tree/per-component-build

It doesn't build yet, but I've finally gotten it to typecheck to ConstructPlan, which means the real work can begin :)

[mgsloan]

Great, thanks @ezyang ! I skimmed it, and it looks good so far. Feel free to open a PR before its ready, as soon as you are ready to start getting review feedback.

[mikeplus64]

How's this come/coming along?

[ezyang]

Very, very slowly XD

[LeanderK]

any progress? 🙂
I don't want to be rude, it's just that I would really like to delve deeper into backpack. But not being able to integrate it into stack-projects is pretty discouraging (all of my projects use stack and there are also a lot out there 😕). I think it's a bit of a chicken or the egg problem, what comes first? Support or usage?

[stites]

I noticed that this ticket has a call for help via "A year into Backpack." Since this is needed for the hasktorch project to get into the LTSs, I've posted a bounty for this issue.

[matthew-piziak]

From @ezyang's blog.

In Stack issue #2540 I volunteered to implement Backpack support for Stack. However, over the past year, it has become abundantly clear that I don't actually have enough spare time to implement this myself. Looking for brave souls to delve into this; and I am happy to advise about the Backpack aspects.

Hey @ezyang, it was nice of you to volunteer, but when you realize you don't have time for a ticket it's good etiquette to say so in the ticket thread. That way people know it's unassigned and up for grabs. Don't lick the cookie—pass the plate!

You've done great work in characterizing this issue so far and I don't want to diminish that. But as they say, the last responsibility of a maintainer is to find a replacement, and it's good to do that in a centralized way. Many developers are on tenterhooks by this feature, so it's important that its status—viz. is somebody working on it / does it need a maintainer—is clearly advertised.

[mboes]

@matthew-piziak Cookie licking sounds pretty incongruous to this particular situation. @ezyang did a tremendous amount of work to actually get Backpack merged into GHC, and also on this particular ticket. This task isn't assigned to anyone and @ezyang makes it clear in his blog post that he'd be happy to see someone else pick this up.

[matthew-piziak]

@mboes You're right, I realized now that I had a poor understanding of the ticket's history. I apologize for the misleading language in my comment. In any case I am happy that this issue's status has been clarified.

[dbaynard]

Since @ezyang's blog post last week, I've been investigating what still needs doing — essentially https://github.com/commercialhaskell/stack/issues/2540#issuecomment-354624333 and https://github.com/commercialhaskell/stack/issues/2540#issuecomment-319570811, which are

1. to migrate to a per-component build instead of just a per-package build, and
2. to implement support on top of that for old and new packages, with or without backpack.

In order to do 1, there is substantial refactoring required. Some of that is indeed in progress now (e.g. pantry).

I suggest the following: if you wish to take on this issue, pop a comment here. The stack team can help mentor somebody through the implementation. For my part, I'd happily take it on — but I can't make that commitment right now.

And do continue to comment if you want to support the backpack changes.

[ezyang]

Thanks everyone for the interest. As I mentioned in my blog post, I'm happy to advise on Backpack bits. You can find me on #ghc on Freenode if you have questions.

[telser]

@dbaynard I am willing to work on this, but would definitely need some help/mentorship in the "where do I start" phase.

[snoyberg]

@telser That's great to hear! I think we're getting close to a PR to do the refactoring of the source map/build plan stuff, which should open the doors significantly for doing this work. I'll be sure to update this issue when it's ready.

[telser]

@snoyberg Awesome that the refactoring is getting close. Thanks for the update. Is there a branch I should follow the progress of, or anything else to get me up to speed on this?

[snoyberg]

@qrilka has been doing the work on the new-build-plan branch (https://github.com/commercialhaskell/stack/tree/new-build-plan). Probably the build-overview.md doc on that branch is the best explanation of what we're going to be doing.

[dbaynard]

Quick update: #3922 has been resolved

[flip111]

I'm wondering doesn't it make more sense to try to land backpack support in stack 2.0 and include it in in https://github.com/commercialhaskell/stack/blob/new-build-plan/doc/build-overview.md ?

I'm just considering this because backpack support could be a big refactor and cause some architecture changes??

Or is this taking too much at once and possibly delaying stack 2.0 by too much? Maybe backpack support would be better of in a stack 3.0 version.

Just by observing the discussion now backpack support seems to be stuck on the new build process. Which is fine in the sense that maybe this is the best way to go to wait for this. Just that backpack support would be a long way off.

Second observation is that the work already done by ezyang on the build per component branch https://github.com/ezyang/stack/tree/per-component-build seems to be superseeded by the work on stack 2.0

Sorry if this was already obvious from the thread. Just trying to get a clear picture of the state of backpack support in stack as it is now.

[mbj]

BTW: Where to donate currency to support this feature? I cannot donate my time currently.

[flip111]

@mbj one bounty was already opened here https://www.bountysource.com/issues/37368931-backpack-for-stack but i think you can use other websites as well and create a new bounty

by the way the bounty is now $395

[snoyberg]

@flip111 I think this issue is now just waiting for someone to take it on, and getting it into Stack 2.0 sounds good to me. @qrilka's changes to the build plan work have been merged, so there should be no open blockers for this.

[dbaynard]

@ezyang It would be really helpful to have integration tests for backpack's functionality — what would you recommend?

And to be clear: what specification does stack need to support? Various sources imply the thesis (which I've skimmed but no more) or the proposal; none is explicit. The GHC wiki seems quite out of date (there's a reference to next.hackage.haskell.org, which is down). I thought there may be some integration tests for cabal-install, but found none.

I've been trying to put some tests together (see https://github.com/dbaynard/stack/compare/b8d8859...dbaynard:fb61d68) and stack seems to be mostly working… of the examples in the proposal, only the following internal library is not (Cabal is throwing an error checkBuildTargets: multiple copies enabled: an error I managed to resolve for another case).

library concat-bytestring
    build-depends: str-bytestring, concat-indef
    reexported-modules:
        -- Concat from concat-indef is instantiated
        -- with Str from str-bytestring.  We can
        -- reexport it under a qualified name for
        -- more convenient use.
        Concat as Concat.ByteString

Incidentally in terms of installing hackage packages, hasktorch-signatures-types built fine with master, as did primitive-containers; unpacked-containers built but led to a registration error, Invalid package ID: "unpacked-containers-0-LfzPMTpGfdOB6DZgwnNLSt+9O4xGsIBaOn7knYCYMH2U".

[ezyang]

Hi @dbaynard

It would be really helpful to have integration tests for backpack's functionality — what would you recommend?

There are a bunch of tests for Backpack in Cabal's repository at cabal-testsuite/PackageTests/Backpack/ which are worth testing against. You'll have to know a little bit about cabal.project files to interpret them (check the https://www.haskell.org/cabal/users-guide/nix-local-build-overview.html for more details), let me know if you have questions.

And to be clear: what specification does stack need to support? Various sources imply the thesis (which I've skimmed but no more) or the proposal; none is explicit.

Use the thesis (the proposal is still not updated, as the wiki page says), and also the code in Cabal as the reference implementation (it lives mostly in Cabal/Distribution/Backpack). It would be preferable if Stack could use this code directly, but I understand if there are technical issues doing so.

The GHC wiki seems quite out of date (there's a reference to next.hackage.haskell.org, which is down).

I updated the wiki page. Hackage proper now supports Backpack using packages, so there's no need for next.hackage.

The errors are a bit more interesting, I'll have to look at the diff to see what's going on.

[ezyang]

the following internal library is not (Cabal is throwing an error checkBuildTargets: multiple copies enabled: an error I managed to resolve for another case).

Do you know how ./Setup is being invoked to build this package? I think that would give useful information. I suspect the problem is that Stack is specifying a list of components to be built to Setup, and this doesn't work with Backpack (instead, you have to use per-component builds, where you call Setup configure individually for each individual component to be built.)

unpacked-containers built but led to a registration error, Invalid package ID: "unpacked-containers-0-LfzPMTpGfdOB6DZgwnNLSt+9O4xGsIBaOn7knYCYMH2U".

This is during a call to ./Setup install, right? Is it possible to get the command line that Cabal is being invoked with here too?

[dbaynard]

Thanks, @ezyang.

I'll have to look at the diff to see what's going on.

I haven't pushed the stanza I put in the comment, which doesn't build.

I suspect the problem is that Stack is specifying a list of components to be built to Setup, and this doesn't work with Backpack (instead, you have to use per-component builds, where you call Setup configure individually for each individual component to be built.)

That is the case. I was quite surprised the other stanzas built — the confirmation on the correct behaviour is helpful.

unpacked-containers built but led to a registration error, Invalid package ID: "unpacked-containers-0-LfzPMTpGfdOB6DZgwnNLSt+9O4xGsIBaOn7knYCYMH2U".

This is during a call to ./Setup install, right? Is it possible to get the command line that Cabal is being invoked with here too?

The call which fails is here (full log below — the ./Setup used is Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3; it's just the simple defaultMain.). I assumed the failure relates to the changes to identifiers in the spec — though I was just as interested that the other installs succeeded.

$HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --user --no-user-package-db --package-db $HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb describe --simple-output unpacked-containers --expand-pkgroot
Invalid package ID: "unpacked-containers-0-LfzPMTpGfdOB6DZgwnNLSt+9O4xGsIBaOn7knYCYMH2U"

[debug] Run process within /tmp/stack16592/unpacked-containers-0/: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 configure --with-ghc=$HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc --with-ghc-pkg=$HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --with-ghcjs=$HOME/.local/bin/ghcjs --with-ghcjs-pkg=$HOME/.local/bin/ghcjs-pkg --user --package-db=clear --package-db=global --package-db=$HOME/.stack/snapshots/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb --package-db=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb --libdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/lib --bindir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/bin --datadir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/share --libexecdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/libexec --sysconfdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/etc --docdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/doc/unpacked-containers-0 --htmldir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/doc/unpacked-containers-0 --haddockdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/doc/unpacked-containers-0 --dependency=base=base-4.12.0.0 --dependency=data-default-class=data-default-class-0.1.2.0-FeIQ5tLoVZBHMSgrT9zptQ --dependency=deepseq=deepseq-1.4.4.0 --dependency=ghc-prim=ghc-prim-0.5.3
[debug] Process finished in 1315ms: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 configure --with-ghc=$HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc --with-ghc-pkg=$HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --with-ghcjs=$HOME/.local/bin/ghcjs --with-ghcjs-pkg=$HOME/.local/bin/ghcjs-pkg --user --package-db=clear --package-db=global --package-db=$HOME/.stack/snapshots/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb --package-db=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb --libdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/lib --bindir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/bin --datadir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/share --libexecdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/libexec --sysconfdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/etc --docdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/doc/unpacked-containers-0 --htmldir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/doc/unpacked-containers-0 --haddockdir=$HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/doc/unpacked-containers-0 --dependency=base=base-4.12.0.0 --dependency=data-default-class=data-default-class-0.1.2.0-FeIQ5tLoVZBHMSgrT9zptQ --dependency=deepseq=deepseq-1.4.4.0 --dependency=ghc-prim=ghc-prim-0.5.3
[debug] Encoding /tmp/stack16592/unpacked-containers-0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/stack-config-cache
[debug] Finished writing /tmp/stack16592/unpacked-containers-0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/stack-config-cache
[debug] Encoding /tmp/stack16592/unpacked-containers-0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/stack-cabal-mod
[debug] Finished writing /tmp/stack16592/unpacked-containers-0/.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/stack-cabal-mod
[info] unpacked-containers-0: build
[debug] Run process within /tmp/stack16592/unpacked-containers-0/: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 build --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"
[debug] Process finished in 13483ms: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 build --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"
[info] unpacked-containers-0: copy/register
[debug] Run process within /tmp/stack16592/unpacked-containers-0/: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 copy
[debug] Process finished in 163ms: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 copy
[debug] Run process within /tmp/stack16592/unpacked-containers-0/: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 register
[debug] Process finished in 813ms: $HOME/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 register
[debug] Run process: $HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --user --no-user-package-db --package-db $HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb describe --simple-output z-unpacked-containers-z-example --expand-pkgroot
[debug] Process finished in 59ms: $HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --user --no-user-package-db --package-db $HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb describe --simple-output z-unpacked-containers-z-example --expand-pkgroot
[debug] Run process: $HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --user --no-user-package-db --package-db $HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb describe --simple-output z-unpacked-containers-z-utils --expand-pkgroot
[debug] Process finished in 46ms: $HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --user --no-user-package-db --package-db $HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb describe --simple-output z-unpacked-containers-z-utils --expand-pkgroot
[debug] Run process: $HOME/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/ghc-pkg --user --no-user-package-db --package-db $HOME/Projects/stack/test/integration/tests/backpack/files/.stack-work/install/x86_64-linux-tinfo6/lts-13.2/8.6.3/pkgdb describe --simple-output unpacked-containers --expand-pkgroot
Invalid package ID: "unpacked-containers-0-LfzPMTpGfdOB6DZgwnNLSt+9O4xGsIBaOn7knYCYMH2U"

[ezyang]

Errr... is the Invalid package ID coming from the ghc-pkg invocation, or from Stack's parsing of the described package? I find it a bit hard to believe that ghc-pkg would report this error

[dbaynard]

I’m fairly sure it's stack. Yes, I shouldn't have described that as the “call which fails” as it could well be something after the registration is successful (and I haven't had the chance to dig in to it today).

[ezyang]

My guess is Stack is either choking on the 0 version number, or the + at the end.

[junjihashimoto]

@dbaynard your test of backpack passes on latest stack(2.2.0).
https://github.com/junjihashimoto/stack/commit/7e054f1c7d0980cf4fa70e6d2ea28defcda2c767

> stack --version
Version 2.2.0, Git revision f6258124cff9a7e92bcb5704164a70e149080e88 (dirty) (7597 commits) x86_64 hpack-0.31.1
>  stack build --flag stack:integration-tests stack --interleaved-output --exec "stack-integration-test -m backpack"
...
Running integration test 1/1: backpack
Success!
Successful tests:
- backpack

No failures!

But hasktorch's backpack outputs following errors.
https://github.com/hasktorch/hasktorch/pull/149

> stack build
...
Building all executables for `hasktorch-signatures' once. After a successful build of all of them, only specified executables will be rebuilt.
hasktorch-signatures        > configure (lib + internal-lib + exe)
hasktorch-signatures        > Configuring hasktorch-signatures-0.0.1.0...
hasktorch-signatures        > Error:
hasktorch-signatures        >     The following packages are broken because other packages they depend on are missing. These broken packages must be rebuilt before they can be used.
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+4wgymxdvbu48ub
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+Eini2ZdVHrv2yF
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+CeHBBWsdvNA1KE
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+JJWdgvcZNIYFde
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+4armw0d584q7vA
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+5JnLirPLHIOFSv
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+6n2x9a9EtLxGQi
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+LKZrGARs75HJPQ
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+5OyVmYeSdjB19F
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+IuZq5bMJvKTARY
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+9VQdccrNwqrLP9
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+4SCFSbKW2FYbD6
hasktorch-signatures        > planned package hasktorch-signatures-0.0.1.0 is broken due to missing package hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ+HukoqrlfWvm16y
hasktorch-signatures        >

--  While building package hasktorch-signatures-0.0.1.0 using:
      /home/junji-hashimoto/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.3 --builddir=.stack-work/dist/x86_64-linux/Cabal-2.4.0.1 configure --user --pacs
    Process exited with code: ExitFailure 1
Progress 8/10

[junjihashimoto]

In the error message, hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ's suffix is e.g. '+4wgymxdvbu48ub'.
In stack-work-directory, hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ's suffix is nothing.

$ ls .stack-work/install/x86_64-linux/db46aa9101c1f1fd35ba0f672aed08fa5e46f333436ca8483af9a0f90eed1f80/8.6.3/lib/x86_64-linux-ghc-8.6.3/|sort
hasktorch-ffi-th-0.0.1.0-9XO1QH55lfdHHeg428QwbD
hasktorch-ffi-thc-0.0.1.0-DyyEz3HipUL6TgeBuvKgDT
hasktorch-signatures-partial-0.0.1.0-FkbKtqZSXzwLiEkfE3RTVZ
hasktorch-signatures-support-0.0.1.0-HfWEGeSexxdBfMKWUSmxtV
hasktorch-signatures-types-0.0.1.0-G77PA53TJGA4pbD8hzrJqN
hasktorch-types-th-0.0.1.0-7hWvCSDXMBf79Ao3AY0KpP
hasktorch-types-thc-0.0.1.0-LTZn9ma8xFK5EycRVu8DQq
libHShasktorch-ffi-th-0.0.1.0-9XO1QH55lfdHHeg428QwbD-ghc8.6.3.so
libHShasktorch-ffi-thc-0.0.1.0-DyyEz3HipUL6TgeBuvKgDT-ghc8.6.3.so
libHShasktorch-types-th-0.0.1.0-7hWvCSDXMBf79Ao3AY0KpP-ghc8.6.3.so
libHShasktorch-types-thc-0.0.1.0-LTZn9ma8xFK5EycRVu8DQq-ghc8.6.3.so

The message' source is here.
https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Backpack/Configure.hs#L185-L198

When we use cabal, there are some inplace-directories.

$ ls hasktorch/dist-newstyle/build/x86_64-linux/ghc-8.4.4/hasktorch-signatures-partial-0.0.1.0/ |sort
build
cache
hasktorch-signatures-partial-0.0.1.0-inplace+3C4ISEtx5O5IuP45RidCSq
hasktorch-signatures-partial-0.0.1.0-inplace+63Pdb5gxFi42UnHTpXlUBk
hasktorch-signatures-partial-0.0.1.0-inplace+81iwCEcH3fHBhs0YpwUJ75
hasktorch-signatures-partial-0.0.1.0-inplace+9Am4HhOMR2X97nt56Cvcan
hasktorch-signatures-partial-0.0.1.0-inplace+AXOLuCJNRQD6nMHhhE0kiO
hasktorch-signatures-partial-0.0.1.0-inplace+CCjaLHTcBkGBn3bBpvQ2cN
hasktorch-signatures-partial-0.0.1.0-inplace+EQByKRIIjtj7Vm0YxuwKfG
hasktorch-signatures-partial-0.0.1.0-inplace+FedmswJnxai5Lz8FjYS6dX
hasktorch-signatures-partial-0.0.1.0-inplace+FwahM6uC1x29H12NMJZDy2
hasktorch-signatures-partial-0.0.1.0-inplace+HflA5lA4jhx4mCpPnZFHgE
hasktorch-signatures-partial-0.0.1.0-inplace+JUrUdFmdkgiJAX9OyJaz3e
hasktorch-signatures-partial-0.0.1.0-inplace+JW7CKjZPiQK5bZaWyQuCDq
hasktorch-signatures-partial-0.0.1.0-inplace+KIhA5jmaNkd22UhV6buuU9
package.conf.inplace
setup-config

[junjihashimoto]

So I think we need some rebuilding processes.
Is the code of cabal's rebuild here?

https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectBuilding.hs#L607-L618

[dbaynard]

@dbaynard your test of backpack passes on latest stack

These tests should be considered work in progress. Thanks for identifying an issue! I'm not able to answer your questions, but I will be able to take a look, soon.

[ezyang]

Let me know if you want me to take a look. There isn't anything obvious that pops out from the error message, besides, "Sounds like Stack isn't compiling enough instantiations of the signatures"

[junjihashimoto]

@ezyang Thx! If you take a look, it would be a great help.
If you think the looking is not worth to proceed with this issue, you do not need to look at it.
Maybe it is better to put some tests in stack's test.
The code outputing above erros is https://github.com/hasktorch/hasktorch/pull/149.
If you want any codes, logs and images(docker), I can provide you.

[TomLippincott]

@ezyang @junjihashimoto I'm getting the same errors: let me know if I can end-user-test any potential solutions.

[haitlahcen]

I don't know if its related but I faced the same issue (Invalid package ID) on a backpacked project and fixed it by adding + in the parser.

[cartazio]

this seems like a pretty egregious bug. why isn't it fixed?

[jkachmar]

@cartazio For what it's worth, backpack is currently not fully supported by any build tools outside of cabal-install. In light of this I wouldn't consider the lack of support for it within stack to be a particularly "egregious" bug, but rather a sign that there is limited interest in supporting this functionality beyond cabal-install.

For reference, please note the following tracking issues showing that support for backpack is not a particularly high priority elsewhere in the Haskell build tool ecosystem:

nixpkgs
rules_haskell
haskell.nix

[cartazio]

while i can empathize, i dont sympathize.
This is a bug in those tools, and perhaps they just haven't been motivated enough to take care of them.

[cartazio]

if they support ghc, this is a bug. If they just support haskell 2010, fine. but this means any post Haskell 2010 feature support is underspecified.

Do these tools support extensions/changes relative to Haskell 2010? What reference implementation is guiding them? I think you're saying "not ghc/cabal, but something more nebulous"

[jkachmar]

if they support ghc, this is a bug

I suppose my point here is that we've gone several years with all of the alternative build tools to cabal-install _not_ supporting this feature, which (to me) means that regardless of whether it's considered to be a bug upstream it's certainly not a pressing problem for consumers of these tools.

Do these tools support extensions/changes relative to Haskell 2010? What reference implementation is guiding them?

My point was more that there doesn't seem to be much pressure on any build tools to support backpack, and there doesn't seem to be much negative backlash from the community for not having solved them (that rules_haskell issue is nearly two years old at this point).

To be fair, this is one of those tricky things in the space of open source community management. If the compiler implements a feature, but not enough people care to use it (or push for its adoption), is it really a bug not to support that?

EDIT:
This hadn't occurred to me when I wrote this up initially, but I think it's also important to consider this from the perspective that others may not consider a missing feature to be synonymous with a bug.

In the nixpkgs issue, you can see that the maintainer has even marked this functionality as an "enhancement".

[flip111]

it's certainly not a pressing problem for consumers of these tools

Most of the projects can move forward without backpack. Just that it offers really nice features which people like to use. "Not a pressing problem" doesn't justify the attention this issue got and the nearly $500 bounty on it

[cartazio]

I’m starting some work in progress evaluations on how / if folks such as
backpack can dramatically improve the experience of both maintaining
vector and making it easier for users to write their own factor families.
It’s still up In the air if I can get it to work the way I want.

But if it pays for itself in the ways I hope, the longest I’m going to sit
on such a release to allow other tools to catchup is 6 months.

I’m pretty confident that if I get all the pieces lined up that It can
essentially just be a minor version bump of vector. But if so, I’ll have to
give y’all a grace period to fix this bug. I guess this is your 6 month
notice. Since waiting to be certain it pays off will just drag this ticket
out further.

On Mon, Feb 3, 2020 at 1:14 PM flip111 [e-mail] wrote:

it's certainly not a pressing problem for consumers of these tools

Most of the projects can move forward without backpack. Just that it
offers really nice features which people like to use. "Not a pressing
problem" doesn't justify the attention this issue got and the nearly $500
bounty on it

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2540?email_source=notifications&email_token=AAABBQQ2GUDBLLN35GVIGNTRBBNJJA5CNFSM4CN45YHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKU22VA#issuecomment-581545300,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABBQSBFUPIVGEOSBJA5ODRBBNJJANCNFSM4CN45YHA
.

[cartazio]

to be clear: (and a stack contributor nicely pm'd me on IRC to help me clarify what i mean)
1) i'm the maintainer for vector
2) i'm exploring ways to make vector better for everyone (even users who just assume the api is gonna be the same always)
3) one direction that seems the most humane (vs commit war crimes of CPP or TH) is evaluating if backpack can support evolving vector while reducing the current material code footprint of the current vector code base (the Boxed/Storable/Unboxed vector apis are mostly the same/ share the same code, but with type specializations for improved inference and friendly UX for newcomers).

many packages depend on vector directly or indirectly. If vector starts using backpack, this impacts users of tools that dont support it.

If backpack is an effective tool to support evolving vector aggressively while maintaing super stable user interfaces and much more friendly "write your own new vector instances " workflow for users who want, idk, memmory mapped or gpu or tape-drive fusioned vector data types, i want to be able to do this. Or I want YOU to be able to this.

Many people use stack as a personal choice, and I dont want stack users or vector users to suffer. Accordingly it seems pretty important for backpack to be supported should this design mix pan out. Otherwise any library using vector directly or indirectly will not be buildable with stack.

[cartazio]

on a meta note: i was concerned about how certain folks would react to my input on that ticket, i'm disappointed to be exactly correct about the type of abuse certain people hurl on the twitters

[flip111]

Summarized and paraphrased information so far.

---

Using components (parts of a package): allows for an architecturally sound implementation of Backpack, as the process of instantiating requirements of a Backpack only makes sense for libraries, rather than an entire package (an executable and test suite cannot be instantiated in any meaningful sense).

---

Interaction with Backpack. Backpack needs to perform mix-in linking on components, and then a further expansion step to instantiate components. Mix-in linking occurs as we are expanding the package graph into a component graph, while instantiation happens as a separate step after expansion.

At the moment stack does not have a graph for components. "mix-in" relates to which types are used for a package from which other package. See: https://github.com/commercialhaskell/stack/issues/2540#issuecomment-285612143

---

There are some key library functions for configuring and instantiating packages, and new-build just calls out to them. The two key functions toConfiguredComponent and toLinkedComponent

and a LinkedComponent has all the information to actually build with Backpack (the model is that every source package produces a configured component, which is mix-in linked into a linked component; then you go ahead and instantiate all the linked components getting the final graph of build products you need to build.)

---

how difficult it would be to get Stack to support a per-component build model?

Could be tricky! The current code is more package centric than component centric. This is for a variety of reasons. Off the top of my head:

1. Dependencies vary based on whether --enable-tests or --enable-benchmarks is passed to Cabal, so this needs to be figured out on a package level
2. Avoiding package reconfiguration - best to configure all the components we want at once
   And no doubt, more. So, no, when the stack code is talking about a 'Package' it's definitely not talking about a 'Component'.

Open question: Does stack know how to register multiple libraries produced by a setup script? --> If it can then mechanisms there could be re-used to do components (as one library would be one component)

---

So perhaps a reasonable approach to start with is to assume that any package which exposes a Backpack library, ONLY contains that library, so that if I need to instantiate a package multiple times, this would only involve creating duplicate copies of the package

An approach to start getting a proof of concept? Maybe better to have real components to get "architecturally sound implementation".

---

Best plan for implementation:

1. See point 3 in https://github.com/commercialhaskell/stack/issues/2540#issuecomment-319570811
2. Re-use types from https://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/Distribution-Backpack.html
3. Re-use or take inspirations from tests https://github.com/haskell/cabal/tree/master/cabal-testsuite/PackageTests/Backpack and https://github.com/dbaynard/stack/compare/b8d8859...dbaynard:fb61d68
4. Take inspiration from latest attempt https://github.com/commercialhaskell/stack/compare/master...ezyang:per-component-build It (used to) typecheck ConstructPlan

---

Feel free to correct me :)

@ezyang what did you mean with "register" in your opening post?

[flip111]

@snoyberg can you explain how far this file has the type of component that is needed to support backpack ?? https://github.com/commercialhaskell/stack/commits/master/src/Stack/Types/NamedComponent.hs Is this in principle the same "component" but might need some enhancement to store backpack specific information ?

EDIT: also is it ok for stack to use types like https://github.com/haskell/cabal/blob/1a31242dfe1f0376051057370e25cc846e319ea8/Cabal/Distribution/Types/Component.hs

I'm not sure what parts of cabal can being pulled in or not

[Bodigrim]

Well, for what it's worth, I've been recently discussing with colleagues switching back from stack to cabal as our main build tool because of Backpack support.

Of course, there is not much pressure at the moment, because there are few Backpacked packages, because package authors avoid Backpack, because it cannot be built by one of two main Haskell build tools. It is a pity that packages cannot explore features, merged into GHC four versions ago. Changing this status quo would be a massive shift IMO.

On the other hand, this is open source: people are free to pop in, fix an issue and get away with praise/glory/bounty. It would be unfair to blame maintainers of stack for not catering everyone's needs immediately. Backpack support is no small thing.

[evanrelf]

I agree with @Bodigrim. Very few Haskell packages take advantage of Backpack because a major build tool (stack) doesn't support it. Which means there's little motivation for stack developers to add support. It's a negative feedback loop.

[sjakobi]

Note that there are several GHC bugs related to backpack, although I don't see anything too bad.

[flip111]

@ezyang are the OpenUnitId and DefUnitId from Distribution.Backpack serving the same function as the proposed types

data UnitIdentifier = UnitIdentifier PackageIdentifier [(ModuleName, Module)]
data Module = Module UnitIdentifier ModuleName

My guess would be that these "units" are not the same kind of units, because one includes PackageIdentifier and the other doesn't. Confirmation would be appreciated.

[flip111]

@ezyang

* rather than only build a `PackageIdentifier`, it builds both a package identifier and an instantiation (to be defined shortly). Let's call such an identifier a `UnitIdentifier`

Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ?

[flip111]

On how to blend backpack code with stack code ...

Searching in cabal's codebase for toLinkedComponent brings up the buildComponent function. Even if this function was on top level and being exported on first glance it seems there is too much cabal specific stuff in there.

On the one hand for stack's purposes it seems useful to do the expansion of tasks to build that is needed for backpack as late as possible. So after tasks are made expand them in yet even more tasks.
https://github.com/commercialhaskell/stack/blob/a7a6c35436157d3756a079e83303e5548640e980/src/Stack/Build/ConstructPlan.hs#L206 This prevents having to modify code earlier and thus having to carry changes through all code paths.

On the other hand toConfiguredComponent and toLinkedComponent need information that will only be available earlier than toTask so this information needs to be carried through anyway.

First step seems to be to figure out where the arguments from toConfiguredComponent come from and if they can be constructed (have data constructors exported) and where in the stack constructPlan this information is available. After that the same for toLinkedComponent.

After that see if the return value of toConfiguredComponent correspond to a type Stack has, and wether this return type (LogProgress ConfiguredComponent) can be embedded in a Stack type.

[bbarker]

I agree with @Bodigrim. Very few Haskell packages take advantage of Backpack because a major build tool (stack) doesn't support it. Which means there's little motivation for stack developers to add support. It's a negative feedback loop.

As a very new Haskell user, I was exposed to backpack (while still getting my bearings straight with regard to Haskell build tools), only to realize it would be a no-go if I wanted to use it with Stack. I've since been tempted to use it again, but realized the issue was still open. Had I been a user of backpack all along (which has only been a little over a year), I'm sure I'd have thought of many more places I could have used it. Just my 2c - I understand all too well that development time is limited.

[ezyang]

Sorry about the delay, I saw this in my GH notifications but sat on it for a week.

@flip111

what did you mean with "register" in your opening post?

referring to:

Stack will need to understand how to register multiple libraries produced by a Setup script

My recollection from the last time I looked at the Stack code, was that everything in Stack revolves around installing packages (there's a dep graph of packages, you install them one by one). The comment on private Backpack use is simply pointing out that sometimes installing one package results in multiple libraries getting installed, and as long as Stack can deal with this situation, all is good.

---

are the OpenUnitId and DefUnitId from Distribution.Backpack serving the same function as the proposed types

Yeah, they're not the same, precisely because the proposed type here refers to packages rather than components.

Components are a pretty big deal for usability of Backpack, because a lot of things you want to do involve having to write many small libraries and then recombining them together. If you'd have to make (and distribute) a separate package for each of these libraries, it would be a serious source of friction.

---

Is it possible that such an instantiation requires a new package? Or is it fine to do these instantiations after package dependencies have been calculated ?

It's fine to do it after depsolving; that's how Cabal-install does it. Depsolve the package structure, and then elaborate it into a more fine-grained component structure.

---

As for the last few comments, the kit in the Cabal library is what stack can usefully use, the stuff in cabal-install is a useful reference for how to use the APIs but I expect none of it is salvageable. That's OK; I have a "mini" implementation of the Cabal-install logic for Setup.hs and it was mostly fine.

To me, the problem was always how to drag Stack into the component world. That required more time investment learning about Stack that, sadly, I never found time to do in the end.

[cartazio]

@snoyberg is this on the todo queue of y’all ?

[theobat]

Following:

To me, the problem was always how to drag Stack into the component world. That required more time investment learning about Stack that, sadly, I never found time to do in the end.

I think for now, it's better to follow the progress here, for anyone interested:
https://github.com/commercialhaskell/stack/issues/4745
Also, given that component-centric builds seem to require significant changes, my guess is backpack support in stack is unlikely to land before some time.