Cabal: How to correctly install a locally-built library in Cabal 3?

[acrylic-origami]

I'm having difficulties installing and using a library built from source. It's likely just a documentation/my understanding issue, but to do the following:

1. Build a cabal library package from source that depends on other [Hackage] cabal packages;
2. Install the package into either the user or global package db;
3. Depend on the package in another project on the same machine.

I could only find one working method that I cannot believe is the Right Way:

1. runhaskell Setup.hs configure --global; must be global. User installs aren't found by other projects (which I don't understand).
2. For all Hackage dependencies, manually register them into the global package db. That means ghc-pkg register --global /path/to/user/package.db/package-name.conf.
   
   
   
   • This was needed because setup build was failing with "Encountered missing dependencies". I saw the call it was making to discover packages was /usr/local/bin/ghc-pkg dump --global -v0, where the package (installed by cabal install --lib) was missing. Trying to install with cabal install --global --lib package-name still doesn't show up in ghc-pkg dump --global.
   
   
   • This is the main pain point for this otherwise documented workflow. Should Hackage packages be registered by default to the global db? Alternatively, should Setup look in the user package db too?
   
   
3. runhaskell Setup.hs build
4. sudo runhaskell Setup.hs install
5. Include the package elsewhere in build-depends as usual.

This is with Setup as Distribution.Simple.defaultMain on GHC 8.6.5 and Cabal 3.0.0.0. GHC 8.4.2 + Cabal 2.4.0.0 surprisingly works with a simple cabal build; cabal install;, so maybe it's a version thing?

Any other combinations seem to fail. In particular, the intuitive cabal new-install --lib: the package isn't found in ghc-pkg list nor cabal list --installed and is an "unknown package" to other libraries. Interestingly, trying to use the package via -fplugin without build-depends will instead say that it's a hidden module of our library.

In case this is rare workflow, at least to develop source plugins it's a vital one. This way, I don't have to specify a source-repository-package and build it again and again in every codebase I analyze, or upload it to Hackage for every edit I want to test.

[elliottslaughter]

I've had a similar issue, where I want to write "scripts" that make use of the Pandoc API. I have not been able to find a set of new-style commands that will let me use ghc --make to link against an installed copy of Pandoc. See https://github.com/jgm/pandoc/issues/6066 for the details of what I've tried.

The only thing I've found that works is to go back to the v1- subcommands, i.e. cabal v1-install pandoc.

It would be nice if there were a way to use the new-style subcommands, otherwise I guess I'll just not be updating past the point where the v1- subcommmands are removed.

[phadej]

@elliottslaughter there is cabal install --lib, which will create a package environment with pandoc. This issue about the case, when you want to install the local (say modified) version of a library. I think that cabal install --lib should "work" in that case to (modulo the reported issues like https://github.com/haskell/cabal/issues/5559, where it fails when you try to update an environment)

[elliottslaughter]

@phadej If you see https://github.com/jgm/pandoc/issues/6066, I did try this, with the result:

$ cabal install --lib pandoc
$ ghc-pkg list pandoc
.../ghc/8.8.1/lib/ghc-8.8.1/package.conf.d
    (no packages)
$ ghc --make behead.hs 
Loaded package environment from /Users/elliott/.ghc/x86_64-darwin-8.8.1/environments/default
[1 of 1] Compiling Main             ( behead.hs, behead.o )

behead.hs:4:1: error:
    Ambiguous module name ‘Text.Pandoc’:
      it was found in multiple packages: pandoc-2.9.1.1 pandoc-2.9.1.1
  |
4 | import Text.Pandoc
  | ^^^^^^^^^^^^^^^^^^

I also tried --global --lib (won't copy that output to save space, but see the other issue).

Is this a bug? Should I open a new issue?

[phadej]

$ ghc-pkg list pandoc

is wrong, cabal v2-install --lib pandoc doesn't register anything in global package database, that's the core point of v2-build approach.

If you instead look into ~/.ghc/ghc-8.8.1-.../environments/default, there should be pandoc.
Also if you look ghc-pkg list --package-db=~/.cabal/store/ghc-8.8.1/package-db there will be pandoc (and a lot of other stuff).

So, ghc -package-env=default --make behead.hs should work (at least in theory).

---

I'm really sorry that documentation is not up to date, if you find something missing the documentation PRs are more than welcome.

---

Arguably the cabal v2-install --lib libname should tell which environment file it rewrote, maybe it will clarify the situation further. That should be an easy change to make. (Currently it seems to end, like it did nothing further then building the library).

[elliottslaughter]

Ok, that works, though I had to also run cabal v2-install --lib pandoc-types because otherwise pandoc-types is a hidden package, apparently. I guess I'll need to do this with every Pandoc dependency that I use in my scripts.

I agree that modifying the output of cabal v2-install would help, and you could go even further and provide some usage hints (because after all, how is anyone supposed to use an installed library if the commands don't do so by default?).

I'm not honestly sure what to suggest for the documentation. It seems the new-style subcommands introduce a new abstraction (the environment), but this is something shared between Cabal and GHC. Presumably the Cabal documentation wouldn't be the place to find out that I need to run ghc -package-env=default --make .... But you're right, there is a general lack of information, and it's hard to know where to even look at the moment.

[acrylic-origami]

Hmm, I'm still not able to get it working with cabal v2-install --lib libname. Other packages can't see the lib once installed. I do see it in ~/.cabal/store, but it doesn't show up in cabal list --installed either. Maybe it isn't getting registered? This is the default GHC 8.6.5 + Cabal 3.0.0.0 combo installed by ghcup on a fresh Ubuntu image. Here's what I get:

$ mkdir issue6478-lib && cd issue6478-lib
$ touch LICENSE
$ cabal init --lib
$ cabal v2-install --lib issue6478-lib
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - issue6478-lib-0.1.0.0 (lib) (requires build)
Configuring library for issue6478-lib-0.1.0.0..
Preprocessing library for issue6478-lib-0.1.0.0..
Building library for issue6478-lib-0.1.0.0..
[1 of 1] Compiling MyLib            ( MyLib.hs, dist/build/MyLib.o )
Installing library in /home/leaf/.cabal/store/ghc-8.6.5/incoming/new-936/home/leaf/.cabal/store/ghc-8.6.5/issue6478-lib-0.1.0.0-18a99bae1973d0ff27a0d1eac6b057cf6cf68066ecae7c323cc34066a07a2f13/lib
$ cd ..
$ mkdir issue6478-test && cd issue6478-test
$ cabal init
$ sed -i 's/build-depends:/build-depends: issue6478-lib, /g' issue6478-test.cabal
$ cabal v2-build
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: issue6478-test-0.1.0.0 (user goal)
[__1] unknown package: issue6478-lib (dependency of issue6478-test)
[__1] fail (backjumping, conflict set: issue6478-lib, issue6478-test)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: issue6478-test, issue6478-lib
$ ls ~/.cabal/store/ghc-8.6.5
issue6478-lib-0.1.0.0-18a99bae1973d0ff27a0d1eac6b057cf6cf68066ecae7c323cc34066a07a2f13
incoming
package.db
$ ls ~/.cabal/store/ghc-8.6.5/package.db
issue6478-lib-0.1.0.0-18a99bae1973d0ff27a0d1eac6b057cf6cf68066ecae7c323cc34066a07a2f13.conf
package.cache
package.cache.lock

[acrylic-origami]

I see now that ghc --make does work on packages installed with v2-install --lib, just not when those packages are named in the Cabal config file. Is it really just that it isn't registered properly for Cabal to find it? This note from the first comment seems relevant:

Interestingly, trying to use the package via -fplugin without build-depends will instead say that it's a hidden module of our library.

Also, I couldn't pick this out from the conversation, so I reproduced @elliottslaughter 's secondary issue by installing the dependent package twice, i.e. two versions end up in ~/.cabal. This isn't solved by -package-env, but by removing the old versions. Sorry if this is old news.

[asr]

I'm getting a similar problem when installing the upstream versions of shellac and lambda-shell.

Cabal and GHC versions:

$ cabal --version
cabal-install version 3.0.0.0
compiled using version 3.0.0.0 of the Cabal library 

$ ghc --numeric-version
8.6.5

First, I install the shellac library:

$ git clone https://github.com/robdockins/shellac.git
$ cd shellac
$ cabal install --lib
Wrote tarball sdist to /tmp/shellac/dist-newstyle/sdist/Shellac-0.9.9.tar.gz
Wrote tarball sdist to
/tmp/shellac/dist-newstyle/sdist/Shellac-editline-0.9.9.tar.gz
Wrote tarball sdist to
/tmp/shellac/dist-newstyle/sdist/Shellac-readline-0.9.9.tar.gz
Wrote tarball sdist to
/tmp/shellac/dist-newstyle/sdist/Shellac-haskeline-0.2.1.tar.gz
Wrote tarball sdist to
/tmp/shellac/dist-newstyle/sdist/Shellac-compatline-0.9.9.tar.gz
Resolving dependencies...
Up to date

And the shellac library is in ~/.cabal/store/ghc-8.6.5:

$ cd ~/.cabal/store/ghc-8.6.5
$ ls | grep Shellac
Shellac-0.9.9-71a5d5c15c193a908448cf9a7f3239951c5c0f6bf20708c697ddd938c81ae597

Now, I try to install lambda-shell:

$ git clone https://github.com/robdockins/lambda-shell.git
$ cd lambda-shell
$ cabal install
Wrote tarball sdist to
/tmp/lambda-shell/dist-newstyle/sdist/LambdaShell-0.9.9.tar.gz
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - Shellac-0.9.9 (lib) (requires build)
 - readline-1.0.3.0 (lib:readline) (requires build)
 - Shellac-readline-0.9.9 (lib) (requires build)
 - LambdaShell-0.9.9 (exe:lambdaShell) (requires build)
Configuring library for Shellac-0.9.9..
Preprocessing library for Shellac-0.9.9..
Building library for Shellac-0.9.9..
...
[ 4 of 10] Compiling System.Console.Shell.PPrint ( src/System/Console/Shell/PPrint.hs, dist/build/System/Console/Shell/PPrint.o )

src/System/Console/Shell/PPrint.hs:67:21: error:
    Ambiguous occurrence ‘<>’
    It could refer to either ‘Prelude.<>’,
                             imported from ‘Prelude’ at src/System/Console/Shell/PPrint.hs:50:1-38
                             (and originally defined in ‘GHC.Base’)
                          or ‘System.Console.Shell.PPrint.<>’,
                             defined at src/System/Console/Shell/PPrint.hs:96:3
   |
67 |         []  -> left <> right
...

and the installation fails because cabal-install is trying of install shellac from Hackage (and this version does not support GHC 8.6.5).

Any ideas?

[acrylic-origami]

@asr Interesting, it also seems then that v1-install into the user package db won't be resolved by cabal elsewhere when the package is also published to Hackage. I was able to get your package to build by installing shellac into the global package db with runhaskell Setup.hs install.

[asr]

I was able to get your package to build by installing shellac into the global package db with runhaskell Setup.hs install.

I could reproduce the installation using the global package db. However the installation failed installing shellac in the user package db. Weird.

it also seems then that v1-install into the user package db won't be resolved by cabal elsewhere when the package is also published to Hackage.

No. Using v1-install worked for me.

[acrylic-origami]

@asr Sorry yes, I understand now: the second package must also be built with v1 commands to find the first installed by v1-install.

[jrp2014]

It would be v good to have some common use scenarios drawn out in the documentation. (I'd offer to help, but don't have enough knowledge to do so reliably.) For example:

• it would be good to have an upfront explanation of the hierarchy of basic concepts and how they inter-relate: environment (in various flavours: ghc, package, project, etc), profile, project, package, library, executable, test, ... global v local v user, tabulating the differences between v1 and v2

• Basic use case: building an executable from hackage (trivial)

• Including a hackage library in a project that you are developing (trivial)
• Including a local version of a library in a package that you are developing that is not on hackage
• Including a local version of a library in a package that you are developing that is also on hackage
• packages that have both libraries and
• Multi-package projects, etc
• Compiler plugins
• How to address the most common diagnostics (eg, hidden packages, version conflicts (less of a problem these days) etc

This task-based documentation would complement the reference material in the manual.

I'm sure that this would not be an arduous task and would significantly reduce the cost of entry to building robust cabal configurations.

Perhaps this already exists somewhere, but I wasn't able to find it in a way that I could grok.

[fgaz]

cabal install --lib puts the packages in an environment so that they are visible to ghc, but cabal still cannot refer to them from another project unless you have a way to unambiguously specify that package's hash, so I think op's problem can be solved with a cabal.project that refers to both packages:

packages: path/to/package1 path/to/package2

then cabal build/run/whatever package2

It would be v good to have some common use scenarios drawn out in the documentation

I also think the docs could benefit a lot from a cookbook if there isn't one already. Maybe you could open an issue with a list of common scenarios, then we can start filling out the "how"s?

[acrylic-origami]

@fgaz This is a pretty good solution. Somehow I had never seen this usage, although searching for similar topics I'm finally starting to notice it, albeit usually buried under some sandbox-related solution.

I also see that it won't work for more than one level of local dependency (see #5060). Also seen in that issue are some further difficulties of truly installing then depending on packages as opposed to building in every package the cabal.project way. But for my little project ecosystem the latter usually works out okay.