Cabal: Build local package with local packages of its own?

[mitchellwrosen]

Hi, apologies if this issue tracker is not the right spot to ask basic usage questions -

Say I have a project with a package foo and a local package bar:

-- foo's cabal.project
packages: ., bar

-- in foo's cabal file
library
  build-depends: bar

Now, bar itself might have some local package baz, upon which the bar library depends.

-- bar's cabal.project
packages: ., baz

-- in bar's cabal file
library
  build-depends: baz

Is it possible to build foo without adding baz to foo's project definition, like so?

-- foo's cabal.project
packages: ., bar, bar/baz

Thank you!

[hvr]

@mitchellwrosen if I understood your question correctly: yes, the last cabal.project should just work as expected. Didn't it work?

[mitchellwrosen]

Yes, that last cabal.project would work, but I'm wondering if it's possible to build foo without mentioning baz, as it's a transitive dependency.

[hvr]

ah ok, no; we don't support chaining cabal.project files in that way; and I don't think that cabal.project files would compose very well, especially the more settings they contain, it's easy to get into contridictions/conflicts.

[mitchellwrosen]

Ah, okay. Perhaps cabal currently isn't great for our use-case at work, for which we're using stack, and I imagine is incredibly common.

We have many private github repositories containing unpublished packages, and no private hackage server (it's a bit heavyweight).

I thought for a moment that git submodules + cabal new-build would be a great solution. Packages could independently pin different versions of private dependencies by their git commits, and cabal could just build each package per to its own cabal.project.

But unfortunately, it sounds like local packages' local packages must necessarily bubble all the way up to the top, which is both tediously leaky _and_ I imagine it would be an error to have two different local packages (say foo/submodules/bar and qux/submodules/bar) that contain (different versions of) the same package.

Does this workflow make sense? And do you happen to have any advice on how to proceed here? Thanks for the help by the way!

[ezyang]

Yes, as you point out, solution you've suggested here is not actually a good solution for your problem.

Here is one possibility: build a monorepo-style cabal project for your entire universe. This has all of the attendant benefits and downsides of a monorepo (benefit: every package in your company's repo is guaranteed to work with each other; downside: you need to make everything work together.) cabal new-build only builds things that you actually ask for, so you don't have to worry about build-time trouble for things you don't care about. There are also some cabal specific downsides: we will do O(n) depsolving every time you change a cabal file for the entire project.

[mitchellwrosen]

Thanks :)

[23Skidoo]

no private hackage server (it's a bit heavyweight)

If you decide to go that route after all, you may find https://github.com/scrive/static-hackage helpful. This is what we use at $WORK.

[gbaz]

There are also local repositories: https://www.haskell.org/cabal/users-guide/installing-packages.html#secure-local-repositories

This lets you just configure a file-system based setup for repos with no webserver. I think there may need to be a little script tooling to make this convenient -- arguably this is something that could use good porcelain.

[mitchellwrosen]

Thanks @gbaz, that's the solution I'm toying around with the moment. It could definitely use some porcelain ;)

Specifically, "legacy" local repos seem a lot simpler than messing with hackage-security keys, yet they are totally undocumented. So going the hackage-security route, hackage-repo-tool is also underwhelmingly documented and not very flexible at the moment. (I have to re-run hackage-repo-tool update at least once every three days to be sure my own filesystem isn't trying to MITM me? Wat? Heh.)

[hvr]

at least once every three days to be sure my own filesystem isn't trying to MITM me

does it help to set the key threshold to 0? (if not, we should add some additional conf flag to bypass the freshness checks)

[mitchellwrosen]

I honestly have no idea. Hasn't been three days yet.

On Jan 23, 2018 1:56 PM, "Herbert Valerio Riedel" [e-mail]
wrote:

at least once every three days to be sure my own filesystem isn't trying
to MITM me

does it help to set the key threshold to 0?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/haskell/cabal/issues/5060#issuecomment-359893308, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABBlpkzaAWiWKm0lOpXktHhBosEs1xOAks5tNitzgaJpZM4RljYd
.

[gbaz]

Happy you're giving it a shot! It would be great to get some feedback and suggestions on how to make working with hackage-repo-tool easier, or even just a documented workflow.

[bgamari]

@mitchellwrosen, I look forward to hearing how this works out for you. I have some rather sizeable projects myself which I manually synchronize project files. Having a better solution would be great.

[ezyang]

BTW, we're running hackage-repo-tool in the Cabal test suite, I managed to make it work enough to get Cabal running with the repos it does. https://github.com/haskell/cabal/blob/master/cabal-testsuite/Test/Cabal/Prelude.hs#L487

[sol]

@mitchellwrosen I have a similar use case. How did this work for you?

[mitchellwrosen]

@sol It didn't go very well, all options are painful.

1 - A monorepo as @ezyang suggested is an interesting idea, but would be too massive of a change to our organization to even consider at the moment. I also have doubts about how it would play out in practice, particularly around making backwards incompatible changes and having to either fix all downstream packages yourself, or, I don't know, have some mechanism for gracefully managing the "latest working" universe alongside the "latest and probably broken" universe. More investigation and thinking required here.

2 - Local hackage repos are okay but it's so tedious and inconvenient to bump dependencies, as it involves packing a little tarball, git push this, git pull that, rebuild the world. It doesn't interact nicely with the nix store at all, since there is so much thrashing. Every little change that you want to integrate into some other project ends up requiring a patch-bump.

So, for now, we are sticking with stack and most of our private repos point at SHAs. We _try_ to leverage version bounds, making an "official" (git tag) release every once in a while, which cleans up our stack.yamls a bit, but realistically our packages basically _always_ only build against the _latest_ version of every private library (i.e. we don't typically have any build-depends: my-lib >= 0.1 && < 0.4, it's always a >= 0.3 && < 0.4, which becomes >= 0.4 && < 0.5, and so on). So, maybe this is more trouble than it's worth. And still, it's painful to tell stack the locations of all of our private dependencies' private dependencies' private dependencies... and keep everything building together. Very often we have build errors due to inaccurate or missing version bounds in cabal files, because following PVP for fast-moving packages is difficult, error-prone, and unduly burdensome.

I haven't put too much thought into what features our tooling is missing. I'm just sort of doing what seems to work okay (stack + git) and moving on with my life. But I'd love to hear about what is or is not working for other people, how to improve things, etc, because every time I run into a build error, I'm like, "Huh. A little gust of wind knocked over our house of cards. Time to pick them all up. Yay."

[bgamari]

Thanks for great summary, @mitchellwrosen!

[gbaz]

Local hackage repos are okay but it's so tedious and inconvenient to bump dependencies, as it involves packing a little tarball, git push this, git pull that, rebuild the world.

It sounds like this is where a little program wrapping the hackage-repo-tool would have payoff? I.e. it seems most of this should be able to be automated if the right workflow patterns could be hashed out...

[sol]

@mitchellwrosen thanks a lot! I'm still not giving up on it.

BTW, did you manage to specify a custom config file to use instead of ~/.cabal/config on the command-line. @hvr indicated on IRC this sholud be possible, but I can't find anything in the help output (cabal new-build --hepl)? I have a approach in mind that would greatly benefit from the ability to do so (I'll share more if it works out).

[gbaz]

Setting a custom config file is not a new-build specific option, but scopes over all cabal commands, so you see it with cabal help -- also note that such global flags need to be placed _before_ the [COMMAND [FLAGS]] block.