Ghcide: Multi-cradle/v2-build support

[merijn]

So, as far as I can tell the main problem all the build tools have with v2-build is figuring which component a file belongs to. I was brain-storming on IRC with @mpickering how hie-bios could at least provide a work around by simply providing hard-coded mappings from directories to cabal components.

He's already whipped up a WIP before I had a chance to look at it. So the question now would be: What would it take for ghcide to make use of this/support v2-build once this functionality makes it into a hie-bios release?

[cocreature]

@ndmitchell already started working on multi-environment support in #24. However, that motivation for that was mostly to load a bunch of unrelated packages. For multiple components, I expect that you probably want to load all of them into the same session which is a bit tricky/impossible in general if the flags are incompatible. Until GHC starts supporting loading multiple components into the same session properly, I expect that some merging of flags might be more useful.

[merijn]

or multiple components, I expect that you probably want to load all of them into the same session which is a bit tricky/impossible in general if the flags are incompatible.

Sure, that'd be the ideal, together with auto-detecting them from cabal files. But in the interim if it only works for a single component at a time/per session then I'll take it, because that's still infinitely better than nothing working at all like right now.

[mpickering]

I am waiting for @ndmitchell to comment on https://github.com/mpickering/hie-bios/pull/53 because I suspect it will not work for ghcide without some modifications.

I believe it implements the API that @merijn was suggesting though and I agree it would be convenient to be able to specify multi-component projects in this way.

[ndmitchell]

Do we yet know what GHC flags are incompatible? I would imagine things like language extensions etc are compatible. We deal with include files so they are incompatible. Specifying different versions of different packages, or different package home tables would be incompatible - but my guess is that is pretty rare. I'm increasingly wondering if we should have two modes:

• If you have two distinct hie.yaml files then the packages are 100% unrelated. They can both exist in the IDE, but they don't share anything. The main reason for having this is so that you don't have to open the "root" of a package to get ghcide working on it, and can support huge megarepos where people are only working on one project within it.
• If you have one hie.yaml file then I'm increasingly thinking you should get a single GHC session with compatible flags. If that isn't possible, it sucks, and we'd have to do something else. But everyone _wants_ the compatible flags version, so I'd like to be convinced its out of reach before trying plan B.

[cocreature]

What do you mean by compatible? Extensions are definitely not compatible in the sense that you can just union them together and things will work. The most obvious example of that is NoImplicitPrelude but there are more subtle examples like MonoLocalBinds (implied by TypeFamilies).

That said, I’d be fine with telling users that it’s up to them to ensure that unioning their flags is valid. stack ghci seems to get away with that as well.

I do wonder if we could use a single HscEnv that takes care of the package setup which is usually the expensive part but modify things like extensions and warnings in the DynFlags per file before we parse/typecheck but it might just not be worth the complexity.

[mpickering]

Making a single component in the IDE is not something I am keen on (at all). It is unpredictable and leads to frustration. If a build tool decides it wants to set up the environment that way, then fine, but it's no longer my responsibility when it breaks horribly!

What I don't understand really is the objection to a single HscEnv per component. You need to use an HscEnv because each component will have different package dependencies so things start breaking quite quickly if you try to reuse a HscEnv and swap out the package db.

Disadvantages to one HscEnv per component.

1. Reloading is quite slow across components as you have to restart the cradle.
2. It uses more memory because of not sharing anything to do with the package db across components.

Until GHC supports loading multiple components into one session I think it's the safest/most robust option.

[ndmitchell]

Should have been clearer. I meant have a single base HscEnv that contains things like package database information, which must be compatible between all components (and usually is - albeit some have a subset). Then set the extensions locally in the DynFlags on a per module basis (or precompute and have two cached HscEnv's with slightly different DynFlags in them) - so things like incompatible extension are no big deal. What I don't know is what a big deal is, and what causes issues.

@mpickering everyone expects multiple components to just work. It's the top FAQ, by a long way. I go as far as to always use the direct cradle because I consider it a fundamental and essential way of working. It's not just slow, it requires manual intervention, which is grim.

[cocreature]

If the restarts are manual, one HscEnv per component seems really confusing. Now you have to keep track of which file is in which component to know when you need to restart.
If the restarts are automatic it might be feasible from a UX perspective but that also doesn’t sound super easy to implement and it’s clearly slower.

Similar to Neil, I also always load projects as a single component an ghci (one of the very few cases where ghc environment files actually end up being convenient) since I find that much more convenient to work with.

[merijn]

Similar to Neil, I also always load projects as a single component an ghci (one of the very few cases where ghc environment files actually end up being convenient) since I find that much more convenient to work with.

How do you work like that if you have multiple components?

I get that properly computing a compatible environment is tricky, but why not just...not? I mean, sure that's not ideal, but it's not very likely to break for the common use-case and having "sometimes breaks" support for v2-build is better than "no support for v2-build". And once something "mostly works" it can always be improved once the necessary upstream features exist.

As for tracking which component a file belongs to, the entire basis for the hie-bios change is simply "just make the user define that", then use that. I wager 95% of all cases source files of different components are split trivially by path and if not, well, too bad for those 5%, then.

There's also the hacky solution I was working on in my own ghcid based program which maintained a cache of path-to-component mappings (loading a component tells you all modules/files belong to it), and if the mapping was unknown I'd just try 1 component at a time until something worked. Which is, of course, terrible in terms of performance but it worked pretty well and you only have to load each component once to figure it out for the rest of time.

[ndmitchell]

How do you work like that if you have multiple components?

Quickly and efficiently. I treat them all as a single lump and then track edits across the whole stack in milliseconds. It's wonderful! If you tell me I can't have that experience for another 2 years or more I'd be a very sad bunny, especially since I know I can make it work today (and have been doing so for at least 5 years).

I don't see this as a "no support for v2-build" or "merge components" - I think they are orthogonal.

[merijn]

Quickly and efficiently.

No, I mean literally "how?", because at this point I'd commit murder if I can get my warning/error highlighting and type querying back even in 80% of the cases for my code.

If I knew how to define a cradle to work for my multi-component v2-build codebases I wouldn't be here...

[ndmitchell]

Hehe, I took the context entirely the wrong way!

I just define a direct cradle. I specify all the -I flags to point at all the directories. I -I into the autogen directories. It's a pain, and it's easier because I have a cabal v1 style global install so things are just globally available. It should be possible with a v2 install too, but I don't know how - looking at what hie-bios figures out as the arguments for each component and smushing them together should roughly work.

[cocreature]

@merijn I was talking about ghci not ghcide. You need to get your package environment from somewhere. If you get it from nix it’s just available globally so things should work in ghcide as well. If you get it from cabal v2-build, you can use a .ghc.environment file for ghci but I expect that we probably don’t pick them up in ghcide (but I’ve not tried it). Once you have the package environment, it’s a matter of writing a .ghci file for GHCi or a direct-style cradle for ghcide that sets the include dirs and default extensions/warnings/….

All of that is annoying to do so I absolutely think we should automate this. However, the result is much more convenient for any project that I spend a non-trivial time working on so I think spending some time on implementing a way of doing this automatically is definitely worth it over loading everything in separate components.

[istathar]

Does anyone know of a work-in-progress branch that we might try and help on now that hie-bios 0.3.2 is being used? I should know better, but I keep trying to use VS Code to work on projects that have multiple (sub) libraries in them and of course the result is suboptimal. Is there any way I can help?

[cocreature]

I don’t think anyone has taken a look at this so far. I was hoping that I might be able to find some time during the Bristol hackathon.

[3noch]

I should mention that for multiple package support in Obelisk we've implemented a terrible hack that actually works reasonably well so far. First, all your packages must share a common dependency tree, so no improvements there. But for flag merging, we actually do something sort of clever/evil. In the ghci session we enable a preprocessor and configure it such that it is able to determine which cabal/hpack config maps to each source file. It then injects the appropriate flags/language-extensions/etc into the file that ghci actually sees. This allows us to have conflicting settings in each cabal file without having to merge them in any awkward ways. There are no doubt edge cases we haven't yet uncovered, but it's working better than it probably should :P.

[3noch]

The one clear downside we've noticed so far is that some dependencies export the same modules (:angry:) and if two packages in your session end up bringing two of these into "scope" then suddenly both of them suffer from ambiguous imports. We use PackageImports manually to work around this for now.

[runeksvendsen]

In the ghci session we enable a preprocessor and configure it such that it is able to determine which cabal/hpack config maps to each source file. It then injects the appropriate flags/language-extensions/etc into the file that ghci actually sees.

Can you elaborate on the above trick? I'm looking for workarounds for this issue.

[3noch]

We had to write a custom preprocessor to read cabal files and inject their configurations into each source file. The implementation is used in ob run in https://github.com/obsidiansystems/obelisk

[istathar]

With https://github.com/digital-asset/ghcide/pull/522 having been merged is this issue now redundant or closed?

[jneira]

@istathar thanks for point it out, closing