Cabal: Better failure on missing module in `exposed-modules` or `other-modules`

[ekmett]

If I forget a module in exposed-modules or other-modules, but a module I _do_ remember to add references I get a very hard to track down error.

cabal will let me build a package like that locally.

Hackage lets me upload the package, but then users get problems, because the resulting .hs files don't get included as they aren't referenced by the cabal file.

It would be really nice if there was a way that cabal could complain during cabal build if a module gets brought in transitively that isn't in either of those lists.

I'm sure I'm missing some corner cases it'll break, like Paths_foo or some such, but I felt the need to ask as this probably hits me about every 3 months.

[23Skidoo]

I have ideas about how to fix this, but it will require a patch for GHC. See also #1455.

[hdgarrood]

Here's another potential approach to this, which I expect would be significantly simpler: before building, search through hs-source-dirs recursively for any valid Haskell modules which are not mentioned in any of exposed-modules, other-modules, main-is, etc., and emit warnings if any are found.

This approach assumes that there is no reason for someone to want valid Haskell modules in their Haskell source directories, but for those modules to never be referenced in the cabal file. It seems fairly probable that this is a safe assumption, but I'm not really sure.

[MedeaMelana]

You would do this on cabal build? Or only on cabal dist?

[hdgarrood]

Maybe both? It would be nice to be made aware of this earlier, and I can't imagine that it would be a noticeable performance hit.

[hdgarrood]

Also it seems that if a module is referenced in other-modules and it doesn't exist in any of the source dirs, there's no warning either. See https://github.com/purescript/purescript/pull/980 for example; the actual module was called Ctags but the cabal file contained CTags.

[MedeaMelana]

Could this be a Mac OS filesystem case sensitivity issue where it did work for the author?

[hdgarrood]

Ohh, true. I'm on Linux, and I think he was on Mac. Perhaps cabal should check that the case of module names in the cabal file and files on the file system match up too.

[hvr]

Related: commercialhaskell/stack#32

[ttuegel]

@23Skidoo I just realized, I think we can do this without modifying GHC! All we must do is scan the build directory for interface files; that will indicate which modules ghc --make loaded. To avoid false positives, we should clean out the build directory when reconfiguring, but I think that is required for the new build infrastructure anyway.

[23Skidoo]

@ttuegel Interesting idea. I still think that having the full build graph available to Cabal is the right way forward, but this could serve as a backwards-compatible fallback.

[ezyang]

Note that the way Stack does this is by using ghc --show-iface, which can be parsed to extract out the transitive closure of all local modules a module depends on. That list can then be compared to ensure that a user has listed all of the modules.

[ttuegel]

Note that the way Stack does this is by using ghc --show-iface, which can be parsed to extract out the transitive closure of all local modules a module depends on. That list can then be compared to ensure that a user has listed all of the modules.

The GHC manual specifically says not to rely on parsing that format, as it may change without notice. I guess we can always hold an axe over them if they change it.

If Stack has a working implementation of this, then copying it is the only reasonable thing to do.

[ezyang]

Erm, I have to eat my words slightly; it is not --show-iface they use, it is -ddump-hi as they are compiling. See https://github.com/commercialhaskell/stack/blob/master/src/Stack/Build/Execute.hs#L1430

[ttuegel]

Naturally, the manual doesn't specify the format produced by -ddump-hi or its stability. :grinning: Well, if it works it works.

[Yuras]

While we are waiting for solid solution, we can teach ghc to warn about implicit dependencies, not listed in the command line. It can be enabled via -fbuilding-cabal-package. It is literally 10 lines of code.

[Yuras]

Hmm, looks like cabal doesn't list all modules when building executable, so recomp007 test case in GHC fails. So either cabal should list all modules, or we need a separate flag in GHC. Does it sound like a plan? I want to fix this issue finally because it keeps biting people.

[hvr]

@Yuras

While we are waiting for solid solution, we can teach ghc to warn about implicit dependencies, not listed in the command line. It can be enabled via -fbuilding-cabal-package. It is literally 10 lines of code.

Yes, please! this was actually my plan as well... but never got to it

IMO, Cabal should list all modules to GHC; however, an additional -W... flag to control this warning sounds fine to me as well; especially since we can then also promote it to an error easily via the new -Werror=... feature

[Yuras]

I made a ticket and a patch for GHC.

[deepfire]

I'm sorry, as I feel that I'm going to ask a stupid question, but -- why do we even need the full list of modules in .cabal files in the first place -- since it's deducible?

It certainly seems like a maintenance burden on package authors..

What is on the other side of the scales?

EDIT: s/packages/modules/

[23Skidoo]

It's not (fully) deducible if CPP is used to import some modules conditionally (for example, for platform-specific stuff). Plus there's the exposed-modules vs. other-modules distinction for libraries. Plus in the case of libraries the dependency graph can have multiple root nodes, and there's no way of knowing what they are.

Also Cabal currently doesn't know anything about the module dependency graph, though arguably it should.

[hvr]

...besides CPP there's also TH and custom pre-processors that can dynamically affect the import graph. Moreover, if GHC can rely on cabal to provide it the complete list of modules (and Cabal needs to have this information reliably early on to operate correctly (file-monitoring & sdist) anyway), we can optimise/simplify the logic in GHC in future, as well as consider different ways to invoke GHC (e.g. by passing GHC explicit filenames rather than module names which happens to solve another issue I've had in the past). So this -Wmissing-home-modules warning can be considered the first phase towards improving all this.

[ttuegel]

It's not (fully) deducible if CPP is used to import some modules conditionally (for example, for platform-specific stuff).

It's not even partially deducible as a module with the same name can be exported by multiple packages.

[gbaz]

does -Wmissing-home-modules fix this enough to close the ticket?