Cabal: Module listing discovery

[parsonsmatt]

Now that cabal has common stanzas, the only thing keeping me using hpack is that hpack will, by default, autodiscover all of your modules and stuff them into exposed-modules (you can disable this by explicitly defining that stanza). This feature is really convenient to me.

Mimicing that behavior seems like it might be against the Cabal philosophy, as it's implicit -- "If exposed-modules is not defined, collect all modules not listed in other-modules and make them exposed-modules."

A middle ground solution that provides convenience with a degree of explicitness is glob patterns for the module listings. Consider this syntax:

exposed-modules: *

This would find all modules in the source directory and add them as exposed-modules. Kinda like hpack's autodiscovery, but we're explicitly writing "please find all the modules for me."

You could also write:

exposed-modules:
    Control.Monad.*

which would only collect modules under the Control.Monad namespace into exposed-modules -- any other modules would need to be either explicitly added to exposed-modules or other-modules.

[23Skidoo]

It'd be also convenient to be able to specify module names to be treated as exceptions from auto-exposing with glob patterns: other-modules: Control.Monad.Internal.*.

[gbaz]

Right. I'd think we'd want some logic relating the two. Explicit always takes precedence over glob. So other-modules explicit takes precedence over glob in exposed-modules, and exposed-modules explicit takes precedence over glob in other-modules. If both are globs or both are explicit, I'd suggest an error, rather than trying to be too clever and pick a potentially nonobvious direction to resolve the ambiguity?

[gbaz]

I also don't know if there are interaction effects with backpack that we'd have to worry about -- i.e is it genuinely always the case that given a glob we can simply scan a file-listing without inspecting any contents to determine what it expands to?

[parsonsmatt]

Yeah, given the following modules:

Foo
Foo.Bar
Baz
Baz.Internal

I'd expect the following:

exposed-modules: *
-- other-modules: [] -- no other-modules clause

-- expands to:

exposed-modules:
    Foo
    Foo.Bar
    Baz
    Baz.Internal

exposed-modules:
    Foo
    Foo.*
 other-modules:
    Baz.*

-- expands to:

exposed-modules:
    Foo
    Foo.Bar
other-modules:    
    Baz.Internal

and a warning/error that module Baz is not in either listing, as happens now

exposed-modules:
    *
other-modules:
    Baz.Internal

-- expands to:

exposed-modules:
    Foo
    Foo.Bar
    Baz
other-modules:
    Baz.Internal

Because the explicit other-modules overrides the * globbing.

exposed-modules:
    Foo
other-modules:
    *

-- expands to:
exposed-modules:
    Foo
other-modules:
    Foo.Bar
    Baz
    Baz.Internal

For the same reason as above.

I have no idea how this would interact with Backpack :thinking:

[cdepillabout]

It would be really nice to get this change into cabal.

This is the sole reason we are still using hpack at work.

[domenkozar]

I'd be happy to chip in to get this one done. It's the last annoyance I have that hpack fixed :)

[phadej]

I’d be 100% against allowing files with globs to be uploded to Hackage, as ”which package exposes XYZ module” will be impossible to answer only considering the index.

I’d like to have solution to this, but losing the declarativeness is huge concern, and if it’s not addressed, this will be wontdo.

On 5 Aug 2019, at 14.21, Domen Kožar [e-mail] wrote:

I'd be happy to chip in to get this one done. It's the last annoyance I have that had hpack fixed :)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[domenkozar]

What if glob would be allowed for local development but rejected by hackage. For hackage sdist could substitute glob with the actual files.

You kind of lose git repository to hackage mapping, but that was never the thing with revisions anyway.

[hasufell]

You kind of lose git repository to hackage mapping, but that was never the thing with revisions anyway.

Yep, revisions are already a wart (knowing that they are a sad necessary fix), break freeze files if you don't freeze the index, cause problems for distributions, are infrastructure specific, ...

Let's not take this as an argument of good design to allow more semi-defined behavior.

I feel like hpack is exactly what fills the gap here and careful maintainers commit both the hpack and the .cabal files to their repository. Is this really cabals job? If not hpack, then it sounds more like an IDE feature.

[phadej]

FWIW, I released https://hackage.haskell.org/package/cabal-fmt-0.1 which in addition to (opinionatedly) formatting your .cabal file can expand exposed-modules, e.g.

  hs-source-files: src
  -- cabal-fmt: expand src
  exposed-modules:
    Foo
    Foo.Bar

i.e. in the case when new modules are added, I run cabal-fmt --inplace my.cabal and exposed-modules are re-populated.

The functionality is bare-bones, let's see what will be needed. In one project, I simply moved non exposed-modules into other hs-source-dirs (actually other-modules, and main-is: Main.hs is in other dir now).

[chshersh]

cabal-fmt looks helpful and useful. However, the UX of calling command to discover modules is not the same as automatic support of modules discovery by the build tool. In that case, I might as well write the module name manually.

hpack showed the benefits of such package configuration features like metadata deduplication and automatic module discovery. And if you can reduce duplication of stanza information within a single package using common stanzas, a lot of people are still using hpack because they found the automatic module discovery feature handy. For me, the drawbacks of using hpack outweigh the benefits of its features, and I'm not using it in my projects. However, this particular feature of the module listing discovery is very useful. It makes Haskell development experience smoother, more pleasant and more beginner-friendly, which is very important for Haskell.

As a maintainer of multiple open-source Haskell libraries and applications, I welcome every contribution. And I want various people to be able to contribute to my projects with as little hassle as possible. That's why I support both build tools cabal and stack. And I find the number of configuration files required for a single Haskell repository too big to my taste:

package-name.cabal
cabal.project
cabal.project.local
stack.yaml
stack.yaml.lock
package.yaml

and there could be more, the list is not exhaustive

I think that reducing the number of configuration files required for a project, and the number of tools and formats people need to know to develop Haskell projects while preserving the same features is a goal worth trying to achieve. It seems to me that this particular minor (in terms of implementation) feature of module listing discovery can push cabal much further and helps already fractured community to become closer to the consensus of package development.

[phadej]

It seems to me that this particular minor (in terms of implementation) feature of module listing discovery can push ...

I'll be happy to see a PR.

My experience on Cabal and cabal-install (and downstream tooling using Cabal as a library) says very opposite. This change is not minor, as .cabal file interpretation becomes file-system state / tarball dependent. (The extra-source-files is different, at least for now, as those are only interesting for sdist command).

EDIT: note the fact that hpack "works" is because there is clear stage separation. package.yaml is compiled to pkgname.cabal, and pkgname.cabal is used as input to actual build tooling. Blurring the boundary, requiring to .cabal files to be interpreted before further consumption it tricky.

[gbaz]

I think the idea could be that sdist should expand globs before producing the tgz, as domenkozar suggested. This would mean that downstream tooling making use of sdist'ed packages would not have to worry about such things.

[chshersh]

In @kowainik we've implemented an alternative experimental approach for automatic module listing discovery using the custom setup Cabal feature — the autopack Haskell library that automatically finds all Haskell files in the corresponding hs-source-dirs and populates exposed-modules.

• https://github.com/kowainik/autopack

So, if you don't want to maintain the list of exposed modules manually, you can give it a try.

[szabi]

I’d be 100% against allowing files with globs to be uploded to Hackage, as ”which package exposes XYZ module” will be impossible to answer only considering the index.

A solution would be that Cabal autogenerates an explicit module listing metadata file at the same time, similar to the Path_<module_name>. Hackage and other discovery tools could use that.

[finlaydotb]

I created #7016 to mention how tedious it was to manually update the .cabal file upon addition or removal of modules.

Since then I tried Stack and I will not be going back to cabal-install again! Chiefly because of the fact that with Stack I do not need to manually update modules in cabal files...that alone is a killer UX feature worth the switch.

Do not know if this count much for data points regarding this feature request but I thought to share.

[hasufell]

Since then I tried Stack and I will not be going back to cabal-install again!

It isn't really stack, it's still hpack you are interfacing with. Stack just runs it automatically for you. It seems this confuses users already about what their tooling is really doing. So I'm not even sure this is a feature or misfeature.

That said, it's very easy to add a patch to cabal-install to run hpack just like stack prior to doing anything. Hpack can be used as a library. But I'm confident this patch will get rejected.

[finlaydotb]

It isn't really stack, it's still hpack you are interfacing with.

I know. But the fact that Slack provides a unified UX...ie I do not have to go tinker with hpack myself, is a win.