Cabal: Make cabal new-doctest

[phadej]

cabal doctest doesn't work with nix builds:

% git diff
diff --git a/distributive.cabal b/distributive.cabal
index 94ad11e..f0d049e 100644
--- a/distributive.cabal
+++ b/distributive.cabal
@@ -12,7 +12,7 @@ bug-reports:   http://github.com/ekmett/distributive/issues
 copyright:     Copyright (C) 2011-2016 Edward A. Kmett
 synopsis:      Distributive functors -- Dual to Traversable
 description:   Distributive functors -- Dual to Traversable
-build-type:    Custom
+build-type:    Simple
 extra-source-files:
   .travis.yml

...

 % cabal --version
cabal-install version 2.1.0.0
compiled using version 2.1.0.0 of the Cabal library 

% cabal doctest
Run the 'configure' command first.

[angerman]

I assume we are talking about nix-style builds, and the need for new-doctest?

[phadej]

Yes.

[23Skidoo]

If this can be fixed on the cabal-install side only, the fix can still make it into cabal-install 2.0 (or 2.0.1).

[quasicomputational]

I think that the design of the current cabal doctest will want some changes when it becomes cabal new-doctest. I'm just going to jot down some thoughts here.

• I'm a bit worried that a naive implementation will take a long time to run on larger projects. I think incrementality is likely necessary: some kind of state will have to be kept, and only the actually-changed components (including transitively changed) re-doctested. This may not be an issue if doctest is actually blazing fast, but I don't believe it is.

• There should probably be a way to opt out of doctests on a module-by-module basis, as well as a component-by-component or package-by-package method. The module-by-module method could be something like an {-# OPTIONS_DOCTEST no-doctest #-} pragma, which I guess would be doctest's concern and not Cabal's, but the coarser-grained method would be in Cabal's court, and should probably go in the .cabal file.

• We'll need a doctest-depends field. Motivating example: a library that provides lenses without depending on lens, but which wants to use lens in its examples.

• Should doctests be their own components? We'd need some way to distinguish, e.g., the doctests of lib:foo and exe:foo, since we can't map both of them to doctests:foo.

• Should cabal new-test run doctests by default? I think it probably should.

[angerman]

@quasicomputational I do like your approach. Note that cabal doctest was only a fist stab at the issue, mostly driven by my disliking for custom setups.

I'm on the fence though if doctest should be considered a test-suite, if it is though we should rather have a test-suite-extra-depends instead of doctest-depends.

That said, on the whole I'm not convinced that treating doctest special in any form is the right approach anymore. I'd rather see some form of wrapping doctest into a proper cabal test-suite and treat it in a more generic fashion.

[quasicomputational]

I think I'm coming around to the idea of doctests being a full-blown test-suite of their own, with all the benefits of being able to refer to them with component syntax, use conditionals, etc. I think some kind of special treatment is inevitable, because doctesting is profoundly unlike other tests (e.g., it requires source access, it needs extra dependency information), but minimising that special treatment is a good goal.

How about with a new test-type? Something like this:

  -- Does have doctests
  ...
  haddock-example-depends:
    lens

executable foo
  -- Does have doctests
  ...

executable bar
  -- Doesn't have doctests
  ...

test-suite doctests
  test-type: doctests-1.0
  doctest-components:
    lib
    exe:foo

haddock-example-depends is, I think, the most specific name for the type of dependency incurred here, which doctest needs to know about. It ought to go on the stanza of the component under test, because it's really information about it. In theory other tools might also want to consume this information, but I'm struggling to think of concrete examples that aren't doctest shaped.

A doctest test-suite wouldn't have the usual build information fields. buildable makes sense; other-modules, cc-options, etc, less so.

doctest-components would only be able to refer to components from the same package; if it's not present I think it should default to *, meaning "all components in this package". This makes the decision to use doctest at all in a package opt-in but with a low threshold (literally two lines), which is about as nice a UX as we might hope for.

I also realised that we likely don't want special-cased time-saving mechanisms for doctests: other test suites run unconditionally, and if we give users the power to toggle doctests on and off they can come up with schemes of their own (e.g., a fast-tests flag enabled by default that turns off the doctests).

The fun thing here is that it requires practically no cabal-install changes: the knowledge about how to run test suites is in Cabal.

I can see a path to implementing this. @angerman, are you still intent on working on this, or should I put together a 1.0 myself and see what people think? This is missing the boat for 2.4, but if we can get it in for 3.0 and I can start getting rid of cabal-doctest (a wonderful hack, but still a hack), I'd be quite happy.

[phadej]

doctest-components should be singular, it will make things simpler. With common stanzas repetition can be reduced, so need for copy & paste is not a valid concern.

And yes, implementing this is simple:

• Intstall doctest as you would install it using with build-tool-depends
• prepare component (i.e. install additional dependencies, haddock-example-doctest above)
• build a command line to run doctest
• ...
• profit!

There should be a doctest-version field, cabal-doctest works around that by having doctest in build-depends. FWIW, every feature in cabal-doctest is needed by someone, so it's good experiment.

---

BONUS: for interested people, I'm working on another HACK solution cabal-doctest-cli:
https://github.com/haskell-servant/servant/compare/master...phadej:cabal-doctest-cli
It's less reliable than cabal-doctest, but doesn't require build-type: Custom, its TODO isn't that long:

• implement hardcoded parts (7.10.3)
• recognise stack, support it
• refactor

In short: it's kind of doctest using .ghc.environment files, yet we "read" the environment from plan.json (or whatever is in Stack).

[quasicomputational]

Good shout on N test suites for N doctestable components simplifying things; I'd been attracted to the ease of use of having a "doctest everything" default, but that would definitely be more complex. Also, yeah, doctest-version needs to be a thing, along with doctest-options. doctest-version needs to be a required field, right?

Can we get away without doctest-source-dirs and doctest-modules? IIUC the point of both of those is to support doctesting modules which aren't in any other component, but I'm not convinced that that's something we would want to support. Since there's a workaround (create a fake executable with buildable: false solely to doctest it), I don't think they're necessary for a 1.0; we can add them later if it's really a pain point and we decide it's sensible.

[phadej]

Yes, doctest-source-dirs + doctest-modules is used (as in cabal-doctest) only in servant to test few "this should fail to compile" things. They are a corner case we don't need support. Also they could be moved somewhere else. Even to external "test-only" package in servant, there are indeed workarounds.

[quasicomputational]

Discussion in #3788 suggests that a new test-type won't work as well as it could: versions of Cabal that don't understand it will noisily fail, but doctests not working isn't something that should break the whole package. So it'll probably be better to have a doctests stanza, since that does get ignored silently, so it can be used in packages that want to keep their cabal-version low.

[quasicomputational]

Still thinking aloud, but a the doctest binary used can't be solved as an independent component in the way that build-tool-depends or setup can: it needs to agree with the component under test about GHC and QuickCheck, at least. My hunch is that this isn't a particularly big complication: just treat doctest as an unqualified goal.

[phadej]

Why it does need to agree with QuickCheck? doctest doesn't even depend on QuickCheck.

[quasicomputational]

Oh yeah, you're right. I had it in my head that the runner binary needed to be linked against it, but it actually only touches it in interpreted code.

[erikd]

I've been having a look at this issue, and can't really figure out what a solution to this would look like.

I am currently using cabal-install from git and running cabal test in a project that uses doctests and passes the test when running stack test. When this fails under cabal, it seems to be that the tests are not being built with the packages listed in the cabal file.

[shajra]

Over the last few days I stumbled into an intersection of Cabal, doctests, and also the impact on some build ecosystems like Nix.

Since new-doctest is not yet implemented, I'd like to put a perspective out there in hopes it might have some influence. Specifically around two sentiments:

• Not many people do "does not compile tests."
• Setup.hs is so open-ended that we can not support building components one-at-a-time.

Regarding both points, I feel like there's a bias towards compromising to the legacy of what we have over giving users more features. In some ways, this feels inverted from how I see Haskell the language. But at this point, if resources were finite and fungible (definitely not fungible in real life), I'd trade off progressive language features for progressive tooling.

With respect to the argument that "does not compile tests" are limited to just a few projects like Servant, I'd like to consider that we might have far more of these if they were better supported by tooling. We're often trying to make typesafe DSLs in Haskell, and free theorems, though amazing, are never comprehensive enough in practice. To really know if something is typesafe, I need to test more than the happy path, especially with a system with as much complexity as GHC.

Regarding building components one at a time, I feel we can use all the help we can get to have a better caching story in Haskell. Build times can be really bad, which affects developer productivity more than I think we admit to ourselves. Every little step helps. I think building components one-at-a-time is one of those steps. I know what we have _is_ cached locally, but not necessarily in a distributable way (Nix- or Bazel-style).

So I agree that Setup.hs is extremely free-form. But would it be possible to at least provide a specified path forward to allowing packages to opt into a way to build components one at a time and pass artifacts amongst them? Obviously, legacy projects with extremely free-form Setup.hs files would not be ready to opt into this new world without some rework. They'd have to be compiled all in one step, as they are now. But most projects with a simple/default Setup.hs are already good to go.

So I think for me, this is in part about doctests, which I do think are great, for the reasons I mentioned above. But it's also larger than doctests for me. I'm interested in thinking about how to begin a process of constraining the freedom Setup.hs gives us, with the goal is setting us free to not only have less hacky doctests, but also any number of neat things people have the mind to do in the future.

So I haven't said much technical here. Part of that is because I think there are devils in the details that I don't know about yet, and I don't want to speak out of turn. But I would like to help where I can. Mostly I wanted to see if I could get my sentiment to have some traction. And also learn more about these details where devils are, so I can see how I can contribute best.

[shajra]

Related to Setup.hs @michaelpj makes a good point in another thread that a primary motivation for Haskell.nix compiling components separately is for cross-compilation, which I think might be a compelling motivation in addition, and beyond doctests even.

https://github.com/input-output-hk/haskell.nix/issues/388#issuecomment-570166607

[phadej]

doctests (or tests in general) in cross-compilation setting is problematic to begin with. How to run them?

[shajra]

I think @michaelpj's comment alludes to the kernel of a solution.

the setup has to be compiled for the build arch, while the other modules are compiled for the host arch.

But it's certainly work to get there. I think this is where the Haskell.nix folk have a perspective I'd be really interested in, because I think they actually cross-compile a non-trivial number of packages, and have a better sense for what went well and where there are problems. I'm not sure who in that project would be worth tagging into this conversation. Maybe they are already already watching/participating in this thread.

[angerman]

@phadej

doctests (or tests in general) in cross-compilation setting is problematic to begin with. How to run them?

that's why we have --test-wrapper 😄 You may have a way to invoke a program either through emulation or scp+ssh or something. Cabal shouldn't be aware of the details here, and a shell script sandwiched between cabal and the executable can do the trick most of the time. I think we did talk about this at some point prior? 🤔

On to the doctest issue at hand. We should acknowledge that @sol did a great job with providing an inline testing facility. By design doctest relies on ghci, and this makes it a bit special compared to other test facilities we have. Most of the haskell tests we write end up being executables that are then executed. Thus when cross compiled they end up being a regular executable for the target architecture and as such can be executed there. For doctest this would mean we need ghci on the target, which is non-trivial as it involves reading and loading object code.
We do have iserv though, and we use it extensively to get TemplateHaskell to work. Similarly we can get a remote ghci via iserv (with some limitation). And I believe @Ericson2314 has pushed most of the required changes into ghc by now. If it's not already working out of the box, I believe it's very little that would be left to fix up.

With that said, I believe we could make doctest work in a cross compilation setting (where we have iserv), by making doctest use ghci via iserv. This of couse complicates things a bit as doctest now needs to run on the build machine, unlike all other tests which would run on the target.

The most practical way forward for now would be to test native -> native, and hope for the best with cross compiation without testing. That's not very satisfactory and as layed out above I believe we have most of the infrastructure in place to make cross testing more viable; I don't see myself being able to spend much time on this this right now though.

[Ericson2314]

I envision some metaprogramming thing to extract the doctests, after which they can be compiled at run like any other test suite. Today's GHCi monster confused the metaprogramming and test running parts, breaking cross.

Even simpler than --test-wrapper is just treating benchmarks and test suites like plain executables. I don't need cabal-install to run my tests, I'll happily make another Nix derivation for that which Nix will schedule on the right sort of remote builder to run it. Problem solved!

[angerman]

Even simpler than --test-wrapper is just treating benchmarks and test suites like plain executables. I don't need cabal-install to run my tests, I'll happily make another Nix derivation for that which Nix will schedule on the right sort of remote builder to run it. Problem solved!

haskell.nix does this for Setup.hs not behaving with stdout/stderr properly (and simplicity). But there are those that don't use nix and like to use cabal (cabal-install).

[Ericson2314]

Right I do think that --test-wrapper is a good idea for precisely those reasons and I support it. I just want to convey how it is inessential and further illustrate how by separating concerns: 1) extracting the code from the docs 2) building the code 3) running the code, things become simple.