Stack: Run specific test suites defined in the cabal file

[jdnavarro]

I usually have multiple test suites defined in a cabal file, for example one for tasty and another one for doctests. The current default for stack test is exactly what I would expect, but I'd be nice to be able to run specific test suites typing smoething like stack test doctests or stack test --suite doctests.

Notice that cabal-install already does this, i.e: cabal test doctests.

[berdario]

I just stumbled onto the same issuei.

For the record, stack test foo will select package foo... as can be seen from the --help

TARGET If none specified, use all packages

This package afaik should be the same one specified in the stack.yaml... which would be useful to restrict the tests that you want to run, but in a different way from the one used by cabal test. This probably means that implementing stack test suite won't be doable (--suite sounds just fine)

Frankly, the only time I ever wrote Haskell tests before now, I wrote a simple script and run it with runhaskell... no cabal goodness. So I'm still looking around for the best way to structure stuff in my projects.

Right now, my solution is to use tasty, and a single main for it... and in the test/ directory I'll keep multiple modules (for integration tests, unit tests, etc...)
Then, if I want to run just a single kind of test, I'll just use the tasty -p (pattern) option, like this:

stack test --test-arguments '--hide-successes -p dummy'

This doesn't seem to be the most stylish way to run tests, but it should let me do everything I want

Then again, I'm not sure what would be the advantage of having inside the .cabal a test-suite vs an executable section, if I would use the test-suite just like the latter... maybe it's just the fact that stack install and cabal install would only install stuff marked as executables for the end users?

Maybe suggestions for a better setup (how to split up tests into integration/unit tests, etc.) might be helped by a new/more complete template/project-skeleton created with stack new? (if the solution is to rely on hspec/tasty/whatever, maybe it could make sense to make one of these the "blessed" alternative?)

[jdnavarro]

I'm also not sure what would be the ideal way to structure tests. My current setups are driven by practical limitations. For example, doctests can't be run from tasty, otherwise I'd try to stick with just one tasty executable.

In my case, the other use case for using different test executables is when running quick but essential tests vs slow but exhaustive ones, even if tasty is used to create both test runners. A custom argument in tasty could be used or even a cabal flag, but I just find it more straightforward to define 2 different test executables. Maybe this use case is better addressed from tasty and with the right template than having stack deal with several test suits. I have to think about what would be the easier way from tasty only.

I still would like to have a separation between application executables an test executables. As you mention, I think it's useful for end users.

Right now, I'm not considering any alternative to tasty for running tests but I'm not sure if that's the consensus in the community. Do hspec users run their tests using tasty, too? When using hspec in standalone mode, do they use it exclusively or write tasty tests separately?

[mgsloan]

Y'all are in luck! This is already supported, but should probably be documented better in the commandline help. To run a specific test suite, do stack test PKG:TESTSUITE. For example, stack test conduit:test

[jdnavarro]

Cool!

I was about to make a PR extending the documentation for the TARGET metavar, but I'm not sure where to document this feature so that it doesn't create confusion in other subcommands. For example, stack ghci PKG:TESTSUITE is not supported.

[mgsloan]

Here is some recent and relevant discussion of how targets are going to be handled: https://github.com/commercialhaskell/stack/issues/651#issuecomment-129094820

This change will likely address this issue, and stack supports the main issue topic, so closing. Thanks for thinking to make a PR! Good point about the effects of that change on the doc for other commands.