Stack: Unhappy about `stack test --test-arguments`

[benma]

Hi

I wonder if there is a way to improve the long winded way to pass test arguments.

It took me a good 10 minutes including my best google-fu to find out how to run only a subset of my hspec tests (I am new to hspec). The end result is:

stack test --test-arguments "-m <filter>"

This is pretty long-winded, and when I return to using Haskell and Hspec after some weeks, I have to look it up again. Repeating this in custom bash functions or Makefiles is also not very nice.

I think Haskell as a whole would benefit if common operations like this would be easier to type and remember. Take Go, which focuses on simple usage and batteries included:

go test -run <filter>

Maybe stack could have specific subcommands for testing frameworks like hspec, so one could run stack hspec -m <filter> or similar (and all other args that hspec is taking

I am aware that stack might not be the optimal place to cater to individual packages, but on the other hand, stack is more and more the goto tool for Haskell development, and I think it is worth including more batteries to make Haskell and everything around it even easier to use.

Thoughts?

[mgsloan]

I agree, I've often thought the same.

I can think of a couple modifications that would make it quite short:

• stack test -- -m <filter> - this would cause stack test to be more than just an alias for stack build --test. It would also give a meaning to what comes after --, as used in stack exec -- cmd.
• This might be a crazy idea, but what if stack test "-m <filter>" worked?

[benma]

What about a combination of your two proposals? Every flag that does not belong to stack test gets passed as test arguments, and also everything after --. If there is a clash of stack test flags and test arguments, you can move it after --, but for most flags, that shouldn't be a problem.

That might be a bit hacky as it is not forwards compatible, so I am would also be happy if only your first proposal would be implemented.

Apparently, @gregwebs also had the same idea: https://github.com/commercialhaskell/stack/issues/172#issuecomment-111135123. There is some discussion on it.

[gregwebs]

@benma The idea of automatically passing flags to tests that do not belong to stack test creates fuzzy semantics. Now I have to remember what flags belong to stack test and understand collision behaviour where everything works but my flag doesn't have the effect I thought it would. When you upgrade stack and a new flag is added, your test script can break.

[gregwebs]

I am also unhappy with --test-arguments because now I have to deal with quoting things (that could need quotes!). It was decided that there was a principal at stake that must be interpreted a certain way and as a consequence it destroyed all ease of usability. Nothing bad will happen from using -- and as per my last comment it makes CLI semantics very clear, and it is very easy to type.

[mgsloan]

Yup, I'm in favor of implementing the stack bench -- args and stack test -- args approach.

[benma]

Great. Sorry I am new here, is this now greenlighted, or are there other people who need to vote?

Yup, I'm in favor of implementing the stack bench -- args and stack test -- args approach.

Would this replace --test-arguments or would we keep that for backwards compatibility? I.e. provide the flags using either style, but not both?

[gregwebs]

Have to keep it for backwards compatibility. Either merging the 2 or throwing an error when they are both used seems fine to me.

[mgsloan]

Yup, we should keep both anyway so that you can do stack build --test --bench --test-arguments a --bench-arguments b.

Sorry I am new here, is this now greenlighted, or are there other people who need to vote?

Consider it greenlighted! We don't really have a set process for that, I'd say the amount of discussion and consensus required is related to the overall size and impact of the change. Of course, people should feel free to disagree with a particular decision and discuss it.

For a backwards compatible CLI convenience with little downside, I feel quite comfortable saying we should do it. The discussion in #172 isn't "let's not do the -- thing", instead it's "Lets do this reasonable thing and circle back to the -- later". Now we are circling back :)

[benma]

I can think of a couple modifications that would make it quite short:
stack test -- -m - this would cause stack test to be more than just an alias for stack build --test. It would also give a meaning to what comes after --, as used in stack exec -- cmd.

@mgsloan, @gregwebs unfortunately, this doesn't work as smoothly as expected. The problem is that stack test [TARGET] takes a list of targets, and everything after -- is treated as positional arguments by optparse-applicative.

From optparse-applicative:

However, parsers always accept a special argument: --. When a -- is found on the command line, all the following words are considered by argument parsers, regardless of whether they start with - or not.

So stack test target1 target2 -- --foo would result in the targets ["target1", "target", "--foo"], leaving nothing for the test arguments.

The trick with the -- delimiter works in stack exec more or less by accident, because stack exec has only one positional argument, so it's unambiguous.

I am still thinking about what can be done about this, but so far, I couldn't come up with anything nice. Maybe we can add a patch to optparse-applicative which enables to separate positional arguments before and after --, if not already possible.

Thoughts?

[benma]

Huw Campbell, maintainer of optparse-applicative, seems to have faced the same problem, and chose to pass along all unrecognized flags.

http://www.huwcampbell.com/posts/2017-06-09-announcing-optparse-0.14.html

Fixing this issue inspired me to also alleviate a pain point which my colleagues and I had been having regarding mixing arguments and wrapping commands. At Ambiata, we created a build tool called mafia which acts as an opinionated wrapper around cabal, providing a nix like package cache, safe sandboxing, and submodule handling amongst other things.

When using it though, to pass an option to cabal underneath, the user would need to first use -- to tell optparse to treat all remaining options as arguments, and the argument list would be sent to cabal. Now, with the forwardOptions builder attached when creating the parserInfo structures, any unrecognised options to mafia can be passed through as is. I believe this will actually help in quite a few places where people are using mixed environments

@HuwCampbell I'd appreciate your thoughts, as you seem to have put in a lot of thought around this issue.

Related: https://github.com/pcapriotti/optparse-applicative/issues/269

[benma]

Another thing we could do is to use a different separator than --, in which case it could easily be disambiguated. To be consistent, this would need to be used also in stack exec etc., so it might be not a welcome change to everyone who already trained their muscle memory.

[HuwCampbell]

That's possible, to get it to work well though (collecting flags as well after it was mentioned) you would have to use a subparser which only matches it in the front of an alternative which holds the arguments which come first (it would need an OnlyArguments policy as well. It would be an ugly hack.

[benma]

Ugly indeed!

[kirelagin]

Oh my god, --test-arguments is so terrible, I have been fighting with it for an hour or so already. I am trying to do the simplest thing possible – a shell script that does some setting up and then calls stack test passing all its arguments to the test runner. I’m starting to think it’s almost impossible to make it work reliably with spaces.

After fighting with bash (😠) I settled on "${*@Q}" which is a string with each individual argument helpfully quoted by bash. I was 95% this would work, but, apparently instead of simply passing the single --test-arguments argument to the shell, stack tries to interpret it itself or god know what happens, but the problem is that bash’s @Q quotes arguments by single quotes and then #3091 happens.

It’s starting to look more and more like I’ll have to write an actual Python script that will be requoting the arguments. Wow.

[kirelagin]

In case it will help anyone, I ended up doing:

# Wrap every argument in single quotes, because that’s what stack wants :/
args=( "${@/#/\"}" ); args=( "${args[@]/%/\"}" )
stack test --test-arguments "${args[*]}"

I doubt it will work reliably, especially will all kinds of different quotation marks, and whitespace, and glob expansion, etc., but it seems to do what is expected in our use cases (i.e., mostly, --match with spaces).

[tuomohopia]

Sooo... what's the way to do this now? Say I want to match a test like this it "POST /batteries/edit/:id edits the battery when authorized" $ ..., how would I go on about this?

I'm on WIN10.