Dredd: Machine-readable reporter

[honzajavorek]

...or a different easily machine-readable format. Would significantly simplify writing some integration tests (such as those in #338). Filing this issue so people can add other use cases - probably not worth implementing just because of couple of tests.

[netmilk]

Good idea I think. Might improve the integration possibilities. +1

[prasanna981]

+1
Would be useful to add something like this https://www.npmjs.com/package/newman#json-reporter-options. This would make it easier to integrate with CI tools

[honzajavorek]

I think this should be implemented using the TAP protocol: https://testanything.org/

[honzajavorek]

Let's do this. The idea is:

• Let's only keep the default CLI reporter, Apiary reporter, and let's add TAP reporter. Something like this.
• Users could use --reporter=cli, --reporter=apiary, or --reporter=tap to indicate which reporter they want to use. They should be able to somehow use multiple reporters (e.g. to see CLI output and to produce xunit file at the same time), on both *NIX and Windows. It needs investigation whether this is possible with TAP. We also need to write docs on this with ready-made examples for *NIX and Windows. And we should test the basic cases. This is gonna be a breaking change for Dredd, so let's have some solid "migration docs" in place when the version hits the npm. The --output option would possibly disappear.
• We need to think about capturing server output - https://github.com/apiaryio/dredd/issues/593 - and how does it play together with TAP as the server output is obviously an additional noise to TAP. Also, what about hooks logging - hooks.log, console.log, and hook runners stdout?
• Users then could use any TAP tools to do whatever they want - xunit, dots, HTML, nyan cats, ... We will maintain less reporters. Dredd's E2E tests could benefit from Dredd having a machine-readable output. That's win-win-win.
• Maybe we could use tape to ease our work, needs investigation.
• We could add Dredd to this page and here.
• Reworking Dredd's E2E tests to use TAP instead of parsing the CLI reporter output allows us to revamp the CLI reporter - https://github.com/apiaryio/dredd/issues/765 - in the future without consequences to E2E tests.

Open questions

Following needs to be answered before we do this:

• [ ] Is TAP a good idea at all?
  
  
  
  • https://github.com/avajs/ava/issues/920#issuecomment-304694295
  
  
• [ ] Do current Dredd reporters have working equivalents on npm?
  
  
  
  • dot - https://github.com/scottcorgan/tap-dot
  
  
  • html - https://github.com/gabrielcsapo/tap-html, https://github.com/Quobject/tap-to-html (ugly)
  
  
  • markdown - https://github.com/Hypercubed/tap-markdown (insecure, outdated deps)
  
  
  • nyan - https://github.com/calvinmetcalf/tap-nyan
  
  
  • xunit - https://github.com/aghassemi/tap-xunit
  
  
• [ ] How to replace multiple reporters support?
  
  
  
  • Looks like TAP doesn't support anything like that. User would need to save an intermediate file 😕 We would essentially lose multiple reporters support out of the box.
  
  
• [ ] How would Windows support look like?
  
  
  
  • According to searching the internet, Windows shell should support | > < >> << redirections, I did not expect that. Thus this is probably a non-issue, I'd test it though.
  
  
• [ ] How to handle server and hooks/hook handler output?
  
  
  
  • Capture & let the user control whether it's outputted or not. Possibly output to stderr alongside the stdout TAP (that's what AVA seems to do).
  
  

Resources

• https://testanything.org/
• https://github.com/sindresorhus/awesome-tap
• https://www.node-tap.org/tap-format/

[honzajavorek]

@garyluu @juliengerboud @mlambley I noticed you are using the HTML reporter. What is your use case for it? Why do you prefer it?

[honzajavorek]

Thinking about this further:

• Although the TAP protocol is vague and underspecified, it's superset is pretty well described here. We could make the TAP reporter according to that spec and I think everything important could fit there.
• dot, xunit, nyan reporters can be easily provided by tap- libraries.
• Both the html and markdown reporters we have right now are so low-quality I'm actually surprised anyone is using them. The HTML tap- libraries are better than what we have, so let's recommend them. I could not find a single person in Dredd issues using the markdown reporter. I wouldn't be very sad to deprecate it without replacement. After all, anyone is welcome to build whatever they like on top of TAP, the data are going to be there.

So the remaining issue is how to support multiple reporters. I've seen quite a few people here to use Dredd with dot reporter together with apiary or html reporters, as in

as a Dredd user I can see whether anything failed in the console and I can go to the file to see the verbose output

TAP steals stdout, so that wouldn't be possible. And the stderr would be probably ocuppied by redirected server and hooks output... 🤔

[garyluu]

didn't end up using the html reporter, but i was trying to create a web link that quickly displayed the results

[michalholasek]

Regarding the stdout noise of parent process coming from server and hook instances - it could be good idea to leverage subprocess.send() method to handle incoming information from server/hooks and decide in main Dredd's process what should go to stdout instead of just writing directly to main process' stdout from server/hook processes.

[juliengerboud]

@honzajavorek The idea was to "store" the report as an artifact of TeamCity build. This way, users checking a build creating the report would just had to click on the file and the report would have been display directly into the browser.

[honzajavorek]

With @michalholasek we designed following steps to take:

1. Use debug for Dredd application logging. Separate application logging from the reporters output completely.
   
   
   
   • Outputs debugging info only if asked.
   
   
   • Fixes brievity of the dot reporter (https://github.com/apiaryio/dredd/issues/593), the --names option, and other parts of Dredd.
   
   
   • Enables https://github.com/apiaryio/dredd/issues/765 in the future.
   
   
2. Add the TAP reporter alongside current reporters and add respective tests and docs. Add deprecation warning to reporters intended to be replaced by TAP with a link to this issue and a call for comments.
3. Let's see whether TAP covers all use cases for people. If yes, let's remove the other reporters in the future. If not, let's design the solution together with interested users.

Regarding the server/hooks output, we can capture it as suggested in https://github.com/apiaryio/dredd/issues/593#issuecomment-251946163 and put it into special TAP comments as part of the stdout. The stderr would be reserved for output of the debug utility.

[honzajavorek]

@garyluu @juliengerboud Thanks for sharing! Did you need the CLI output as well or was the HTML file sufficient for you? I mean, if you run Dredd like you described, did you care about what it outputs to the console at all?

[garyluu]

The last time I tried, I needed the CLI output as well (in order to get the summary):
https://github.com/apiaryio/dredd/issues/1001

[juliengerboud]

@honzajavorek The CLI log is quite useful! It is easier to read IMHO.

[mlambley]

It's been... quite a while, but I'll try to remember my use case. I wanted the cli to give brief information (a single line for each test, showing perhaps method, url, result). The problem I had was that the cli reporter was cluttered with json dumps of failed tests. There were two options for the cli reporter: show errors inline or show errors at the end. I wanted the option to not show errors at all.

The dot reporter wasn't ideal either because you couldn't see the test information (and if memory serves, it still dumped json data anyway).

A HTML file is a much easier place to review long strings of json and identify where the errors are. I wanted the cli reporter to give summary info, with detailed information in the html.

I don't know how TAP will fit into this, but my suggestion was to have an interface for creating custom reporters and be able to configure dredd to say "read extra reporters from this directory". I think another suggestion I had was to create a "quiet cli reporter" which was a copy of cli reporter with all of the "show errors" code removed.