Calva: [bug] "Run all tests" doesn't give feedback

[eric-hu]

Hi there,

Thank you for making this tool! It's been a great improvement for me over other Clojure plugins I've tried.

I'm unable to successfully use "Calva: Run all tests". I'm running a Clojure service inside a Docker container and connecting to that with Calva. In the docker server logs, I can see logging related to my tests, but Calva/VSCode doesn't report on any passes or failures.

In the VSCode output for "Calva says", all I see is:

Running all tests

At the time the tests appear to stop running (the server stops logging anything), nothing more is printed out, and I don't see any visual feedback over my code (like failing tests highlighted).

Could this be an issue with my dev machine configuration? Or is this an unsupported use case?

[PEZ]

Hello!

No, this is a bug, I am pretty sure. Just that I have forgot to file the issue. But now you helped with that. Hopefully this is not too hard to fix. It used to be working, so something has moved.

[PEZ]

I might have fixed it now, @eric-hu. Please give the lates version of Calva a try.

[eric-hu]

Hi @PEZ , I just tried run-all-tests. I can see test activity in my server log, but Calva says reports No tests found. 😱, ns: 0, vars: 0

[rymndhng]

I'm also seeing issues on the latest and am encountering both issues:

1. run all tests in namespace always reports No tests found. 😱, ns: 0, vars: 0
2. run tests on save appears to executes the tests, but I don't see any output about test success/failures.

[PEZ]

Hello,

Thanks for the reports. I am sorry for the inconvenience. Will try to look at this today.

[rymndhng]

@PEZ is there any debugging info you’d like us to provide to help out?

[PEZ]

That is a super nice offer, @rymndhng!

I have not had the time to look in to this yet, but I suspect it has to do with me reverting back to a previous configuration of the tool chain for building Calva (using the :node-library target of shadow-cljs again, because I ran into too many problems using :npm-module). I think I happened to revert some conversions between Clojure and JavaScript data structures at the same time.

That was what my last fix was about, anyway, I thought I could confirm that it was all fixed, but apparently not.

Regardless of if this is the root cause or not, I think it is not all too hard to pinpoint where things go wrong. (At least not for me who wrote the code that traverses the test results data structure.) That code is in TypeScript land and we can place breakpoints in the code and examine the data. That is what I will do when I manage to free up the time slot.

If you have the energy for it, please set up the development environment and help chase the bug(s) down 😄 The [instructions] are fairly up-to-date, I think.

[PEZ]

So looking at this now a moment. Trying to make sense of it since things has seemed to be working when I use it. Here are my findings so far:

• Running all tests seems to run the tests twice, or over counting, at least I get all results doubled.
• Explicitly running tests in the namespace just reports ”no tests found”.
• Running namespace tests on save (a config setting of Calva) works.
• Re-running previously failing tests works.

So, @eric-hu and @rymndhng, we three have different results from running all tests from me. This might not be too curious if I am right about that it is the traversing of the results that is the problem. Anyway, I think I need to know a bit more about your configuration of the project. Is it possible for you to create a small project exposing the problems?

There is one thing you might not know about running all tests. Calva still has a very stupid and limited way of connecting to the nREPL server. I can't poll test results as they come in, so if you are running a batch of tests that take a long time to finish you will have to have patience before there is anything reported.

About namespace tests, I should have the clues now to start fixing the problem. With some luck I will wake up the all tests too.

[PEZ]

Well, I found a problem with running namespace tests. Fixed it. Please upgrade Calva and see if you also can run namespace tests now, please.

The problem was something completely unrelated to the parsing of test results, btw. It was something that has changed in vscode that made Calva fail in figuring out the current namespace. The reason it failed was that opening the Output pane for showing the results changed the active document (quite interesting because the output area is not really a document). This fix temporarily just skips opening up the output pane. There is a new view in vscode for showing test results anyway and Calva should be using that instead. But that is for later!

[PEZ]

The problems with running all tests remains, though, so we need to keep this issue open a while more. I recommend creating a vscode task for running all tests. Maybe even provide a problem matcher for it. If anyone does such a thing I'd be happy to include it in Calva.

[prgoes]

@PEZ Don't know if this will help, but I fired up a debugging environment for Calva and this is what I found when trying to run all tests:

• At line 111 of testRunner.ts you send a message to nREPL with something along the lines of { :op "test-all" :session [some-uuid] :load? 1}

• At line 126, however, the result you get back has something like Op = "test-all", Session = [some-uuid], Status = ["done", "unknown-op", "error"]

So, from what I could guess... it seems like nREPL is not accepting that {:op "test-all"} it's being sent?

I'm sorry I can't be much more useful but I'm not actually a Clojure developer I'm just exploring Clojure and wanted to get it running on VS Code.

EDIT:

I'm seeing mostly the same behavior as I attempt to run namespace tests.

[PEZ]

@prgoes , thanks for looking at it! What versions on nrepl and cider/nrepl are you using?

[prgoes]

@PEZ

nREPL seems to be at 0.2.12 (got it from the "lein repl" command)
cider/cider-nrepl seems to be at 0.16.0 (got it from the shadow-cljs.edn)

in shadow-cljs.edn there's also a dependency on tools.nrepl at 0.2.12.

Please, let me know if I'm looking at the wrong stuff or you need some further testing/data.

[PEZ]

I think you are looking at the right stuff. :smile:

Do you get the same results if you use cider/nrepl 0.18.0 and nrepl 0.4.5?

[prgoes]

@PEZ no success here. Just upgrading the cider-nrepl to 0.18.0 force an implicit upgrade to nrepl to 0.4.4. Didn't work then. Then I manually forced it into 0.4.5 but also with no luck. Currently, dependencies look like this:

:dependencies
[[org.clojure/core.async "0.4.474"]
[cider/cider-nrepl "0.18.0"]
[nrepl "0.4.5"]
[org.clojure/tools.nrepl "0.2.12"]]

[PEZ]

Guys, here's a VSIX package with sort of a release candidate for next Calva:
clojure4vscode-1.3.61-SNAPSHOT.zip
It has the potential to address the no-feedback issue. Please consider testing it for us. (You need to rename it from .zip to .vsix before installing in vscode.)

Before getting your hopes all too worked up, this vsix does not include a deliberate attempt to fix the bug. But it has a completely new nrepl client that resulted in a lot cleaner API for the nrepl ops and in my own testing it seems to improve the test runner. (Even if it still has issues.)

[PEZ]

OK. So this has taken an awfully long time, and I am sorry for that. However, I think I have nailed it now with the latest Calva, 1.3.63.

I recommend using cider/cider-nrepl "0.20.0".

[PEZ]

To me it seems that this bug is finally fixed. Closing it. Feel invited to open new issues should the test runner misbehave.