Vscode: Problem matchers for error messages that span multiple lines

[massimocode]

• VSCode Version: 1.4.0-insider 01a3f8
• OS Version: Windows 10 x64

I'm unable to get a message built up from several lines in the output. In the output below, I would like the message to contain the spec name, and the reason it failed. Any ideas if this is possible?

I have the following output from NUnit:

Project dotnettesting (.NETCoreApp,Version=v1.0) was previously compiled. Skipping compilation.
NUnit .NET Core Runner 3.4.0 
Copyright (C) 2016 Charlie Poole
Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.10586
        Runtime: win10-x64
Test Files
    c:\projects\dotnettesting\bin\Debug\netcoreapp1.0\dotnettesting.dll
Errors and Failures
1) Failed : CalculatorSpec.ShouldFail
  Expected: 0
  But was:  3
at CalculatorSpec.ShouldFail() in c:\projects\dotnettesting\Calculator.spec.cs:line 12
2) Failed : CalculatorSpec.Will_Fail
  Expected: 0
  But was:  3
at CalculatorSpec.Will_Fail() in c:\projects\dotnettesting\Calculator.spec.cs:line 17
Run Settings
    WorkDirectory: c:\projects\dotnettesting
Test Run Summary
  Overall result: Failed
  Test Count: 3, Passed: 1, Failed: 2, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 2, Errors: 0, Invalid: 0
  Start time: 2016-07-22 19:16:50Z
    End time: 2016-07-22 19:16:51Z
    Duration: 0.362 seconds
Results saved as c:\projects\dotnettesting\TestResult.xml
SUMMARY: Total: 1 targets, Passed: 0, Failed: 1.

I have configured the following problem matcher:

"pattern": [
    {
        "regexp": "^\\d+\\) Failed.*$"
    },
    {
        "regexp": "^(.*)$",
        "message": 1
    },
    {
        "regexp": "^(.*)$",
        "message": 1
    },
    {
        "regexp": "^at (.*\\(\\)) in (.*):line (\\d+)$",
        "location": 3,
        "file": 2
    }
]

[dbaeumer]

It is partly possible but without actually capturing both messages. A message can only be captured once. Can you share a project showing this output. Then I can have a look why it doesn't work?

[massimocode]

@dbaeumer sure, I've just uploaded the sample project here:
https://github.com/massimocode/dotnettesting

Let me know if you need anything else.

Ideally what would be cool is if we should capture both lines relevant to the failure so that it gets reported as "Expected: 0 But was: 3", literally just replacing the new lines with spaces or something.

[dbaeumer]

@massimocode thanks for the example. I cloned it and run the task, however it produces the following output for me (which I have to admit is not parseable by the framework right now):

Project dotnettesting (.NETCoreApp,Version=v1.0) will be compiled because Input items removed from last build
Compiling dotnettesting for .NETCoreApp,Version=v1.0
Compilation succeeded.
    0 Warning(s)
    0 Error(s)
Time elapsed 00:00:01.5436183

NUnit .NET Core Runner 3.4.0 
Copyright (C) 2016 Charlie Poole
Runtime Environment
    OS Platform: Windows
     OS Version: 10.0.14393
        Runtime: win10-x64
Test Files
    p:\mseng\VSCode\Playgrounds\bugs\dotnettesting\bin\Debug\netcoreapp1.0\dotnettesting.dll
Run Settings
    WorkDirectory: p:\mseng\VSCode\Playgrounds\bugs\dotnettesting
Test Run Summary
  Overall result: Passed
  Test Count: 3, Passed: 3, Failed: 0, Inconclusive: 0, Skipped: 0
  Start time: 2016-08-15 10:22:13Z
    End time: 2016-08-15 10:22:13Z
    Duration: 0.233 seconds
Results saved as p:\mseng\VSCode\Playgrounds\bugs\dotnettesting\TestResult.xml
SUMMARY: Total: 1 targets, Passed: 1, Failed: 0.

I am using dotnet --version: 1.0.0-preview2-003121

[massimocode]

@dbaeumer Apologies for that, the code I checked in had passing unit tests (so no errors to report).

I have since updated the repository to have the same 2 failing tests as in the original issue description.

The repository is here:
https://github.com/massimocode/dotnettesting

Thanks!

[dbaeumer]

I will have another look then. Thanks.

[indiejames]

I need this same functionality (capturing a message across multiple lines). I also have output from my tests like

Expected: 0
But was: 3

that I would like to capture as the 'message'.

[dbaeumer]

@indiejames we are working on making tasks API so that people can code this. Having a declarative approach to this requires me to invent a programming language in JSON :-(

[doudou]

@dbaeumer do you have an idea how high in the priority list a reporting API is ? We're comparing Atom with VSCode, and I really like VSCode. But on this front, the atom build plugin & problem reporter are light years ahead...

[dbaeumer]

It is OnDeck but not planned for a specific milestone.

[jpihl]

I had the same issue with gtest. I found a workaround which might help some people until this feature is available.

Here's my problemMatcher for gtest:

"problemMatcher": {
    "owner": "cpp",
    "fileLocation": ["relative", "${workspaceRoot}/test/src/"],
    "severity": "error",
    "pattern": [{
        "regexp": "^(.*):(\\d+):\\s+.*$",
        "file": 1,
        "line": 2
    },
    {
        "regexp": "^((:?Value of:.*)|(:?  Actual: .*)|(:?Expected: .*))$",
        "message": 1,
        "loop": true
    }]
}

I'm on linux so I can't build your test project, but I bet it can be done in a similar way.

The drawback of this approach is that it will produce a problem per line in the problem message, but you get the error message(s) when you hover over the line.

An excerpt of the output from my test:

...
[----------] 2 tests from test_copy
[ RUN      ] test_copy.same_size
../../test/src/test_copy.cpp:25: Failure
Value of: storage::is_equal(storage::storage(d1), storage::storage(d2))
  Actual: false
Expected: true
[  FAILED  ] test_copy.same_size (0 ms)
[ RUN      ] test_copy.not_same_size
[       OK ] test_copy.not_same_size (0 ms)
...

[AshleyYakeley]

This is a pain point for Haskell users, too. Here's a typical multi-line error from GHC:

    /home/ashley/Projects/Haskell/Truth/pinafore/lib/Pinafore/Language/Read.hs:253:28: error:
        • Could not deduce (ToTypeF
                              (Pinafore.Language.Type.PinaforeType baseedit)
                              Pinafore.Entity.Predicate)
            arising from a use of ‘qConstExpr’
          from the context: HasPinaforeEntityEdit baseedit
            bound by the type signature for:
                       readExpression3 :: forall baseedit.
                                          HasPinaforeEntityEdit baseedit =>
                                          Parser (PinaforeTypeCheck (QExpr baseedit))
            at lib/Pinafore/Language/Read.hs:(210,1)-(212,50)
        • In the second argument of ‘($)’, namely ‘qConstExpr p’
          In the second argument of ‘($)’, namely ‘return $ qConstExpr p’
          In a stmt of a 'do' block: return $ return $ qConstExpr p
        |
    253 |          return $ return $ qConstExpr p) <|>
        |                            ^^^^^^^^^^^^

The best I can do with the current problemMatcher is capture the first line of the full message (• Could not deduce (ToTypeF).

[AshleyYakeley]

If I wanted a quick fix for this, I'd create an append boolean JSON key similar to loop, but appending the message to the current problem instead of outputting a new problem.

[AshleyYakeley]

So it looks like the UI work done in 1.31 on multi-line messages is part of a fix to this issue, but providing an API is still to be done, is that correct?

[alexr00]

The UI work will be useful for this, but yes, there is still Task API that will need to be added.

[cmeiklejohn]

If I wanted a quick fix for this, I'd create an append boolean JSON key similar to loop, but appending the message to the current problem instead of outputting a new problem.

Did you ever come up with a good solution for this? I'm using Koka and I've got similar issues with multi-line error messages. I'm fine with capturing them only for the hover output, but I can't seem to write a problem matcher that captures them correctly.

Here's a typical error:

/mnt/c/Users/chris/Documents/GitHub/koka-example/src/test.kk(3,14): error: Invalid type
  type context : fun test() : io {
                 }
  type         :              io
  inferred kind: E
  expected kind: V
  because      : The result of a function type must be a type

My current matcher is good enough for capturing the first line:

"pattern": [
                    {
                        "regexp": "(.*?)[(](\\d+),\\s?(\\d+)[)]: (error|warning): (.*)",
                        "file": 1,
                        "line": 2,
                        "column": 3,
                        "severity": 4,
                        "message": 5
                    }
                ]

...but, ideally I'd like to capture the remaining lines in the hover output. Is there any easy way to do this?

[AshleyYakeley]

Currently there is no way to do this, I believe.

[benny-medflyt]

It would be nice if visual studio code problemMatcher could parse a "universal" JSON format. so you could configure your task as:

"pattern": "json"

and then the output of the program would be expected to be (single-line) JSON lines such as:

{"message": "Unknown variable pictur", "location": {"line": 4, "col": 12}, "severity": "error"}

Since it is JSON, the "message" field can contain "\n" characters for errors that span multiple lines. And everyone can write their own custom filter scripts (in any language they want) that will convert the output of their favorite program into this universal JSON output format.

As a bonus, this format can be extended in the future. For example, for the hypothetical error above, my compiler might give me a hint such as: "perhaps you meant picture". This could result in a "quickfix" field such as:

{ ..., "quickfix": [{"title": "Rename to picture", "replacementLocation": {"line": 4, "startCol": 12, "endCol": 18}, "replacementText": "picture" }]}

Many tools already have a JSON output format (for example tslint --format json), VSCode could help by encouraging a standardized format

[ncannasse]

+1 for JSON support for problems matcher, or any other way to solve this issue that's been around for many years ;)

[alexr00]

This would be solved by programmatic problem matching. Duplicate of https://github.com/microsoft/vscode/issues/59337

[vscodebot[bot]]

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for existing issues here. See also our issue reporting guidelines.

Happy Coding!