Haskell-ide-engine: Get tests working on windows

[bubba]

Currently stack test fails on windows due to a plethora of errors, so it has been turned off in our AppVeyor builds. (haskell-lsp-test will probably need to be fixed for it)

[fendor]

Do the tests explicitly fail or not terminate? Because currently they do not terminate.

[bubba]

I am not sure. As far as I'm aware lsp-test builds but doesn't work at runtime on Windows https://github.com/bubba/lsp-test/issues/28

[jneira]

I am taking a look at this, in both local and appveyor env (with this branch: https://github.com/jneira/haskell-ide-engine/blob/appveyor/appveyor.yml).

[alanz]

@jneira There is also an effort to get this working in Azure, which may be the preferred long-term CI.

See https://github.com/haskell/haskell-ide-engine/pull/1238

[jneira]

@alanz thanks for pointed it out!

[jneira]

Hi, i am starting to analyze the tests in windows, and the unit ones are failing for me:

Failures:

  test/unit\LiquidSpec.hs:50:7:
  1) Liquid, liquid haskell plugin, gets annot file paths
       expected: "D:\\dev\\ws\\haskell\\haskell-ide-engine\\test/testdata/liquid/.liquid/Evens.hs.vim.annot"
        but got: "D:\\dev\\ws\\haskell\\haskell-ide-engine\\test\\testdata\\liquid\\.liquid\\Evens.hs.vim.annot"

  To rerun use: --match "/Liquid/liquid haskell plugin/gets annot file paths/"

  test/unit\LiquidSpec.hs:55:5:
  2) Liquid, liquid haskell plugin, reads errors from json file
       uncaught exception: IOException of type NoSuchThing
       D:\dev\ws\haskell\haskell-ide-engine\test\testdata\liquid\.liquid\Evens.hs.json: openBinaryFile: does not exist (No such file or directory)

  To rerun use: --match "/Liquid/liquid haskell plugin/reads errors from json file/"

  test/unit\LiquidSpec.hs:94:5:
  3) Liquid, liquid haskell plugin, reads types from vim.annot file
       uncaught exception: IOException of type NoSuchThing
       D:\dev\ws\haskell\haskell-ide-engine\test\testdata\liquid\.liquid\Evens.hs.vim.annot: openFile: does not exist (No such file or directory)

  To rerun use: --match "/Liquid/liquid haskell plugin/reads types from vim.annot file/"

  test\unit\LiquidSpec.hs:111:7:
  4) Liquid, liquid haskell plugin, reads types from vim.annot file 2
       uncaught exception: IOException of type UserError
       user error (Pattern match failure in do expression at test\unit\LiquidSpec.hs:111:7-13)

  To rerun use: --match "/Liquid/liquid haskell plugin/reads types from vim.annot file 2/"

I have liquid and z3 installed and i exeuted the test in mingw64

[jneira]

Not sure if it is related but liquid test/testdata/liquid/Even.hs fails with

realgcc.exe: fatal error: no input files
compilation terminated.
LiquidHaskell Version 0.8.2.4
Copyright 2013-18 Regents of the University of California. All Rights Reserved.

liquid.exe: `gcc.exe' failed in phase `gcc'. (Exit code: 1)

[jneira]

I've tried to build liquidhaskell from source but i've got another error, opened https://github.com/ucsd-progsys/liquidhaskell/issues/1527

[jneira]

In the bright side, wrapper-test, dispatcher-test and plugin-dispatcher-test suites pass all tests

[jneira]

After installing z3 from https://github.com/Z3Prover/z3/releases/download/Z3-4.8.5/z3-4.8.5-x64-win.zip
liquidhaskell is able to generate its files under .liquid and the failures of unit-test have change to:

Failures:

  test/unit\LiquidSpec.hs:50:7:
  1) Liquid, liquid haskell plugin, gets annot file paths
       expected: "D:\\ws\\haskell\\haskell-ide-engine\\test/testdata/liquid/.liquid/Evens.hs.vim.annot"
        but got: "D:\\ws\\haskell\\haskell-ide-engine\\test\\testdata\\liquid\\.liquid\\Evens.hs.vim.annot"

  To rerun use: --match "/Liquid/liquid haskell plugin/gets annot file paths/"

  test/unit\LiquidSpec.hs:63:7:
  2) Liquid, liquid haskell plugin, reads errors from json file
       expected: LP {line = 9, column = 8}
        but got: LP {line = 9, column = 12}

  To rerun use: --match "/Liquid/liquid haskell plugin/reads errors from json file/"

  test/unit\LiquidSpec.hs:100:7:
  3) Liquid, liquid haskell plugin, reads types from vim.annot file
       expected: [LE {start = LP {line = 1, column = 1}, stop = LP {line = 1, column = 1}, message = "GHC.Types.Module"},LE {start = LP {line = 6, column = 1}, stop = LP {line = 6, column = 10}, message = "[{v : GHC.Types.Int | v mod 2 == 0}]"}]
        but got: [LE {start = LP {line = 1, column = 1}, stop = LP {line = 1, column = 1}, message = "GHC.Types.Module"},LE {start = LP {line = 6, column = 1}, stop = LP {line = 6, column = 10}, message = "[{VV : GHC.Types.Int | VV mod 2 == 0}]"}]

  To rerun use: --match "/Liquid/liquid haskell plugin/reads types from vim.annot file/"

  test/unit\LiquidSpec.hs:112:7:
  4) Liquid, liquid haskell plugin, reads types from vim.annot file 2
       expected: [LE {start = LP {line = 1, column = 1}, stop = LP {line = 1, column = 1}, message = "GHC.Types.Module"},LE {start = LP {line = 6, column = 1}, stop = LP {line = 6, column = 10}, message = "[{v : GHC.Types.Int | v mod 2 == 0}]"}]
        but got: [LE {start = LP {line = 1, column = 1}, stop = LP {line = 1, column = 1}, message = "GHC.Types.Module"},LE {start = LP {line = 6, column = 1}, stop = LP {line = 6, column = 10}, message = "[{VV : GHC.Types.Int | VV mod 2 == 0}]"}]

  To rerun use: --match "/Liquid/liquid haskell plugin/reads types from vim.annot file 2/"

The results are very close and the first one seems to be easily fixed normalizing paths.

[jneira]

The other errors maybe are caused by the liquidhaskell version i had to use in windows (built from master), the actual version used by hie is the lastest one in hackage (0.8.2.4) and it fails for me with https://github.com/ucsd-progsys/liquidhaskell/issues/1177

[jneira]

Well, the hard part is the func-test suite, that doesn't terminate. I've executed it killing the hie proccess each time it stalled. The output is here.
The resume is that all failing tests makes the hie proccess hang (or the other way around?).
Some of them logs an uncaught exception but all reach the predicate being tested:

• Failing test
  
  
  
  • [1] FunctionalCodeActions > rename suggestions > works
  
  
  • [2] FunctionalCodeActions > rename suggestions > doesn't give both documentChanges and changes
  
  
  • [3,4,5,6,7,8,9] FunctionalCodeActions > import suggestions > formats with brittany
  
  
  • [10,11,12,13,14,15,16] FunctionalCodeActions > import suggestions > formats with floskell
  
  
  • [17,18,19] FunctionalCodeActions > add package suggestions
  
  
  • [20,21] FunctionalCodeActions > redundant import code actions
  
  
  • [22] FunctionalCodeActions > typed hole code actions > works
  
  
  • [23] FunctionalCodeActions > unused term code actions > Prefixes with '_'
  
  
  • [24] FunctionalLiquid > liquid haskell diagnostics > runs diagnostics on save, with liquid haskell
  
  
• Failing test with explicit exceptions:
  
  
  
  • [5,6,8,9] FunctionalCodeActions, code actions, import suggestions, formats with brittany
  
  
  • [10,12,13,15,16] FunctionalCodeActions > import suggestions > formats with floskell
  
  

So next steps could be:

• Investigate and fix the hie hangs
• Maybe check if the failing test features really work in windows using a editor
• Fix the bugs!

[jneira]

I've got to run succesfully the unit-test suite, installing liquidhaskell-0.8.2.4 and z3-4.8.5-x64-win in windows 10. I had to to make ghc-8.2.1 use gcc of ghc-8.4.2 to get it working (see https://github.com/ucsd-progsys/liquidhaskell/issues/1177#issuecomment-506667899). I am afraid it will be difficult to apply in ci.
So only left func-test!

[jneira]

I've tested manually in vscode some of the func-test fails and i am afraid they are legitimate bugs in hie itself (and no errors caused by the hang or internal error of the test suite):

• Rename suggestion deletes the actual symbol but no replaces it with the correct one
• Import suggestion adds Import module Control.Monad (<0>when) instead Import module Control.Monad (when)
• ...

I'll try first to avoid the hang and then i'll fill issues for each bug (or one for all of them)

[fendor]

The second should be easy to fix, I think I can provide a bug fix for it soon.

[jneira]

I've get to avoid no termination on failing tests changing lsp-test: https://github.com/bubba/lsp-test/pull/44