Haskell-language-server: Eval plugin: tests not detected in comment blocks

[tittoassini]

The plugin only recognises test starting with:
-- >>> 2+2

For consistency and convenience, as well as compatibility with existing doctest tests, it should also recognise tests in comment blocks:
{-|

2+2

-}

[bubba]

I was thinking it would be nice to get the haddocks from the ParsedModule, so we could handle cases like this, but then that would require -haddock to be turned on. Do you think it would be possible to flick it on in the DynFlags unconditionally for each session?

[tittoassini]

Why not using directly the doctest package to extract the doctests?

It is rather easy to use as a library, e.g. I forked it to generate static rather than dynamic tests (so that they could be run with ghcjs), see https://github.com/tittoassini/doctest

[bubba]

The upstream library looks a bit bare in terms of what it exports: http://hackage.haskell.org/package/doctest-0.17/docs/Test-DocTest.html

[pepeiborra]

@tittoassini it should be very easy for you or someone else to hook the Eval plugin with the doctest library. Please send a contribution! The eval plugin lives in https://github.com/haskell/haskell-language-server/blob/master/src/Ide/Plugin/Eval.hs

I don't use doctests personally, so have very little motivation to work on this.

[tittoassini]

@bubba, you are right, it is necessary to expose a few extra files to get to the required functionality from doctest. Unfortunately, that would imply a dependency on a forked version of doctest, at least for a while.

Regarding the possibility of getting the comments from ParsedModules, that's exactly how doctest does it.
It would be the safest option but might it be a little slow? I have no experience with the GHC API to judge this.
Even in this case, a further step is necessary to properly extract the tests from the parsed haddock docs.

Another solution, as done by @pepeiborra in the current plugin, is to parse directly the source code.
It doesn't seem too hard to write an extended parser in megaparsec to fully comply with doctest.

@pepeiborra , fair enough ! I am working on it.

[bubba]

Regarding the possibility of getting the comments from ParsedModules, that's exactly how doctest does it.

It would be the safest option but might it be a little slow? I have no experience with the GHC API to judge this.

Even in this case, a further step is necessary to properly extract the tests from the parsed haddock docs.

The ParsedModule is already fetched anyway during the typechecking process so plugins can request it "for free" with ghcide's machinery, from my understanding

[tittoassini]

@bubba I had a look but I am not familiar enough with the code to see how to get to the ParserModule, any example you can point me to?

[bubba]

@tittoassini I had actually started it a bit but didn't get any further, here's what I had so far:

diff --git a/src/Ide/Plugin/Eval.hs b/src/Ide/Plugin/Eval.hs
index 9b452ec..ffb264b 100644
--- a/src/Ide/Plugin/Eval.hs
+++ b/src/Ide/Plugin/Eval.hs
@@ -37,14 +37,14 @@ import           Data.Time                      (getCurrentTime)
 import           Development.IDE.Core.Rules     (runAction)
 import           Development.IDE.Core.RuleTypes (GetModSummary (..),
                                                  GhcSession (..))
-import           Development.IDE.Core.Shake     (use_)
+import           Development.IDE.Core.Shake     (useWithStaleFast, shakeExtras, runIdeAction, use_)
 import           Development.IDE.GHC.Util       (evalGhcEnv, hscEnv,
                                                  textToStringBuffer)
 import           Development.IDE.Types.Location (toNormalizedFilePath',
                                                  uriToFilePath')
 import           DynamicLoading                 (initializePlugins)
 import           DynFlags                       (targetPlatform)
-import           GHC                            (DynFlags, ExecResult (..), GeneralFlag (Opt_IgnoreHpcChanges, Opt_IgnoreOptimChanges, Opt_ImplicitImportQualified),
+import           GHC                            (hsDocStringToByteString, hsmodDecls, pm_parsed_source, DynFlags, ExecResult (..), GeneralFlag (Opt_IgnoreHpcChanges, Opt_IgnoreOptimChanges, Opt_ImplicitImportQualified), HsDecl(..), DocDecl(..),
                                                  GhcLink (LinkInMemory),
                                                  GhcMode (CompManager),
@@ -37,14 +37,14 @@ import           Data.Time                      (getCurrentTime)
 import           Development.IDE.Core.Rules     (runAction)
 import           Development.IDE.Core.RuleTypes (GetModSummary (..),
                                                  GhcSession (..))
-import           Development.IDE.Core.Shake     (use_)
+import           Development.IDE.Core.Shake     (useWithStaleFast, shakeExtras, runIdeAction, use_)
 import           Development.IDE.GHC.Util       (evalGhcEnv, hscEnv,
                                                  textToStringBuffer)
 import           Development.IDE.Types.Location (toNormalizedFilePath',
                                                  uriToFilePath')
 import           DynamicLoading                 (initializePlugins)
 import           DynFlags                       (targetPlatform)
-import           GHC                            (DynFlags, ExecResult (..), GeneralFlag (Opt_IgnoreHpcChanges, Opt_IgnoreOptimChanges, Opt_ImplicitImportQua
lified),
+import           GHC                            (hsDocStringToByteString, hsmodDecls, pm_parsed_source, DynFlags, ExecResult (..), GeneralFlag (Opt_IgnoreHp
cChanges, Opt_IgnoreOptimChanges, Opt_ImplicitImportQualified), HsDecl(..), DocDecl(..),
                                                  GhcLink (LinkInMemory),
                                                  GhcMode (CompManager),
                                                  HscTarget (HscInterpreted),
@@ -82,6 +82,10 @@ import           System.FilePath
 import           System.IO                      (hClose)
 import           System.IO.Temp
 import Data.Maybe (catMaybes)
+import Development.IDE.Core.Rules (GetParsedModule(GetParsedModule))
+import GHC (unLoc)
+import GHC (GenLocated(L))
+import HIE.Bios.Internal.Log (debugm)

 descriptor :: PluginId -> PluginDescriptor
 descriptor plId =
@@ -120,9 +124,26 @@ extractMatches = goSearch 0 . maybe [] T.lines
         p' = Position (line + spliceLength) 0
         spliceLength = length (takeWhile looksLikeSplice (l : ll))

+-- >>> 42 + 4
+-- 46
+
 provider :: CodeLensProvider
-provider lsp _state plId CodeLensParams {_textDocument} = response $ do
+provider lsp ideState plId CodeLensParams {_textDocument} = response $ do
   let TextDocumentIdentifier uri = _textDocument
+      Just fp = uriToNormalizedFilePath (toNormalizedUri uri)
+  Just (pm, posMap) <- liftIO $ runIdeAction "Eval Code Lens" (shakeExtras ideState) (useWithStaleFast GetParsedModule fp)
+
+  let decls = concatMap foo $ hsmodDecls $ unLoc $ pm_parsed_source pm
+      foo (L l (DocD _ dd)) = [(l, hsDocStringToByteString $ bar dd)]
+      foo _ = []
+      bar (DocCommentNext hsDocStr) = hsDocStr
+      bar (DocCommentPrev hsDocStr) = hsDocStr
+      bar (DocCommentNamed _ hsDocStr) = hsDocStr
+      bar (DocGroup _ hsDocStr) = hsDocStr
+
+  debugm "Eval code lens"
+  debugm $ show decls
+
   contents <- liftIO $ getVirtualFileFunc lsp $ toNormalizedUri uri
   let text = virtualFileText <$> contents
   let matches = extractMatches text

[tittoassini]

Well, isn't this a most marvellous thing, @bubba :-)?

Thanks! I shall take it from here.