Metals: Worksheet mode

[olafurpg]

It would be nice if there was a "worksheet mode" that allows you to interactively evaluate toplevel statements inside the editor and see the results.

Some challenges:

• how to extract static types and runtime values? We could build on top of the Scala REPL or mdoc (https://scalameta.org/mdoc/)
• what will be the JSON-RPC protocol? We could maybe build on top of http://dotty.epfl.ch/docs/usage/worksheet-mode-implementation-details.html or use the Debug Adapter Protocol
• how to display values in the editor? We can use VS Code decorations https://code.visualstudio.com/api/references/vscode-api#TextEditorDecorationType but that approach may not work well with multi-line output.

[ekrich]

This feature would be killer for adoption. Worksheets worked pretty darn good in Scala IDE and some people are still hanging on to the tool because of that feature. Everyone wants to have worksheets like the kind you find for Python which is a big reason for the number of people using the language. These tools that make it easy for the masses are the key to greater adoption for Scala.

[olafurpg]

Adding a few notes from offline discussions with several people including @jvican, @baccata, @alexarchambault

(Brainstorming) We could potentially support "worksheets" in several different forms:

• *.notebook.sc: which will have Jupyter notebook semantics where cells are separated by some magic //%% marker comments (like in the python vscode extension). I believe notebook semantics are desirable for use-cases where you have an expensive fixture that you only want to evaluate once (like setting up a database) and then want to iterate quickly after that. Another use-case is media objects (graphs, plots, images, ..).
• *.worksheet.sc: which will have Intellij/Scala IDE semantics where the entire script is evaluated in one run (no cells) and we place decorators next to each val/var showing the type + runtime value (similar to mdoc).
• *.sc: regular ammonite scripts that are evaluated by delegating to some command like amm --watch .....
• *.scala.md: (just an idea) would be nice if it was possible to write markdown files and have the scala code fences evaluated worksheet-style.

We wouldn't have to implement all of these, I'm just exploring the possible solution space.

Ammonite scripts

It would be nice to support $ivy and $file from ammonite scripts. This would require something like a new --compile-only flag to ammonite that compiles a script without evaluating it. The --compile-only mode would need to emit the following information:

• dependency classpath: the full classpath excluding the main script file
• scalac options (partial unification, etc.)
• (not sure but might be necessary): list of fully-qualified names of classes generated by $file imports.

We need the classpath and scalac options in order to provide completions/type-at-point/parameter-hints.

To show the results of evaluating an ammonite script, we could

• use the Debug Adapter Protocol (DAP) and print output in the "Debug Console", similarly to how run and test will work.
• use the Language Server Protocol (LSP) and print output using window/logMessage.
• do nothing at all, let the user run amm foo.sc from their favorite terminal.

Notebooks

Almond has a nice library API ScalaInterpreter which has a execute method that where we could run individual cells. https://github.com/almond-sh/almond/blob/adb2840e69f63ef6ff25c375b04d7d50c97493a7/modules/scala/scala-interpreter/src/main/scala/almond/ScalaInterpreter.scala#L109-L114. The benefit of this approach is that the user wouldn't need to install almond separately but at the cost of requiring custom development both in the Metals server side and editor client side.

The Python VS Code extension implements in TypeScript on the client side the splitting of code cells via magic #%% comments and submits the code to a running Jupyter server. The implementation seems generic enough that it could be adapted to Scala and the vscode-python team has alluded to adding polyglot support at some point in the future (see https://github.com/Microsoft/vscode-python/issues/5109). This approach has the benefit of requiring less custom work but requires the user to have a jupyter server running on their computer in the background.

I have a feeling it might be good to wait a few months and see if something moves with the vscode-python polyglot support.

Worksheets

Possible implementation approaches:

• do similar instrumentation as mdoc (https://scalameta.org/mdoc/) to capture the bound variables, static types and runtime values. Mdoc solves a few problems well in my opinion: small runtime classpath footprint, customizable type and runtime value pretty-printing (defaults to pprint, which is a lot nicer than toString()), it builds a rich runtime data structure with all the instrumented values (range positions, types, runtime values).
• build on top of the Scala REPL or Ammonite REPL. I don't think we get proper positions of the captured variables from those APIs but that might be solved with some minor fixes or creative hacks.

Markdown

Similarly to regular worksheets, we could build on top of mdoc to instrument and evaluate code fences inside markdown files. The worksheet mode (*.worksheet.sc) could be treated as a markdown file with only a single code fence.

[ekrich]

Just a few comments about features related to notebook support. Thanks for the overview above.

Lots of notebook type concepts are in engineering/math tools like Matlab, Maple, Mathematica etc. A good example I have used is _Mathematica's_ notebooks. Links - http://www.wolfram.com/technologies/nb/ - and - https://reference.wolfram.com/language/guide/NotebookBasics.html

Today of course there is Almond, spark-notebook, and notebook systems in Databricks and the like, let alone the Jupiter and other Python tools.

Useful features:

• Ability to run cells individually or run from start to finish the entire script. I think running the whole script can be too slow if anything expensive is being computed or rendered.
• Ability to write documentation in the script. Standard worksheets can have comments between statements but having the ability to have markdown support or other stylized text can be used for formal documents, course material, or other highly useful purposes. So the idea about markdown would be great. VSCode has some pretty good plugins for visualizing markdown but of course inline interpretation would be fantastic in the future.
• Having code editing features like the current Metals I guess is expected but I thought I'd mention it anyway.

I would like to see a minimal worksheet system to start with more features later. I think it might be an option to have the dependencies in a generated file/directory and be listed in sbt format or something you can just copy from Maven central if you need to add them. When creating a worksheet, maybe the file could be generated like is done for scalafmt? If you have 3 or 4 worksheets it might be tedious to have Ammonite imports on each but it might be useful to add dependencies directly to the worksheet for experimentation. Perhaps even add the current project classpath for the current project. Just throwing some ideas out.

[olafurpg]

Have a basic working implementation that

• shows evaluated code in-line next to the code using non-editable text decoration. This works only with VS Code for now but I believe it should be possible to do the same for other editors, at least Emacs.
• evaluates code using mdoc, this gives us fine grained control over rendering of values (including for example infinite Stream types), diagnostics, cancellation support, and more. I'm sure it would have been possible to achieve similar functionality with ScalaInterpreter from Almond but I am less familiar with that API.

A small preview

Will try to open a PR as soon as I get around to clean up the branch.

[olafurpg]

@ekrich this is a "minimal" worksheet implementation to get the ball moving. The current implementation doesn't provide several features that I would love to support down the road

• evaluate parts of the document (e.g., notebook cells) to avoid repeating slow computation. I have some ideas how to support this with mdoc while ensuring that cells always evaluate in the right order.
• displaying rich media objects (plots, charts, progress bars, etc). This might be blocked by https://github.com/microsoft/vscode/issues/3220. We could maybe build on top of the Notebook support for Python in VS Code 🤔
• writing documentation in the script. I believe it would be quite straightforward to extend the mdoc-based implementation to support "markdown worksheets" in a similar fashion as R Markdown https://rmarkdown.rstudio.com/

[robsonpeixoto]

How to show the type, like intellij and scala-ide?

[tgodzik]

@robsonpeixoto best to do it with Hover I think.