Scala-dev: REPL: Improved REPL

[adriaanm]

Here are some ideas for improving the repl:

• [ ] syntax highlighting
• [x] class-based wrapping (PR)
• [ ] take advantage of Java 9 features
• [ ] improve embeddability of repl in worksheets for fast data platforms

Steal Incorporate the improvements already implemented in the Ammonite REPL:

• [x] improved multi-line editing/history
• [ ] magic imports (ivy dependencies etc)

More ideas / requirements:

• [ ] reload class path when changing/recompiling project (faster test cycle)
• [ ] ctrl-c stops execution of program instead of repl itself
• [ ] test in IntelliJ shell window

[DavidPerezIngeniero]

+1 for incorporating Ammonite, this is a fantastic tool.

[kjsingh]

As long as we keep the performance smooth!

[yinshurman]

in the REPL, enter a Tab will display all the available names imported in the scope which makes a lot troubles in some case,for example,when pasting codes with tab as indent from editors like vim even in the :paste mode ,the screen if full of unexpected names.
by the way ,the auto completion by Tab direct print all the names in the screen makes it hard read on the console.Can't you just display it another way ? For example, like IPython' way.

[yinshurman]

In Python you can investigate any object foo (type, object in a global variable, built-in function, etc.) using foo to see what the thing evaluates to, type(foo) to show its type, dir(foo) to tell you the methods you can call on it, and help(foo) to get the built-in documentation. You can even do things like help("re") to find out the documentation on the package named re (which holds regular-expression objects and methods), even though there isn't an object associated with it.Can scala REPL does things like this ?

[som-snytt]

@yinshurman about tab, are you using a recent version with https://github.com/scala/bug/issues/9336 ? That enables the kludge in jline; but maybe it could do more in paste mode.

[yinshurman]

I'm use scala of version 2.11.7

[adriaanm]

This should be fixed in 2.11.11. For some of the equivalent commands from
python try :help
On Fri, May 5, 2017 at 02:18 yinshurman [e-mail] wrote:

I'm use scala of version 2.11.7

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/scala/scala-dev/issues/325#issuecomment-299419373,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFjy2auJXjsXDbeB30GHg3aySAuhPRpks5r2ulegaJpZM4MRr8R
.

[yinshurman]

It's fixed by add a tip "Display all 435 possibilities? (y or n)" every time you enter a tab.
Instead of direct printing all the available options ,I think it's better to display them another way,for example,when you type foo. and enter a tab, all the available options display in another little window right behind the foo.,and when you selected the option you needed,the window disappeared too. I think it is better comfortable

[som-snytt]

It's a nice consequence of English pronunciation that a windowing REPL or wrepl sounds the same as repl.

[adriaanm]

The plan is to first split the current repl into:

• a layer on top of the compiler (similar to the presentation compiler)
• a shell that implements the UI (and interfaces with jline3).

The api exposed by the compiler should also support other shells, such as ammonite / spark notebooks. Eventually, maybe we can support connecting to this repl compiler instance using the language server protocol (as in https://github.com/lampepfl/dotty/issues/2195)

I've just started hacking on this in my WIP branch: https://github.com/scala/scala/compare/2.13.x...adriaanm:repl-modularize?expand=1

[adriaanm]

https://github.com/scala/scala/pull/5903 takes the first steps towards splitting the REPL into a shell (UI) component and a compiler to support evaluating expressions and definitions + auto-completion (similar to the presentation compiler). The idea is that there could be multiple front-ends (the default, ammonite, spark worksheets,...) that all use the same underling compiler for consistent evaluation and type checking semantics.

Also, the default shell will likely use jline3, which has better support multi-line editing, highlighting, parsing as you type, etc.

[adriaanm]

By the way, we're currently not planning to backport the REPL work to 2.12. Please target 2.13.x for any PR that changes the REPL (or, please fix it first in 2.13 and backport to 2.12).

We are open to discussing a backport of the rework, though -- especially if someone's interested to contribute it :-) (A backport would have to be more careful not to break anyone currently embedding the 2.12 repl).

Finally, we'd love to hear feedback on the interface proposed in scala/scala#5903 from anyone embedding the REPL.

[retronym]

Another wishlist item that was in part prototyped in REPLs of yore would some features that exploit the runtime types and reflection.

• Autocompletion of x.asInstanceOf[<TAB>] could look at the x.getClass, figure out the corresponding ClassSymbol, and then intelligently merge that with the static type of x. Example if x.getClass == classOf[Some[_], and x is of type Option[String], offer to downcast to Some[String].
• Another UI option for that would be to have another level of tab completion used a safe downcast as an adaptation.
• Provide a utility method or command to do some reflection based stringification of objects
• Cheat mode: Add the ability to call private methods and fields by generating reflective calls

An unrelated feature that I think would be beneficial would be a UI to show the expected type at the cursor. Maybe use this to filter or prioritize autocomplete suggestions. I prototyped this in my last round of REPL renovations, but didn't follow through.

[retronym]

One more wishlist item for power mode: a version of reify that just funnels the actual typed argument tree back into userland.

You can simulate this at the moment with a macro and a backchannel:

⚡ scala -cp /tmp
Welcome to Scala 2.12.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112).
Type in expressions for evaluation. Or try :help.

scala> :power
Power mode enabled. :phase is at typer.
import scala.tools.nsc._, intp.global._, definitions._
Try :help or completions for vals._ and power._

scala> jz.TreeOf { class C { def foo = 42 } }
res0: $r.intp.global.Tree =
{
  class C extends scala.AnyRef {
    def <init>(): C = {
      C.super.<init>();
      ()
    };
    def foo: Int = 42
  };
  ()
}

scala> :quit

⚡ cat sandbox/treeof.scala
package jz
import scala.language.experimental.macros
import scala.reflect.macros.whitebox.Context

object TreeOf {
  def apply[A](a: A): Any = macro impl
  def impl(c: Context)(a: c.Tree): c.Tree = {
    val key = "jz.treeof.backchannel"
    System.getProperties.put(key, a)
    import c.universe._
    q"_root_.java.lang.System.getProperties.remove($key).asInstanceOf[Tree]"
  }
}

[adriaanm]

that's beautiful

[zaxebo1]

is there a "plugin loading mechanism for REPL", with a stable API and proper documentation/example, "through which one can add new inbuilt commands(colon commands) to REPL" , say ":my_new_command" , just like the inbuilt commands ":help" , ":paste" etc.

[som-snytt]

Welcome to Scala 2.12.3 (OpenJDK 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> import scala.tools.nsc._,interpreter._
import scala.tools.nsc._
import interpreter._

scala> val loop = new ILoop { override def commands = super.commands :+ LoopCommand.cmd("greet","helper","go for it", s => { println("hi") ; Result(false, None) }) }
loop: scala.tools.nsc.interpreter.ILoop = $anon$1@58f31629

scala> loop.process(new Settings)
Welcome to Scala 2.12.3 (OpenJDK 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> :greet
hi
res0: Boolean = true

Other than that, no. So, no.

[som-snytt]

REPL should clip my stackoverflows for me. Give me a last command to display it. And elide repeating segments:

  at .$anonfun$f$1$adapted(<console>:11)
  at .$anonfun$f$1(<console>:11)
  at .$anonfun$f$1$adapted(<console>:11)
  at .$anonfun$f$1(<console>:11)

[hvesalai]

See also https://github.com/scala/scala/pull/6382

[dwijnand]

jline3 upgrade split off to https://github.com/scala/scala-dev/issues/535 (and scheduled for 2.13.0-RC1)

[retronym]

Here's a test case that shows the inefficiency of the current wrapper schemes that require wrapper nesting proportional to the length of the REPL session.

https://gist.github.com/ad2505e99fc4084e4f9e79aa5b147aad

We should revive https://github.com/scala/scala/compare/2.13.x...adriaanm:repl-wrappers to address that problem. In that branch, the test above fails suggesting an unwanted semantic change:

scala> import bar.foo
<console>:16: error: reference to bar is ambiguous;
it is imported twice in the same scope by
import bar
and import bar
       import bar.foo
              ^

scala>  foo
<console>:17: error: reference to foo is ambiguous;
it is imported twice in the same scope by
import foo
and import bar.foo
        foo
        ^

[som-snytt]

I wanted the nesting level import to compile a directory as a single compilation unit, where files are not just concatenated, but each file is in a magic nesting block. (That requires both increasing and decreasing, for entering and leaving the block.)

I think REPL lines should compile to functions, taking values instead of importing them. That would be closer to the scripting environment, where contexts maintain bindings.

[SethTisue]

@dwijnand currently this is on the 2.13.2 milestone. is there anything here that 1) is crucial for 2.13.2 but 2) isn't already covered by existing PRs/issues that are already milestoned 2.13.2?

[som-snytt]

@SethTisue definitely https://github.com/scala/scala-dev/issues/325#issuecomment-367848584 (self-deprecation)

@dwijnand What's going on? Are you the person tasked with getting jline 3 over the goal line, or something else? From the outside, it's hard to know if it's worth touching anything in that area for obvious reasons (merge conflicts rather than personality conflicts).

[SethTisue]

Are you the person tasked with getting jline 3 over the goal line

That's me (and it's literally the top item in my queue at present, other than making sure all our 2.13.2 ducks are in a line generally).

...but perhaps Dale will help also, depending on how much else he has going on. (He's up to speed now on REPL stuff, having just gotten https://github.com/scala/scala/pull/8712 over the 2.12.x finish line, though merging it onto 2.13 is still in process.)

it's hard to know if it's worth touching anything in that area

Mm hmm. It makes sense to hold off a bit on anything that's likely to more-than-trivially merge-conflict with the JLine 3 work and the class-based changes.

[dwijnand]

@dwijnand currently this is on the 2.13.2 milestone. is there anything here that 1) is crucial for 2.13.2 but 2) isn't already covered by existing PRs/issues that are already milestoned 2.13.2?

I think we're using this ticket as a proxy for "let's not forget about the REPL" issue, every release. So with the class-based changes and hopefully the JLine 3 upgrade, both targeting 2.13.2, this one can be pushed to .3.

@dwijnand What's going on? Are you the person tasked with getting jline 3 over the goal line, or something else? From the outside, it's hard to know if it's worth touching anything in that area for obvious reasons (merge conflicts rather than personality conflicts).
it's hard to know if it's worth touching anything in that area

Mm hmm. It makes sense to hold off a bit on anything that's likely to more-than-trivially merge-conflict with the JLine 3 work and the class-based changes.

@som-snytt Would you benefit from having a, let's say, every-two-week video call? Might help on a regular basis touch base on things that slip or that we forget to communicate clearly.

Seth's on JLine 3 duty. But my branch-merging of class-based hit merge conflicts. Not a huge amount, but enough to likely incur another rebase push on the JLine 3 upgrade. I'm happy to do that rebase, btw, to help the cause (and b/c I have a good tooling setup for rebases, and b/c I know what the cause of the rebase). Oh than that, I don't have any work coming up for the REPL, so no potential future conflicts from me (both merge conflicts and personality conflicts).

[SethTisue]

The JLine 3 upgrade is progressing well at https://github.com/scala/scala/pull/8036 and https://github.com/scala/scala-dev/labels/t%3Ajline3 . There are separate tickets and we'd love to have contributors tackle any of them.

In general, further REPL improvements are a likely area of focus for us in 2020. (It stalled somewhat in 2019, but only because we had to focus on getting 2.13.0 and 2.13.1 out the door. That's done now.)

[SethTisue]

I did a little Twitter poll at https://twitter.com/SethTisue/status/1255633365232840704

unscientific results:

"add dependencies live" is the clear winner, but also the most ambitious as it involves bringing in coursier as a dependency. it might best be tackle-able in combination with replacing our current start scripts with coursier-based ones (as @lrytz observed)

it was nice to see the other two choices I gave, which are easier, both got a good amount of support

in the replies there was interest in displaying type names in their shortest reachable form, as Ammonite already does:

@ scala.collection.mutable.Buffer.empty 
res0: collection.mutable.Buffer[Nothing] = ArrayBuffer()

@ import collection._ 
import collection._

@ scala.collection.mutable.Buffer.empty 
res2: mutable.Buffer[Nothing] = ArrayBuffer()

so I think that should be on the short list as well.

[retronym]

An intermediate solution would be to require coursier to be installed separately and interact with it via its command line interface.

[bjornregnell]

@SethTisue I think showing scaladoc in REPL is, in the minds of the beginner coders that I'm teaching, the most important feature on your Twitter poll short-list. (The other features in the REPL upgrade already in there are really great for beginners as well, like multi line editing. :+1: :1st_place_medal: )

[SethTisue]

This ticket has grown stale, so I'm replacing it with https://github.com/scala/scala-dev/issues/754

For the record, the big improvements that have already landed are:

• The REPL code was refactored into the front end vs. the part that interacts with the compiler
• JLine 3 upgrade, getting us multiline editing and other improvements
• Class-based wrapping by default