Shadow-cljs: Proposal for Dirac integration via shadow.remote

[darwin]

I looked at the remote.md doc, studied its code and briefly glanced over shadow-cljs code.

I'd like to propose a possible way how to make Dirac play with Shadow.

Initial assumptions:

1. User has configured shadow-cljs project e.g. the default acme-app.
2. User adds Dirac Runtime into acme-app (via preloads)
3. User has shadow-cljs server with shadow.remote relay running.
4. dirac-tool is a shadow.remote tool connecting to the relay.

Typical Dirac workflow:

1. User starts dirac in a new terminal sesssion
2. dirac launches Chrome with Dirac DevTools, it also starts playground project in the background
3. the playground project has associated nREPL with Dirac middleware and Dirac Agent running inside the dirac JVM process
4. Dirac middleware will contain implementation of dirac-tool
5. dirac-tool will attempt connection to the relay and list available runtimes supporting cljs compilation
6. these runtimes will be presented to user under dirac :ls
   This is an example output from dirac :ls with figwheel-main "ClojureScript compiler" present, it is selected as indicated by ->, btw. dirac/f270d0e8.1 is the default compiler provided by dirac, also added shadow/build1 as an example.

> dirac :ls
Listing all Dirac sessions currently connected to your nREPL server:
 -> #1 figwheel-main | http://localhost:9500/ | Chrome/81.0.4028.0 | Mac/10.15.3 [f270d0e8]

Listing all ClojureScript compilers currently available in your nREPL server:
    #1 dirac/f270d0e8.1
 -> #2 figwheel.main/dev
    #3 shadow/build1

1. user will be able to select preferred runtime via dirac :switch, assume he switched to some shadow runtime (e.g. dirac :switch "shadow/build1").
2. when user enters a text into REPL
   8.1. it will get processed by dirac nREPL middleware, which will see that shadow runtime is selected, instead of calling own cljs compiler or figwheel's compiler (in-process), it will perform compilation over the wire via shadow.remote api.
   8.2. it will get back compiled js code or some error (java stack trace)
   8.3. it will send the js code for evaluation into the browser (it will be handled by Dirac DevTools and "dirac runtime" as usual)
3. For convenience, user is also able to send commands to shadow-cljs server via Dirac REPL (bridge)

Please note: Dirac nREPL and Dirac Agent are running in a separate process (of dirac) and are not in any way interfering with the initial shadow-cljs setup.

dirac-tool needs following services from (supported) runtimes:

1. :list-cljs-compiler-runtimes
2. :compile-with-cljs-compiler-runtime
3. :shadow-bridge

List CLJS Compiler Runtimes

This is a message for relay? Or it could be a broadcast and each runtime supporting :compile-with-cljs-compiler-runtime can answer independently?

In the first case it could look like this:

{:op :list-cljs-compiler-runtimes}

; >>> 
{:cljs-compiler-runtimes [
  {:name "build1" :id 123}
  {:name "build2" :id 456}
]}

These runtimes would be presented in dirac :ls as

Listing all ClojureScript compilers currently available in your nREPL server:
 -> #1 shadow/build2
    #2 shadow/build1

Compile with CLJS Compiler Runtime

Dirac needs to pass in the cljs code text and also optional :locals and maybe other compiler-env settings, this should be passed through to cljs.analyzer/analyze.

Dirac expectes compiler js code text back with source mapping info - or alternatively some error info which should be presented to the user.

{:op :compile-with-cljs-compiler-runtime 
 :runtime-id 123
 :code-text "(+ 1 2)"
 :compiler-env {
   :locals { ... }
   ...
 }
}

; >>> success
{:result :success
  :compilation-feedback [
   {:kind "warning" :text "some compilation warning"}
   ...
  ]
 :ns <effective-ns-symbol>
 :js "js code text"
 :js-source-map "json text"}

; >>> or in case of a failure
{:result :compilation-error
 :ex "exception message"
 :root-ex "root exception message if avail"
 :details <serialized stack trace somehow>
}

Shadow Bridge

Dirac sends a command as plain text. And shadow cljs will interpret it on its side. Potentially returning some output.

{:op :shadow-bridge
 :command "watch build3"}

; >>>
{:result :success
 :feedback [
  {:kind :stdout "something printed"}
  {:kind :stderr "something printed on stderr"}]
}

A note on stdout/err feedback. Dirac has currently infrastructure to stream back stdout/err produced during compilation (or other operations) back to DevTools console. It would be nice to support this as well. So for example :compile-with-cljs-compiler-runtime could trigger multiple reponses some of them could be just reporting printed output back.

---

@thheller If you agree with this concept, I would propose following steps:

1. you update shadow-cljs general code to support this plan (not sure about the state of shadow.remote)
2. then I could attempt initial implementation of these ops in shadow-cljs (in a branch) + implementation on dirac side
3. we could iterate on remaining issues (I'd bet I will discover some)
4. finally when we get it properly working, you can review it and turn into final code you are willing to merge/support

[darwin]

Just for record. Dirac 1.6.0 added support for REPL via cljs_eval:
https://github.com/binaryage/dirac/releases/tag/v1.6.0

This might be good enough solution for now.

[lucywang000]

@darwin A big 👍 for being able to see cljs variables in the variables list panel, really helpful when debugging!

I find it's not possible to eval cljs code when paused in a breakpoint - I guess it's an intrinsic limitation of the "loopback" mode?

> (+ 1 2)
VM1493:1 Uncaught SyntaxError: await is only valid in async function

[darwin]

I'm not 100% sure but I would say that it is not intrinsic limitation. When page is stopped on a breakpoint dirac uses special debugger api for evaluating expressions in the context of the page. Maybe I just need to enable some flag to allow async code as well. I noticed that there is relative new support in devtools console for top-level await expressions. Maybe I would need to do something similar.

[lucywang000]

Thanks @darwin . I just found your doc page https://github.com/binaryage/dirac/blob/master/docs/about-repls.md and it's really great read about the design of dirac and clojure repls as well.

[lucywang000]

@darwin IIUC this doesn't seem to be able to work. cljs_eval('"foo"') sends the content to shadow-cljs server (through the websocket connection) to parse, which the server compiles down to js code and sends it back to the page to execute. When the page is paused by a breakpoint, there is no way for this to happen.

What's more shadow-cljs server sends ping requests to the client and in a breakpoint the client is not able to respond with a pong, so the server would disconnect the client.

[darwin]

Good points. Let's move the discussion to dirac issues.