Julia-vscode: Debugger ignores Revised code or code edits

[phlavenk]

I'm excited by the usability of the debugger inside recetn vscode julia plugin.
One thing I use debugger for is to check where my bug in code is and i'd like tore-verify, after correcting, that the fix indeed did what i wanted.
In 0.15.18, the code changes get not propagated into the new debugging sessions:

function t1(a, b)
    if a>b
        return (a, b)
    else
        return  a #this is bug
    end
end

function t2(a,b)
    a, b = t1(a,b)
    return a, b
end
t2(4, 2) #works
t2(2, 4) #error

now, I run @enter t2(2, 4), and step until i get to the bug - i'm expecting t1 returns a tuple. I'll correct t1 to:

function t1(a, b)
    if a>b
        return (a, b)
    else
        return  (b, a) #corrected bug
    end
end

and save the file and re-evaluate t1. But, a next @enter t2(2,4) still errors during step-through, even when, on repl, t2(2,4) executes correctly.
I've seen this behavior in more complex situation, where debugger was ignoring Revised changes in a deved package, although the fixed code executed correctly.

[davidanthoff]

Can you explain a bit more in detail how I can reproduce this bug? So far I haven't had any luck.

In particular: I assume you have one file that has the first code block in your example? How to you evaluate that file into the REPL? Ctrl+Enter, or Alt+Enter, or something else?

How do you re-evaluate t1?

I think the more details you provide, the better. Thanks!

[phlavenk]

I'll try to give more detailed workflow:

Edit vscode-mwe.jl in vscode:

function t1(a, b)
    if a>b
        return (a, b)
    else
        return  a #this is bug
    end
end

function t2(a,b)
    a, b = t1(a,b)
    return a, b
end

save it.
Start REPL session:

julia> using Revise

julia> include("vscode_mwe.jl")
t2 (generic function with 1 method)
julia> Revise.track("vscode_mwe.jl")

julia> t2(4, 2)
(4, 2)

julia> t2(2, 4)
ERROR: BoundsError: attempt to access Int64
  at index [2]
Stacktrace:
 [1] indexed_iterate(::Int64, ::Int64, ::Nothing) at .\tuple.jl:90
 [2] t2(::Int64, ::Int64) at C:\Users\petr.hlavenka\Documents\_Projects\ideas\mwe_playground\vscode_mwe.jl:11
 [3] top-level scope at REPL[6]:1
 [4] eval(::Module, ::Any) at .\boot.jl:331
 [5] eval_user_input(::Any, ::REPL.REPLBackend) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\REPL\src\REPL.jl:86
 [6] run_backend(::REPL.REPLBackend) at C:\Users\petr.hlavenka\.julia\packages\Revise\Pcs5V\src\Revise.jl:1073
 [7] top-level scope at none:0

I want to see where my code is wrong:

julia> @enter t2(2, 4) 

and after F11, F11, ... F10...F10.. i step into line return a #this is bug and on return I get error - as expected because I expect to get a tuple.

So I do the edit of this line to correct to return (b, a) #corrected bug, save the file (expecting revise will do it's job of re-evaluating the changes) and redo debugging:

julia> @enter t2(2, 4)

I'm again stepping through t2 and t1 and, although bug has been corrected, i still hit the old bug (seems like i'm stepping through the old definition of t1).
On the other hand, directly calling

julia> t2(2, 4)
(4, 2)

works as expected - Revise worked well.
With this mwe I'm able to reproduce 100%.

extension 0.15.19, vscode 1.44.0, Julia 1.4 win-x64 official installer, win10-x64

[davidanthoff]

Perfect, I can reproduce that now!

I have to think a bit how we can fix this... The extension currently is using its own copy of https://github.com/timholy/CodeTracking.jl, and when you load Revise.jl it will use a different version of the same package. What happens is that Revise.jl will update the state of the global instance of CodeTracking.jl, but not the one that the VS Code debugger uses. I have to think a bit how we can best resolve this, not super trivial and will probably take a while.

[timholy]

As I posted in https://discourse.julialang.org/t/what-juno-features-do-you-use-most/39271/91, this is a critical issue for me. Happy to help try to figure out how to fix it.

BTW, include("vscode_mwe.jl") would not normally turn on tracking with Revise (you'd have to use includet). Another way to reproduce the bug (one that doesn't depend on a specific way of running code) is to dev Example and then @enter hello("David"). After doing it once, add a couple of comment lines above the definition of hello and then @enter hello("David") takes you to the wrong line.

[davidanthoff]

For background: we have the same rule for the REPL process that we have for the symbol server load process. We assume that the process is "owned" by the end-user, so we never load any packages in the normal way, because otherwise we would force the user to use a specific version of that package and then would also restrict the user to specific versions of other packages that might be a dependency.

We _do_ use a few packages, but don't load them using using or require, but for leaf packages (i.e. packages that don't have any dependencies other than stdlib), we just include the main package file from inside our own module. We currently load two other packages that themselves have further deps: JuliaInterpreter.jl and JSONRPC.jl. We use different strategies for the two. For JSONRPC.jl we structured the package such that the main package file only has using and one include statements (https://github.com/julia-vscode/JSONRPC.jl/blob/master/src/JSONRPC.jl), and then we can load it in the REPL process with these lines in such a way that nothing gets added to the list of globally loaded packages. For JuliaInterpreter.jl we use this horrible hack to make sure it uses our private copy of CodeTracking.jl :)

The net effect of all of this is that when you start a Julia REPL in VS Code, it will have one module called _vscodeserver loaded, and any other package we are using is loaded privately into that module, and if the user wants to use a different version of the same package (say JSON.jl), they can load a second copy of that package in the normal way into their process and use that and everything just works.

So generally what is happening here is that the version of JuliaInterpreter.jl that our debugger uses is connected to our private copy of CodeTracking.jl, and the Revise.jl that the user loads is loading a second copy of CodeTracking.jl, and the two instances don't know about each other, and more importantly, the CodeTracking. jl instance that JuliaInterpreter.jl uses is not updated by Revise.jl.

I think there are a couple of ways out:

1. I think in general I don't really understand why JuliaInterpreter.jl needs to rely on global state in CodeTracking.jl :) If that wasn't necessary, it would be a simple solution.
2. We could try to come up with a way that the user loaded Revise.jl "connects" with the VS Cod instance of CodeTracking.jl. I think that would require changes on both our end and in Revise.jl.
3. We ship Revise.jl out of the box with the extension and load it in the same private way that we load JuliaInterpter.jl etc.

I like 3) best by far: it would mean that users can just select the config value for "revise" in the VS Code settings and everything would work, without the need to install the package itself etc. It would also mean that users could use Revise.jl, without for example restricting them to specific versions of deps like OrderedCollections.jl.

I think _by far_ the easiest way to do this would be to just change Revise.jl, LoweredCodeUtils.jl and JuliaInterpreter.jl to the same file structure that I used in JSONRPC.jl (where the main package file has all the using statements in it and one include("core.jl")), and then we would use the same structure that we used to load JSONRPC.jl.

Oh, and when we ship our own copy of packages, we literally just put a clone of the package into the extension.

[timholy]

I think in general I don't really understand why JuliaInterpreter.jl needs to rely on global state in CodeTracking.jl :) If that wasn't necessary, it would be a simple solution.

Try the following experiment:

1. dev Example; using Example; @edit domath(3) to open the source file and see the code
2. Start a REPL session
3. @enter domath(3). Step into domath. Check that x is indeed a local variable.
4. Let the debugger finish execution and return to the REPL
5. Without quitting your Julia session, insert 7 blank lines somewhere above the definition of hello. At this point the definition of hello should now be on line 16, and the definition of domath is on line 23. Save the file.
6. Back in the REPL, @enter domath(3) again. Step into the function. Visually, VSCode's debugger is still pointed at line 16, which is now the method definition for hello. So it looks like you're debugging hello. But x is still a local variable, and you can check that it is prepared to return 8.

In contrast, Juno's debugger in step 6 takes you to line 23. This is a big deal because a debugger is for, well, fixing bugs. And fixing bugs often changes line numbering.

Now, if there's a good way to do this without relying on local state, we could consider it. But we didn't want to add a bunch of file-watching, parsing, header extraction, signature detection, etc, to JuliaInterpreter itself when Revise & CodeTracking already do that.

[timholy]

I think by far the easiest way to do this would be to just change Revise.jl, LoweredCodeUtils.jl and JuliaInterpreter.jl to the same file structure that I used in JSONRPC.jl

That seems plausible. CodeTracking too? The key characteristic being that you get the whole package with statements that are just module, using/import, export, and include?

[davidanthoff]

If after step 6, I run @which domath(3), it correctly shows line 23. So I guess I don't understand why that information is not used by JuliaInterpreter.jl? It _does_ seem as if the location information that is stored in the Julia internal data structures is properly updated by Revise.jl, but then it seems that CodeTracking.jl has a second data structure where it stores location information, and that is used by JuliaInterpreter.jl instead of going back to the location information that Julia itself stores somewhere?

Is there maybe some way that we could just "clear" the data structures in our private copy of CodeTracking.jl before we start a new debug session, so that JuliaInterpreter.jl would go back to the line information that Julia itself tracks, rather than what is stored in the state in CodeTracking.jl?

That seems plausible. CodeTracking too? The key characteristic being that you get the whole package with statements that are just module, using/import, export, and include?

I think we wouldn't have to change anything about CodeTracking, it looks like a leaf package to me, right? I.e. it only uses stdlib packages. If you are on board with the general idea here, I can open PRs against the packages where I think we need to rearrange the code a bit, they should be fairly trivial changes.

[timholy]

If after step 6, I run @which domath(3), it correctly shows line 23

Only if you're running Revise. See:

• https://github.com/timholy/Revise.jl/blob/ed7236de87acece0e4f731fae245fa5682c77c9c/src/Revise.jl#L1282-L1285
• https://github.com/timholy/Revise.jl/blob/37065dba718ee9bf32b181a67536ac57fafc42e1/src/Revise.jl#L1109-L1144
• https://github.com/JuliaLang/julia/blob/147d5b101cea8c845b81c80943a8089bd59f5e1a/base/errorshow.jl#L554-L563
• https://github.com/JuliaLang/julia/blob/147d5b101cea8c845b81c80943a8089bd59f5e1a/base/methodshow.jl#L122-L125

Julia stores no dynamic location information whatsoever. The only location info is m.file and m.line, and that doesn't update unless you redefine the method. (Method is actually a mutable structure, but really bad things happen if you change m.line because the linetablein the CodeInfo can't be updated to match.) Inserting blank lines above a method definition does not trigger reevaluation of all the methods below it (that would be horrible because it would invalidate all kinds of code...imagine adding a couple blank lines near the top of base/array.jl...). Much of this is documented here, but I know it's a lot to read so I can answer questions. Basically, as the Revise docs explain, the crucial information about location is maintained in CodeTracking.method_info, and CodeTracking can act as a query engine for that info, but it's actually Revise that populates and updates CodeTracking.method_info; it's basically a stub unless you load Revise. This design allows other packages like JuliaInterpreter and MagneticReadHead to depend on CodeTracking (lightweight) without depending directly on Revise (heavy). All necessary communication happens via CodeTracking.method_info. Revise maintains additional caches of its own, but these are not relevant for understanding location.

Is there maybe some way that we could just "clear" the data structures in our private copy of CodeTracking.jl before we start a new debug session, so that JuliaInterpreter.jl would go back to the line information that Julia itself tracks, rather than what is stored in the state in CodeTracking.jl?

You wouldn't have any dynamic info, then. Absolutely nothing changes in Julia proper after you load a package, unless you're running Revise.

[davidanthoff]

Only if you're running Revise.

Ah, so if I call @which when Revise.jl is loaded, then it will actually return the line info that is stored in CodeTracking.jl? Then everything makes sense :)

So I do think we should probably just ship Revise.jl with the extension and load a private copy of it and the depending packages, as I outlined above. I think that will just generally give the best user experience.

I'll see locally whether my idea for loading all the necessary packages works or not, if it does, I'll open up PRs!

[timholy]

One question: when do you update? Revise updates right before running code entered at the REPL, not when the user hits "save" on the file. I guess it could call revise() right before launching a debug session?

[davidanthoff]

Right now we have this for code that is not entered at the REPL, but that is sent from the editor into the REPL.

How does Revise hook into the REPL so that it can run before something the user types gets executed? We should probably make sure that works well with the hooks that the VS Code extension installs itself.

But on this whole topic we can also go wild: we could run revise on the source file whenever the current file parses properly (even if unsafed), by sending a message from the extension to the REPL process to do that. Probably not very useful, but once we ship things out-of-the-box we could of course integrate a lot more closely.

[timholy]

How does Revise hook into the REPL so that it can run before something the user types gets executed?

It depends on Julia version. On 1.5+ there is Base.active_repl_backend.ast_transforms, and Revise just adds a function that transforms ex::Expr to the equivalent of revise(); ex. On earlier versions of Julia, Revise co-opts the processing of the REPL backend with its own "event loop." Relevant code is here.

Another thing I wonder: what if the user has Revise installed but isn't running it in that particular session? Do you even have a way of knowing? Would anything bad happen? I can't imagine what use debugging "old code" when you have "new code" sitting in your editor would be useful for, but thought it was worth asking.

1. We could try to come up with a way that the user loaded Revise.jl "connects" with the VS Cod instance of CodeTracking.jl.

Does this option merit further consideration? I've never done anything with network IO, but there do seem to be a number of things that could be done if the user's process could communicate with VSCode. (For example, in another thread I mentioned using backedges as a way of figuring out callees.)

[davidanthoff]

Ah, excellent! I think we probably need to carefully review our various REPL hook strategies, because we are about to merge some stuff that also hacks into the REPL with some pre-run hooks, and we should make sure the two don't get into each other's way. But I don't think that should be difficult.

Another thing I wonder: what if the user has Revise installed but isn't running it in that particular session?

So the way I'm imagining this right now is that we ship Revise.jl as part of the extension, and if the user selects the extension configuration setting "Use Revise", it will be automatically loaded into the Julia REPL process that the extension starts. I think we should probably also add something to Revise.jl that any other attempt to load it into the VS Code Julia REPL process becomes a no-op, so that we don't end up in a situation where the VS Code Revise.jl and an end-user installed Revise.jl get into each others way? We could for example start the VS Code Julia REPL with some env variable set that allows Revise.jl to detect that it is running inside VS Code and then don't properly load (we would just need _another_ hook to allow our private copy to load, of course).

We could try to come up with a way that the user loaded Revise.jl "connects" with the VS Cod instance of CodeTracking.jl.

Does this option merit further consideration? I've never done anything with network IO, but there do seem to be a number of things that could be done if the user's process could communicate with VSCode. (For example, in another thread I mentioned using backedges as a way of figuring out callees.)

So in the Revise.jl context I had meant "connect" not in the network sense, but rather that a user loaded Revise.jl could use the privately loaded VS Code copy of CodeTracking.jl instead of the normally loaded one. But I generally like the idea of just shipping Revise.jl by default much more right now, it just seems such a useful tool that we might as well make it available to every user of the extension.

We do have a full bi-directional communication channel between the Julia REPL and the TypeScript extension that is based on JSON RPC. It is used for lots of different things: plots display, grid display, code execution, the future variable workspace etc. I'm sure I'm forgetting some things right now. Very easy to add additional things to that. Maybe a good topic for a different issue, though?