Vscode-jupyter: Working directory for non-python kernels in a temp directory

[jlperla]

Bug: Notebook Editor, Interactive Window, Editor cells

Steps to cause the bug to occur

1. Create a new jupyter notebook and save it in a file location, or open an existing file
2. Change the kernel to something already installed (e.g. Julia 1.4.2 in my case)
3. Run pwd() to see the working directory

Actual behavior

I am getting C:\\Users\\MYUSERNAME\\AppData\\Local\\Temp\\7bb05f8c-bf2c-4a11-b590-de1cdae0d271 as the path.

This is especially important for Julia since the jupyter kernel activates a Project.toml and Manifest.toml local to the path of a file when it starts. This makes reproducible packages/etc. possible, and is difficult to do after startup.

Expected behavior

It should be be where the file is opened or saved. In my case, 'c:\\Users\\MYUSERNAME\\Documents\\GitHub\\sandbox'

This is what happens with python kernels, for example. In that case, ;pwd returns that string.

Your Jupyter and/or Python environment

• Jupyter server running: Local
• Extension version: 2020.6.91350
• VS Code version: 1.46.1
• Setting python.jediEnabled: false
• Python and/or Anaconda version: 3.7.4 64 bit conda
• OS: Windows
• Virtual environment: conda

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[IanMatthewHuff]

@jlperla Thanks for reporting this to us. The new non-python language support is something that we are just ramping up so there are some hitches with it still. With Python we run a command at startup to set the directory, but we don't have that support for other languages yet.

We do have a command to execute code at every notebook start. I tried this out and it works for Julia to set the working directory, but I'm not sure if this code is executed early enough to work in the case that you are looking at with regards to Project.toml and Manifest.toml. Could you try it out?

[jlperla]

Thanks for looking into this, and the hitches are completely understandable.

I tried it out adding the ; pwd shows the correct directory now in this test.

But as for the timing:
Sadly, the change in directory doesn't seem to happen early enough. I believe that this all comes down to the following in IJulia: https://github.com/JuliaLang/IJulia.jl/blob/master/deps/build.jl#L5 which installs a kernelspec with a commandline option.

That commandline argument to julia tells it to recursively activate a project file after starting up. There may be ways to emulate it ex-post, but I think it is generally a bad idea since it could lead to different behavior with commandline options for julia.

To me, the longterm goal would be for this to work with remote kernels as well, so my preference would be to think things through from that perspective.

Maybe @stevengj has thoughts on this and any possible workarounds?

BTW: If you want to emulate this stuff yourself with Julia, it is pretty easy:

• Create a test.ipynb in the folder you are interested in; change the directory to that file; and ensure you are using the Julia kernel
• Then type in ] activate . which will should create a new Project.toml and Manifest.toml in that directory.
• Do something like ] add Parameters to get a package added to the local project so you can see the difference.

Then later, you can open the file and go ] st to see if it is activated and that package is there. If it says somthing like

]st
Status `C:\Users\YOURUSERNAME\.julia\environments\v1.4\Project.toml`

Then you do not have the package activated

[stevengj]

Why can't you set the working directory to that of the notebook before executing the kernel-launch command? That's what Jupyter does…

[jlperla]

@IanMatthewHuff Hi there. Just to verify that I tried what you asked (as shown above) and it didn't work.

I know there are plenty of competing features, but this makes Julia essentially unusable as a kernel because of how package manifests are tied into the language/jupyter. Any thoughts or progress on this?

[IanMatthewHuff]

No particular common on this issue at this time. However I do want to make sure that we highlight this issue as being a blocker for Julia so I'll pull it back to our triage bucket to make sure it gets the prioritization that it needs.

[greazer]

Let's consider this as must-fix with raw kernel provided it's not overly difficult.

[rchiodo]

Fixing this for raw kernel (our new way of talking to jupyter kernels) is easy (basically what @stevengj said).

Local jupyter (when ZMQ binaries are not available) is a lot harder. We have to tell jupyter to create it's backing file at a relative path to the workspace.

Remote Jupyter is a no op. We don't start the jupyter server in that case so it's up to the user to start it somewhere their julia project is.

[jlperla]

@rchiodo if I understand things, then what you are saying makes complete sense. Although I am not sure I understand the distinctino between "local jupyter" and "raw kernel". Is this something that would frequently happen for normal usage?

[rchiodo]

We have two ways to talk to a jupyter kernel.

1. Through an instance of jupyter notebook that we start locally. This was how we originally implemented our support as we can use the same way to talk to a remote kernel as we do a local one. This is the 'local' jupyter case I mentioned above. In this scenario the working directory of jupyter isn't necessarily the root folder of where your notebook is. This makes it a little more complicated for me to get the kernel for this notebook to run in a different directory (jupyter has no start kernel with this dir option).
2. Talking directly to a kernel we start ourselves. This is our 'new' way that we call 'raw kernel'. It uses ZMQ to talk to the kernel process. This was a lot simpler to control the working directory because it's just a parameter to us starting the process. However the ZMQ package we use is not supported on all systems (some older linux variants for example) so we still need the local jupyter case. We also haven't turned this on for everybody yet as it was an experiment to see how many people would end up not supporting the ZMQ package.

So to answer this question:

Is this something that would frequently happen for normal usage?

We're not sure yet. We hope not. We hope most users will be able to use the ZMQ package we ship.

[DonJayamanne]

I mentioned above. In this scenario the working directory of jupyter isn't necessarily the root folder of where your notebook i

Can we not use a CLI to tell jupyter to set the CWD. I tnink the notebook_dir can work, however this would mean we'd need to always start a new jupyter server for all languages and all notebooks.
Or we have pre-execute scripts where we can do this either for users or users can do this via the pre-execute scripts.

[rchiodo]

Pre execute scripts don't work. It has to be the working directory for the kernel when it starts. @IanMatthewHuff suggested @jlperla try this above.

[jlperla]

I think that helps. If the "local kernel" approach isn't easily feasible, then I think that is OK to ignore. The "remote kernel" I think would work out of the box then, since it controls which directlores are available.

Is the .devcontainer version (i.e. if I start things up in codespaces) going to work similarly to the case where you start your own kernel?

[rchiodo]

Is the .devcontainer version (i.e. if I start things up in codespaces) going to work similarly to the case where you start your own kernel?

Yes. It should be identical. codespaces or ssh or wsl all behave the same as a local machine. We'll try the 'raw' kernel version and if it doesn't work, fall back to the 'local' jupyter (where the jupyter runs on the ssh/codespaces/wsl machine).

[rchiodo]

This should end up in our September release (possibly August, have to see if everybody else is willing to risk this change so close to release).

[jlperla]

@rchiodo I really appreciate the attention to this issue! After this I can test a little more on Julia to see how close it is.

[IanMatthewHuff]

Validated. The Julia experts can check it out more in the wild, but using the helpful example from jlperla it looks good to me:

Just comment back here and I'll reopen the bug if there are still issues here.

[jlperla]

@IanMatthewHuff spectacular! When does this sort of thing hit vscode insiders so I can test it?

Should this also just work with github codespaces if I give an appropriate image with Julia, conda, and your extension installed?

[IanMatthewHuff]

Since the change here is in the Python extension then you would actually need the insiders version of our extension (can use that with either normal VS Code or VS Code - insiders). Info is on this here:
https://github.com/microsoft/vscode-python/blob/master/CONTRIBUTING.md#development-build
It's very easy to swap that back to the release version, so it's safe to use that to test if you want.

I believe that this should work with github codespaces, I've not tested that myself, but it's not using anything that would prevent that. However I would assume that github codespaces doesn't have the insiders version of our extension, so you would not see the fix there yet.