Julia-vscode: Linter doesn't check `LOAD_PATH` for packages

[ssfrr]

The linter seems to think things are undefined even if they are there:

Defined here:

used here:

output in the linter pane:

I think this might have to do with the fact that the SampledSignals package isn't located in the normal .julia/v0.6 package directory, but is in a different directory that I add to my LOAD_PATH in .juliarc.jl.

It also might be more useful in this case to display the warning on the import line with something like "Could not find package SampledSignals" rather than later on when you try to use the identifier.

[davidanthoff]

Yes, we certainly don't handle those cases.

[KristofferC]

IIUC, the way vscode does it right now is to actually load the packages to find out their symbols. Shouldn't it be enough to just load the .juliarc.jl file in the vscode julia process so that LOAD_PATH is updated and the packages can be loaded?

[davidanthoff]

Yeah, that would work, but we really need to move away from this model where the LS process loads user code. And executing .juliarc.jl to me seems even more risky than just loading the packages, who knows what users have in that file...

[KristofferC]

Well, typically stuff that should be OK to execute when you start julia...?

[ssfrr]

Another option would be to add an explicit LOAD_PATH-type option to the configuration, so users can specify other places they keep packages. On the other hand it might not be worth fiddling with until Pkg3 comes out, which I imagine will change this sort of thing substantially

[ufechner7]

Any updates on this issue? Any idea for a hack to make it work for now?

[davidanthoff]

No update, I'm afraid.

[ufechner7]

Pkg3 is out and alive, so there should be a way to solve this issue permanently now. I tried to add my package path to ~/.vscode/extensions/julialang.language-julia-0.11.1/scripts/languageserver
and here to the file main.jl

    ...
    empty!(LOAD_PATH)
    push!(LOAD_PATH, joinpath(@__DIR__, "packages"))
    push!(LOAD_PATH, "@stdlib")
    push!(LOAD_PATH, "/home/ufechner/copter-control/scripts")

    using LanguageServer, Sockets, SymbolServer
    ...

This used to work before, but doesn't have any effect with version 0.11.1 :(

[yongheekim-dev]

Pkg3 now provides dev for this kind of usage and
I've figured that stop using push!(LOAD_PATH, "mypath") and start using pkg>dev mypath solved all my headaches 😄

Linter cheks for packages registerd with pkg>dev.

[twadleigh]

bump

I'm in a situation where I can't make my packages individual repos---they live in a subdirectory of a monorepo of mostly C++---and have to rely on modifying LOAD_PATH in startup.jl.

I'm looking for the right point to hack something into LanguageServer.jl, _etc_., but haven't found it yet.

[davidanthoff]

I think roughly the strategy would be that we pass the content of JULIA_LOAD_PATH env variable "through" the LS process to the symbol server process. In particular, we want the LS process to run with an empty JULIA_LOAD_PATH, but we then want the symbol server process to run _with_ the content of that env variable.

So I think the steps would be to add another arg to the LS process startup around here, similar to oldDepotPath, for the content of JULIA_LOAD_PATH. Then we should probably also make sure that we run the LS server with an empty JULIA_LOAD_PATH here (that is kind of a bug right now, we should actually clear JULIA_LOAD_PATH for the LS process no matter what). Then, when we start the symbol server process here we would set the JULIA_LOAD_PATH env var again for that sub process. That should be feasible from the LS process because we now received the value for that as a command line arg.

Let me know if this explanation is too brief, happy to expand. What this would do is have the LS broadly pick up the content of JULIA_LOAD_PATH. I think that is pretty much the only thing we can do in this situation?

Or we could run startup.jl for the symbol server process, but I'm kind of nervous about that, given that there might be stuff in there that can be funky...

I think in any case we should probably do this in the v0.13.0 cycle, my hope is that we can get v0.12.0 out before juliacon, and this seems like a too big change to me to still do right now.

[ufechner7]

I would already be happy if I could hardcode my load path somewhere. So you mean adding env_to_use["JULIA_LOAD_PATH"] = "/home/ufechner/copter-control/scripts" after line 31 of SymbolServer.jl might work for me?

[davidanthoff]

@ufechner7 it might, certainly worth a try!

[twadleigh]

I was also thinking of just running julia -e 'println(LOAD_PATH)' to grab it without having to mess with the environment.

[twadleigh]

It looks like something more will have to be done in SymbolServer. The server process apparently already runs startup.jl because its LOAD_PATH already includes my custom directory.

[davidanthoff]

Ah, ok, I guess there is another issue: in symbol server we load all the packages that are specified in the env, but of course the packages in LOAD_PATH aren't specified in the env...

So probably we should broadly move to a design where symbol server doesn't eagerly load everything from the active env, but instead the LS sends specific messages to the symbol server "I need symbols for package X", and then symbol server only loads that (from wherever). That would probably also generally help with the startup times for the LS.

[ufechner7]

For me, the following work-around helps:
Instead of:

push!(LOAD_PATH, pwd())
using my_pkg

I use now:

include("my_pkg.jl")
using .my_pkg

Be aware of the dot in the last statement in front of the package name!

Perhaps this helps someone who has trouble with this issue.

[ufechner7]

Due to #807 this workaround does not always help, so fixing the original issue is still important.

[ufechner7]

Bump.

[stakaz]

Is this issue tagged for a particular release or something? Would be great if this would work... Maybe some ideas in the Atom.jl package could help?

[davidanthoff]

If an issue is on the Backlog milestone it means no one is working on it right now and the core team (@ZacLN and I) currently don't have a timeline for fixing/implementing it.

[ufechner7]

As you might know I would love to help to fix this issue. But the structure of the code base is so complex and undocumented that I gave up so far. So improving the documentation of how things work internally might help to attract new contributors.

[kcajf]

This would be really helpful. My code is structured into a series of modules. None of these are large enough to merit being their own "packages". Different pieces of code use different modules, all found via JULIA_LOAD_PATH.

Is this a very strange way to work in julia? It would seem surprisingly that everyone structures their entire codebase into packages..?

[davidanthoff]

I do think that modifying JULIA_LOAD_PATH is not common and not really encouraged. Obviously it works, and it would be nicer if we properly supported it.

[ufechner7]

I do think that modifying JULIA_LOAD_PATH is not common and not really encouraged.

This is not true. I use it in my own company, I am now doing consulting for an US company who is using that a lot in huge Julia projects, and we discussed that in discurse. It is a valid and common way to split software projects in modules.

[ufechner7]

So nice:

push!(LOAD_PATH, pwd())

[stakaz]

I do think that modifying JULIA_LOAD_PATH is not common and not really encouraged.

Maintain a package for each and every small module with about 3 functions inside which you need for a particular product does not look to me the right way ether.

Obviously it works, and it would be nicer if we properly supported it.

That would be a great improvement in my eyes.

[davidanthoff]

Yeah, sorry, I really don't want to give the impression that we don't want to support this, I do! I also know how, hopefully I'll find the time to squeeze this into the next release.

[ufechner7]

@davidanthoff Thank you for looking at this issue!
It is so helpful if the linter is working correctly and I can just say "Go to definition", for example.

[stakaz]

Hey, that sounds great! Thank you very much for this announcement ;)

[reactgary]

Yeah, sorry, I really don't want to give the impression that we don't want to support this, I do! I also know how, hopefully I'll find the time to squeeze this into the next release.

Thanks for the hard work David

[ufechner7]

Would be so nice... :)

[norru]

Bump. Just got bitten by this. Makes the linter a lot less useful.

[davidanthoff]

Alright, I'm trying to understand the scenario here a bit better, would be great if those folks in this thread that use this kind of setup could help me understand their use-case better, and in particular why the normal pkg> dev workflow doesn't work for you.

It would be in particular helpful if you could describe a bit what the folder structure under the folder that you add to your LOAD_PATH looks like. Lets say you add a folder /foo to LOAD_PATH. Does it then look roughly like this?

/foo
  /Module1
    Module1.jl
  /Module2
    Module2.jl
  /Module3
    Module3.jl

Or something else? For example

/foo
  /Module1
    /src
      Module1.jl
  /Module2
    /src
      Module2.jl
  /Module3
    /src
      Module3.jl

In general, I think the only scenario that I can imagine that we might be able to support would be to pick up a custom LOAD_PATH config if it was either a) configured as the JULIA_LOAD_PATH env variable, or b) we add an option in the VS Code extension itself. In that case one could configure it either in the user settings, or in the workspace settings. Would that be flexible enough for your use cases?

I should add that I'm still not a 100% certain how we would do this, but the next step is for me to better understand these scenarios.

[stakaz]

Hello, I am gald you look into the issue. In my user case the setup is very simple. I have a bunch of "little packages" which provide some useful functions for my PhD code but are too small to be real pacakges with Project.toml and so on. So what I am doing is just put some files in a Modules aka foo directory like:

./foo
    Module1.jl 
    Module2.jl 
    ...

and then use e. g. using Module1 in my codes (after adding the path to the foo).

So the point here is that I have kind of packages without pacakge environment. They are not large or general enough to be considered real packages but still it would be a mess to use include to use them.

For me a simple option in VS Code itself would be sufficient but maybe someone else will complain ;)

[norru]

Setup for my project is something like:

monorepo_root\
  src\
    c_sharp\
    c\
    julia\
      script1.jl
    python\
    ...
  lib\
    rust\
    julia\
      Module1\
        src
          Module1.jl
          SubModule1.jl
          SubModule2.jl
      Module2\
    python\
    ...

At the moment I'm adding monorepo_root\lib\julia to LOAD_PATH.

If I edit just the Module1 using dev, the linter picks the symbols in Module1 and its submodules. If I edit script1, which uses Module1, the linter does not pick the symbols in Module1. Same thing happens if I edit Module2 if it uses Module1.

Changing the structure of the project (ie breaking it down into multiple repositories) is not an acceptable solution.

VSC-only setting (or something that can be passed through to the JLS) is a viable solution for me.

[kcajf]

Same structure as Stakaz here.

On Mon, 3 Feb 2020, 03:57 Stanislav, [e-mail] wrote:

Hello, I am gald you look into the issue. In my user case the setup is
very simple. I have a bunch of "little packages" which provide some useful
functions for my PhD code but are too small to be real pacakges with
Project.toml and so on. So what I am doing is just put some files in a
Modules aka foo directory like:

./foo
Module1.jl
Module2.jl
...

and then use e. g. using Module1 in my codes (after adding the path to
the foo).

So the point here is that I have kind of packages without pacakge
environment. They are not large or general enough to be considered real
packages but still it would be a mess to use include to use them.

For me a simple option in VS Code itself would be sufficient but maybe
someone else will complain ;)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/julia-vscode/julia-vscode/issues/307?email_source=notifications&email_token=ABA2BXNOP3L6KO4RHTNJEQDRA7L6LA5CNFSM4D7HVDVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKTAHVI#issuecomment-581305301,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABA2BXK7G6WT6OLI3BSMUVLRA7L6LANCNFSM4D7HVDVA
.

[davidanthoff]

Awesome, this is super helpful! Couple more questions:

@norru which folder do you open as the root in VS Code? Presumably you open monorepo_root, right?

@stakaz in your workflow, I assume you have some folder open in VS Code that has some scripts, but then the foo folder is _not_ a child of the root folder that is open in VS Code?

I'm realising that we have at least one more scenario than I had initially thought: this kind of LOAD_PATH use sometimes points to a folder that is outside the folder that is currently open in VS Code, but sometimes it could also point to another folder that is currently open in VS Code. Those two scenarios will probably require a pretty different strategy...

@norru in your case, is there a reason that you just don't dev both lib/julia/Module1 and lib/julia/Module2 in your root Julia environment?

[norru]

which folder do you open as the root in VS Code? Presumably you open monorepo_root, right

Yes, that's correct for now, but I don't want to be bound to a specific folder structure in the long run.

is there a reason that you just don't dev both lib/julia/Module1 and lib/julia/Module2 in your root Julia environment?

Even if I do that, the linter has trouble picking up the symbols. In fact, to bypass the problems I've temporarily put everything in a single module. And even in that case, the linter doesn't pick up the module's symbols in all the other code using the module, for instance in runtests.jl

[ZacLN]

Hi, I've had a quick skim of the recent comments and think the way forward likely to be something like:
1) Add a LOAD_PATH setting to julia-vscode (an array of paths) that gets passed to a new field of StaticLint.AbstractServer instances (LanguageServerInstance is non-explicitly a subtype of this).
2) Packages defined within these paths (subject to the rules here) will be loaded when the server loads.
3) resolve_imports and associated functions will be changed to search through the packages defined by LOAD_PATH _after_ searching through the symbol server depot.
4) These modules will be treated as though they're user code (rather than a symbol server cache of the compiled module).

@davidanthoff , sound good?

[stakaz]

In my case the Module subdirectory is inside the whole project's root. However, sometimes I also just open julia in a terminal and use these modules. So in general I think the path should be independent of the root, really just the LOAD_PATH from the startup.jl.

[ufechner7]

My directory structure:

~/project
    /src
      Module1.jl
      Module2.jl
      Module3.jl
      project.jl
    /tests
      run_test1.jl
      run_test2.jl
      runtests.jl

runtests.jl is including run_tests1.jl, run_test2.jl etc.

project.jl is using Module1.jl, Module2.jl etc.

Module3.jl is using Module1.jl and Module2.jl

project.jl is the "main" program and the only program that is
using the modules, therefore it doesn't make sense to create
a package.

I just include the src folder in the LOAD_PATH.

Uwe

[kcajf]

I use the JULIA_LOAD_PATH environment variable to achieve this. Could be nice if vscode could read that in!

[aterenin]

I've got the same situation. As a workaround, I can add the following code to my scripts while developing

if isdefined(@__MODULE__,:LanguageServer)
  include("/path/to/source/file.jl")
  include("/path/to/source/file_2.jl")
end

and the linter starts working again.

What would be very nice is if I could somehow tell the linter that some directory src contains user-defined modules, and have it recursively load all files in it.

[davidanthoff]

Now, why on earth does that work? We should never run user code in the LS, so I would have thought that with any recent version of the extension, the trick that @aterenin describes here would not work... Weird, not clear to me why this works :)

[ZacLN]

At a guess I'd say we're ignoring the if block

[ufechner7]

I had to modify the workaround from @aterenin in the following way:

In Module3.jl I had to add:

if isdefined(@__MODULE__,:LanguageServer)
    include("Module1.jl")
    using .Module1
    include("Module2.jl")
    using .Module2
else
    using Module1, Module2
end

and in run_test1.jl I had to add:

if isdefined(@__MODULE__,:LanguageServer)
    include("../src/Module1.jl")
    using .Module1
    include("../src/Module2.jl")
    using .Module2
else
    using Module1, Module2
end

Now most of the warnings disappeared and I can use "goto definition" to find definitions in other modules etc.

Quite happy about this workaround!

I hope it will still work in the upcoming version of julia-vscode.

[davidanthoff]

So I honestly don't understand why this would work...

[AlexLewandowski]

The part that allows LanguageServer to autocomplete and jump-to-definition is the first if-block

include("../src/Module1.jl")
using .Module1
include("../src/Module2.jl")
using .Module2

Of course, the Julia instance that is launched by the language server must have the LanguageServer module. You could run the file just with this.

However, if you're using the REPL and running scripts from the package environment, it will not have the LanguageServer module. Hence, the second and more customary block is evaluated that just imports the modules in src/ i.e. using Module1.

However, I don't understand why the LanguageServer doesn't pick up the module in src/ by default like the REPL. Why doesn't autocomplete / jump-to-def work with just import and without the include?

[zhopan77]

I also noticed the workaround with "include", but would prefer not to use it for two reasons:

1. A very common use case: if you need to use the same module A both in the top script and inside another module B, while module B is also used by the top script, "include" gets messy. This is because every time "include" is used, the code in that file gets dumped into that location, and the scope totally depends on the scope of that location. So if you "include" A module at different places they may not be in the same scope (one is under Main while the other is inside B in this example), and type validation could fail. The answer by Graham Smith on Feb 13 '18 at stackoverflow.com explained this clearly.

2. If you use Revise.jl and therefore has to do "includet" instead of "include", linter/intellisense still does not work.

Therefore, a proper fix for linter/intellisense to see modules loaded from LOAD_PATH with "using" or "import" would be much cleaner.

Thanks!!!

[GregPlowman]

JuliaLang PR Allow specifying startup.jl via environment variable
Could this be relevant to LanguageServer LOAD_PATH issue?
Would it be possible to specify a custom startup.jl to the language server?

[davidanthoff]

No, we'll need to find a different solution here in general. We hope to statically compile the LS at some point, and then we won't be able to use anything like a startup file at all.

We are going to revamp the whole package resolve story at some point, and hopefully we can sort this LOAD_PATH situation out in one swoop with that as well.