Vscode-cmake-tools: [feature request] can module systems be supported?

[svenevs]

Goal

I'm just starting out here, I read the docs start to finish and don't believe such functionality exists yet. I am interested in helping tackle support for module systems (I use LMod, but traditional TCL modules would be obligatory).

I believe the best way to implement this would be some kind of addition (probably to a kit?). I don't know how this would pan out though, because as far as I can tell you do not get consistent access to the same terminal session.

Overview of Typical Workflow

Say I have project foo/ I'm just building from my terminal.

$ mkdir gcc_7.3.0_build/ && cd gcc_7.3.0_build/
$ ml load gcc/7.3.0 # the thing I am trying to figure out how to support here
$ cmake ..

In a traditional module based workflow, if we go back to the repo root and want to test another stack I would say

$ mkdir clang_5.0.0_build && cd clang_5.0.0_build/
$ ml switch clang/5.0.0
$ cmake ..

But for the purposes of this tool, trying to add in a comprehensive listing of

1. Tracking currently loaded modules
2. Deciding whether load vs switch needs to happen
3. ........

I think that what could potentially be successful is an addition to the kit options that just runs a raw command (so not actually module specific).

{
    "name": "My Compiler Kit",
    "commands": [
        "ml load gcc/7.3.0",
        "ml load openmpi/3.1.2",
        "ml load hdf5/1.10.2"
    ],
    "compilers": {
        "__comment__": "properly crafted compiler modules should set these",
        "__comment__": "doing this '__comment__' stuff for syntax highlighter xD",
        "C": "${env:CC}",
        "CXX": "${env:CXX}",
        "Fortran": "${env:FC}"
    }
}

Where each command in commands is executed _verbatim_ (aka user at fault for invalid commands), and equally important: _in order_.

1. Since the modules themselves _should_ set CC, CXX, and FC respectively, maybe instead of finding a way to make sure that commands are run for evaluating "compilers", it would be better to just introduce a "trap" e.g., "C": false to indicate to CMake Tools _please do not set CC environment variable or -DCMAKE_C_COMPILER=..._.
   
   
   
   • Or can we just omit "compilers" altogether?
   
   
2. Order matters for LMod specifically, it's a hierarchical module system. So hdf5 won't be 'visible' until I load OpenMPI.
3. If you aren't aware, what the module systems do are basically just prepend to the important variables: $PATH, $LD_LIBRARY_PATH, $CMAKE_MODULE_PATH, and friends. So they really do need to be loaded for everything: configure, build, running targets, all of it :/

Anticipated Difficulty

The reason a CMake Toolchain file cannot be used for this is because the module needs to be loaded before CMake even starts executing. For example, the module system may change which cmake shows up in $PATH first. Even if that wasn't a problem, the compiler modules should be setting CC, CXX, and FC which needs to happen before CMake runs.

Basically, the problem is that executing the module commands in a subshell will not affect the parent shell. So say we were talking about the configure stage. You would basically need take

$ cd ${build_dir}
$ cmake .. -DCMAKE_......

and wrap it

$ commands[0]
$ commands[1]
$ commands[N]
$ cd ${build_dir}
$ cmake .. -DCMAKE_......

all being executed in the same subshell.

Is this something a VS Code extension can do without becoming overbearingly difficult to maintain?

Other Notes/Information

• The reason I like the commands approach as to doing something module specific is there are a lot of nuances / edge cases for modules you would have to bake in for no real reason. Having commands makes it so that this feature could potentially be used for a myriad of other applications, and we wouldn't have to be bothered by details such as the module command printing to stderr for example.
• I don't know if supporting this on Windows consoles will be easy. For *nix you could basically just ";".join(commands) so to speak.
• I am entirely unattached to this being specific to a kit. I don't really understand how all of the pieces fit together, but this seemed to be the right location.
• I am absolutely willing to help implement this (though I'll probably need some hand-holding on how to actually test the extension in VSCode xD).

Thanks for being you, the CMake Guru / champion for the people. We are lucky to have you! I look forward to your thoughts, even if it's just "that's beyond the scope of this extension".

P.S. I did promise you I'd "break" your extension ;)

[vector-of-bool]

Hello! Sorry for the delay. I've been wrapped up in my full-time job and Pitchfork stuff, as I'm sure you're aware.

This would be best implemented as a Kit, definitely. There is an older feature request open regarding using arbitrary shell scripts to source the environment for a kit.

Another option would be dedicated code for working with tools like LMod, where the code specifically understands how to talk to LMod, and you Kit would explicit call out LMod modules, something like this:

{
    "name": "My Awesome Kit",
    "lmod-load": [
        "gcc/7.3.0",
        "openmpi/3.1.2",
        "hdf5/1.10.2"
    ]
}

I hate having to "shell out" to programs blindly. If module tools like this received dedicated support in this way, it would be easier to handle bugs and more closely integrate with them. I don't know much about LMod or "modules" systems in general. This is a new discovery for me.

Thoughts?

[svenevs]

I've been wrapped up in my full-time job and Pitchfork stuff, as I'm sure you're aware.

Yes. I know the feeling x0

I hate having to "shell out" to programs blindly

A valid concern, pandoras box and all. If you think that dedicated support for module systems makes sense, then I can definitely help tackle this. So yeah the specification for users via kits would be really straightforward / desirable, the promise that CMakeTools makes is to always load them in order. In general, this isn't necessary for TCL modules, but there's no harm in loading in order for TCL.

I'm sure there are other kinds of environment modules available, but realistically supporting TCL and LMod is probably good enough. LMod is in lua, but has absolutely zero issues loading TCL modules (which is really convenient). For reference, most linux distros have an environment-modules or similarly named package that is the TCL modules, which provides the module command. LMod replaces the same command (module is now an LMod thing), and they have an alias called ml. So CMakeTools would just module load x for every x.

1. For the kit, we can just name it module-load, since it will never be the case that a user needs to distinguish between TCL and LMod.
2. As far as I know these tools cannot be used on Windows, so it's worth remembering stating that in the future docs. Should we just say "don't use on windows unless executing module load x will not result in an error?
3. The one thing that is kind of frustrating about module is that output is always put on stderr. The reason to mention it is you can only rely on the exit code, not the presence of output on stderr (not that you would do that anyway...).

Do you have any pointers of where I should start poking around to try and test out if module load can even be done / work?

[vector-of-bool]

Take a peek in kit.ts, which does all the kits stuff. It's getting kind of large, so you may want to put it in a new .ts file (I don't recommend calling it module.ts, since that is nightmare confusing :stuck_out_tongue:).

See if you can get the environment variables loaded from a module system, there you'll go. It may take a bit of poking to figure out what the scatter/gather thing is doing where I start a dozen kit-checks in parallel, but it shouldn't be completely inscrutable? Use proc.ts to execute subprocesses. It'll handle a lot of the piping and stuff for you (including splitting of stdout/stderr).

Before worrying about the docs or anything I'd say to just take a stab at it and see what you can do.

Thanks!

[svenevs]

LOL. So I took a look around and was just kind of playing with things. The more I changed the more confused I got until I realized a very important fact: when I launch code . from the terminal, it inherits from the current shell. That is, if I already had ml load gcc up in there and then do code ., the interactive terminal for VSCode has the module loaded. So ctrl+shift+p and edit cmake, and then I added this thing

  {
    "name": "Module Based",
    "compilers": {
      "C": "${env:CC}",
      "CXX": "${env:CXX}",
      "Fortran": "${env:FC}"
    },
    "keep": true
  }

For me personally, since this works, I see no reason to implement a full feature for two reasons:

1. My stacks have all the same software, just compiled with different compilers. I won't be using CMake Tools to build my software, I only want IntelliSense. So as long as I remember to ml load gcc and code ., I can use the above kit to have completion which is amazing.
2. Given that VS Code is inheriting, this makes implementing this feature definitively more difficult. If modules are already loaded, do you purge? It presents a lot of problems with no clear-cut solution.

So basically, for my use-case (I just want IntelliSense), I can get around this. Thanks for entertaining the idea / hashing it out with me @vector-of-bool! I'm glad I accidentally discovered this trying to implement the feature :upside_down_face:

P.S. It's not that I'm saying the rest of the features for building etc are bad, it's a beautiful plugin. I just work better in a terminal directly :wink:

[vector-of-bool]

Good to hear you've solved your issue. But do note that VSCode uses 'unique'-style processes. If you already have a code instance running and run code <path>, it may or may not start a new VSCode window with the environment from the shell. It may re-use the same environment from the already running instances. This is why I added the VS kits rather than just asking people to start VSCode from a dev command prompt.