Arduino-cli: Feature Request: Export compile_commands.json

[sslupsky]

A useful enhancement to the arduino cli would be the option to generate a compile_commands.json file. This can be used by Microsoft's Visual Studio Code (vscode) application to configure the its' IntelliSense code completion, parameter info, quick info, and member lists capabilities. I find using arduino-cli and vscode is particularly productive for embedded development.

[sudobash1]

I think this is a cool idea, and I have wanted this feature. I think that it would be at least slightly complicated by how Arduino preprocesses the .ino file. It doesn't just feed it directly to the compiler. You can learn about that here.

Without this preprocessing, IntelliSense, clangd and ccls might think they are looking at invalid C++ (which they would be).

cheers

[matthijskooijman]

I'm also interested in this feature, since my current manual compiler-flags configuration for code completion with clangd is a bit fragile (and does not have a complete list of files in the project, preventing refactoring from working).

As for the .ino file, it will indeed be pretty much impossible to let clangd see the preprocessed version, especially when you're editing it (unless the Arduino IDE talks to clangd and does the preprocessing on the fly, but that's not the subject of this issue). But I think it would actually be fine if all .ino files would end up in the compilation database with just the flags used to compile the preprocessed version. This means that if you have just a single .ino file (or multiple that do not depend on each other without forward declarations), and you do not depend on the autogenerated forward declarations (by defining functions in order and/or adding forward declarations manually), clangd should be able to figure things out most of the time.

One additional thing is that preprocessing adds an #include <Arduino.h> line, but I think this could be emulated by adding -include Arduino.h to the compiler flags as well (this could even have been used for the actual compilation, except that would break compatibility now).

So, outputting a compilation database would simply be a matter of dumping all compilation commands ran, and some special casing for the .ino files, which should be fairly easy to implement.

Where should this file be generated? I suspect that the sketch directory is the obvious place, since that's also where editors / clangd will look for it, right?

One thing to think about is partial compilation: When compilation fails, should a database be written with just the commands that were ran? Probably yes if no database existed yet, but what if a database was already present? Then it might be counterproductive to remove entries for files after the failure, since very likely the flags haven't changed. Maybe an existing database should be loaded, updated in memory and written out at the end of the compilation? If compilation failed, it should be written out as-is, if compilation succeeded, any files in the database that were no longer in the compilation should probably be pruned.

[matthijskooijman]

I did a rough implementation of this feature, see the PR linked above. It's still far from finished, but it might serve to trigger some additional discussion. See the PR for a list of things I think need to be addressed.

[ubidefeo]

thank you @matthijskooijman
I saw the PR come in but it's not my turf, so I'll get the team to check it out on monday :)

enjoy the weekend
u.

[matthijskooijman]

Initial support of this has been merged in #1081. Since that merge, all compilations will generate a compile_commands.json file in the build directory. To generate a complete file without actually compiling everything (i.e. when you still have errors in your code), you can use compile --only-compilation-database.

Some open points (notes copied from my earlier attempt in #944):

• Where to write the file: I now write a compile_commands.json to the sketch directory, since that's where editors would look for it. I later realized that it would probably be good to also write a version to the temp build directory (note: I mean the /tmp/arduino-sketch-XYZ directory, not the build subdirectory of the sketch dir where the compilation result is exported), for any tooling that wants to inspect the copied and preprocessed sketch files.
• Updating sketch paths: Currently, the paths are written as they passed to the compiler, so pointing at the copy of the sketch in the temp build dir. This is probably good for the compile_commands.json to be written to the temp build dir, but the sketch dir version should really point to the original sketch files. This means that the file element in the json should be updated, but maybe also any compiler arguments that resolve to sketch files should have their paths updated. This should probably be applied generically (i.e. search replacing back to the original sketch dir) to all arguments, since we can't be sure what a platform.txt does exactly. Alternatively, updating just the file field might be sufficient and can be done by passing the original filename around (does require significant refactoring, I think).
• Handling .ino files: Currently, only one entry is generated for .cpp file resulting from the combined and preprocessed .ino file(s). This is again fine for the temp build version, but for the sketch dir version, I think this should have a separate entry for each .ino file, with an -include=Arduino.h flag added (as suggested in #849 (comment)). This is imperfect, but probably as close as we can get.
• Making this optional: I believe that writing a compile_commands.json to the temp build dir can always be done, but writing to the sketch directory should almost certainly be opt-in (through commandline or config), to prevent user surprise and because the IDE and arduino-cli seem to try hard to prevent touching the sketch dir now. For arduino-cli, it might make sense to always generate the file in the build subdirectory, along with the compiled program, and provide an option to also, or instead of, generate in the sketch directory itself (where it will be mostly automatically picked up by editors or clangd). For the IDE, some extra care should probably be taken to not generate this file in the sketch directory for unmodified examples, even if the user enabled it in the config.