Arduino-cli: Support bundling libraries with a sketch

[hoblins]

There are many more or less useless forum discussions in the web about how to handle libraries in the sketch folder, but this only works for simple header files and according cpp, but not for more complex Arduino libraries like WiFi101.
There are obvious benefits for developers, e.g. freeze libraries at a working state or apply little modifications to standard libraries. As an example there are major changes to the API of the very popular ArduinoJson library in version 6.
I don't have insight into the Arduino IDE, but identified two possible points:

1. As a workaround in the preferences the sketchbook location can be set to the folder of the own program. But this has to be done manually and changed every time another program is handled. So there could be some option for this to be set in the ino file.
2. In the build folder a file build.options.json is generated automatically.
   There are some options where additional library folders could possibly be added? E.g.
   > 1. builtInLibrariesFolders
   > 2. otherLibrariesFolders

[matthijskooijman]

I believe there are open issues about this already, did you look for them?

In any case, I'd really like to see the IDE support a libraries folder (as well as a hardware folder) inside the sketch directory, as you suggest here. That really helps to distribute a self-contained sketch, which is useful in some cases.

[per1234]

I looked for other open issues and found a couple related things, though none that would make this a duplicate:

• https://github.com/arduino/arduino-builder/pull/223 - Adds the requested feature, but only for zipped sketches.
• https://github.com/arduino/Arduino/issues/7555 - Doesn't seem to add anything of value beyond the information that was already in the https://github.com/arduino/arduino-builder/pull/223 thread.

The addition of recursive compilation of the src subfolder of the sketch folder does give us this feature to some extent. However, I suspect hoblins is already aware of that option from their browsing of the forum discussions and found it to be unsatisfying.

The problem with bundling libraries in the src subfolder is that all use of the #include <foo.h> syntax in the bundled libraries for files in the src subfolder, which would work fine if the libraries were installed normally, must be changed to use relative paths to those files. Best practices would be for libraries to not use the #include <foo.h> syntax for internal dependencies anyway so I don't much mind changing those, but when you have dependencies between bundled libraries it gets ugly (e.g. #include <foo.h> -> #include "../../foo/src/foo.h").

[waza-ari]

+1 from my side! Especially if you want to have a project archived and ready to run after a couple of months/years I’d love to see the libraries in every sketch. Also when distributing sketches this feature would be awesome and is the next best thing (easiest) thing to do next to a full blown package manager I’d say.

[probonopd]

Maybe PlatformIO is an option if you want this kind of functionality.

[alandsidel]

I want to add a +1 here too. I'm working on a series of sketches all utilizing the same set of core private headers, and having to make duplicate copies of them in every project because the compiler doesn't understand relative paths is a real pain in the neck. I have maybe half a dozen projects that need to #include "../../includes/somefile.h" and there doesn't seem to be any way to convince the arduino compiler to do this. Even just reverting to #include "somefile.h" and then making that file a symlink to the real file doesn't work.

[matthijskooijman]

I want to add a +1 here too. I'm working on a series of sketches all utilizing the same set of core private headers, and having to make duplicate copies of them in every project because the compiler doesn't understand relative paths is a real pain in the neck.

Huh? But how would the feature requested by this issue help here? This issue requests that a copy of a library inside the sketch folder can be used just like a normal shared library. For your usecase, it sounds like you just need to make your "core private headers" into a library, install it into your sketchook "libraries" directory along with the other libraries, and then you can just include these from all sketches that need them?

[alandsidel]

I want to add a +1 here too. I'm working on a series of sketches all utilizing the same set of core private headers, and having to make duplicate copies of them in every project because the compiler doesn't understand relative paths is a real pain in the neck.

Huh? But how would the feature requested by this issue help here? This issue requests that a _copy of a library_ inside the sketch folder can be used just like a normal shared library. For your usecase, it sounds like you just need to make your "core private headers" into a library, install it into your sketchook "libraries" directory along with the other libraries, and then you can just include these from all sketches that need them?

You're right, the issue as it stands wouldn't be helpful. I was referring to per1234's message a few up where he mentions e.g. #include "../../foo/src/foo.h" which itself wouldn't work either.

My personal goal is to get away from using the Arduino IDE and to VS Code for a variety of reasons, so adding to some IDE specific library manager doesn't seem like a good path forward. The forums have indicated to me that including with relative paths simply doesn't work in the Arduino IDE because the files are copied somewhere else prior to compilation, and it seems like the VS Code arduino plugin seems to do the same thing (I guess?) before calling avr-gcc, or however it enters the ecosystem's toolchain.

[per1234]

I was referring to per1234's message a few up where he mentions e.g. #include "../../foo/src/foo.h" which itself wouldn't work either.

I should clarify that when I mentioned #include "../../foo/src/foo.h" in my previous reply, that was referring to the #include directives in a bundled library that has a dependency on another bundled library, not to #include directives in a sketch.

I'll provide a more detailed explanation:

Let's say you have a sketch with a structure like this:

|_ MySketch.ino
|_ src
     |_ bar
     |    |_library.properties
     |    |_src
     |        |_ bar.h
     |_ foo
          |_ library.properties
          |_ src
               |_ foo.h

With a normal library installation to the libraries subfolder of the sketchbook, if the "bar" library has a dependency on the "foo" library, it can just use #include <foo.h> and the Arduino library dependency resolution process automatically finds the "foo" library. However, if I want to bundle the "bar" and "foo" libraries with my sketch, as in my example structure above, then I must edit the "bar" library and modify #include <foo.h> to provide the relative path to foo.h: #include "../../foo/src/foo.h".

The forums have indicated to me...

Here's the forum thread @alandsidel mentioned: https://forum.arduino.cc/index.php?topic=699636

[Steve132]

I can't believe how this is STILL not complete. It should be so easy to implement.

[alandsidel]

I can't believe how this is STILL not complete. It should be so easy to implement.

I agree. I've found temporary workaround for the time being. In addition to moving away from the Arduino IDE to VS Code, I decided to just make a symlink/junction in the Arduino libraries directory. This is undesireable because it's a manual process I have to remember to do in every development environment, or if/when I need to reinstall on my workstation, but at least it's something I can put in e.g. a readme & Makefile target for future reference.

It's disheartening that simple quality of life features like this for more experienced developers are seemingly rejected or delayed only because they might, sometimes, in rare instances, add a tiny bit of confusion to a beginner.

[pwyq]

+1. It's would be super helpful if this can be implemented.

[forkineye]

Editing libraries to support this or forcing the users to edit build options isn't the answer. Simple automatic support as in arduino/Arduino#11106 would allow us to use git submodules to keep larger projects in tact and properly versioned. Are there any targets or milestones for some sort of implementation or active development on a PR?

[alandsidel]

Editing libraries to support this or forcing the users to edit build options isn't the answer. Simple automatic support as in arduino/Arduino#11106 would allow us to use git submodules to keep larger projects in tact and properly versioned. Are there any targets or milestones for some sort of implementation or active development on a PR?

I'm really not in favor of making the "official" way to do this be via git submodules. Not all of us use git to manage every project, and even when we do, there are good reasons to not use submodules. This would force developers into choosing between not having the build chain work the way they expect like now, or to reorganize their code into submodules that may cause other problems elsewhere in their development cycle.

The problem is in the toolchain as it exists now, and that's where it should be fixed IMHO.

[forkineye]

Editing libraries to support this or forcing the users to edit build options isn't the answer. Simple automatic support as in arduino/Arduino#11106 would allow us to use git submodules to keep larger projects in tact and properly versioned. Are there any targets or milestones for some sort of implementation or active development on a PR?

I'm really not in favor of making the "official" way to do this be via git submodules. Not all of us use git to manage every project, and even when we do, there are good reasons to not use submodules. This would force developers into choosing between not having the build chain work the way they expect like now, or to reorganize their code into submodules that may cause other problems elsewhere in their development cycle.

The problem is in the toolchain as it exists now, and that's where it should be fixed IMHO.

This has nothing to do with forcing the usage of git submodules. It just allows people to easily support them and keep local library code still separated. Did you even read it?

[alandsidel]

This has nothing to do with forcing the usage of git submodules. It just allows people to easily support them and keep local library code still separated. Did you even read it?

Yes, I did. Do you understand the problem mentioned by the issue you're posting in?

If the problem is fixed in the build tools, then submodules will automatically and transparently be supported without requiring special handling. There's no reason to add any special handling for git submodules, and the tool chain shouldn't even know or care what revision control system you're using, if any, in order to work properly.

[forkineye]

This has nothing to do with forcing the usage of git submodules. It just allows people to easily support them and keep local library code still separated. Did you even read it?

Yes, I did. Do you understand the problem mentioned by the issue you're posting in?

If the problem is fixed in the build tools, then submodules will automatically and transparently be supported without requiring special handling. There's no reason to add any special handling for git submodules, and the tool chain shouldn't even know or care what revision control system you're using, if any, in order to work properly.

Yes, I did... It's called a use case. You could delete everything about git and submodules from my issue and it'll still stand. There's no special handling required. FFS read before you reply.

[Steve132]

I implemented this feature and made a pull-request https://github.com/arduino/Arduino/pull/11110