arduino-cli should provide "environments"

[scls19fr]

Hello,

I wonder if the concept of "environment" (comparable to conda environments or Julia Pkg environments) have been considered (it could help to have several versions of a library installed without odd effects).

Here is a possible workflow for dealing with environment with arduino-cli :

Create an environment named myenv

$ arduino-cli env create myenv

It should create an empty directory name myenv in ~/Documents/Arduino/envs

List all existing environments

$ arduino-cli env list

Add packages to this environment

Latest version of a package

$ arduino-cli lib install WiFi101 --env myenv

A given version

$ arduino-cli lib install "ArduinoJson==5.13.4" --env myenv

(this will need #105 to be fixed)

(An environment could also be created directly from a file but this could probably be implemented later)

I assume core is installed in the global environment (because I think it's important to keep ability to have libraries and core installed in a shared manner)

$ arduino-cli compile --fqbn arduino:samd:mkr1000 Arduino/MyFirstSketch --env myenv

This last command will use ArduinoJson==5.13.4 even if, for example, ArduinoJson==6.x.y is installed in the "global" environment.

What is your opinion of such an idea?

Kind regards

[scls19fr]

I assume core is installed in the global environment (because I think it's important to keep ability to have libraries and core installed in a shared manner)

Just an additional idea: libraries installed in an environment should "override" libraries installed in the global (shared) environment

[skandragon]

I faked this up using a Docker image per "environment" I want to use. This lets me be very explicit about what each build platform sees. It's a WIP, and not currently public, but if there is interest I can share.

I know others have similar base images, but for some reason I chose to make one from scratch.

[BinaryBen]

I'm not personally keen on environments, but having a system like the Node Package Manager (NPM) for managing libraries would be awesome. It could just use Github Repos to start with (no need to set up an official library registry... yet):

arduino-cli install --save @FastLED/FastLED

I would love to have a package.json file (or similar) though which lists required libraries, and then have the cli download them to the sketch directory as needed. You can also allow the user to install them globally like npm does, and that would fulfill @scls19fr's request.

EDIT:

Hold up. Libraries can be saved to the the ./src directory in a sketch directory. I could write a simple package manager cli utility that makes use of that fact. It would be a simple case of checking a package file and downloading the relevant code to ./src before calling arduino-cli separately. Is there a use case that having environments provides additional benefits that this possible solution wouldn't @scls19fr?

[masci]

+1 for using Docker, we're working on official releases for images shipping the CLI that should make this even easier.

You can already have something really close to an env by setting up ARDUINO_DATA_DIR and ARDUINO_SKETCHBOOK_DIR env vars to different values for different envs. You could even use Conda itself to manage such environments but any other dot-env system would work as well.

At this point we've no plan to add a sophisticated env manager with specific commands and stuff like global fallback but let's keep this issue open to collect feedback and alternative approaches.

[matthijskooijman]

32 was closed, pointing to this issue for further discussion. The core of that issue was to be able to include libraries in the sketch directory, in order to make a sketch directory more self-contained and easier to distribute to others.

In this issue, a broader solution using environments is proposed, which serves a similar goal, but as @masci already says is quite a lot more sophisticated and thus less likely to be implemented.

Also, it seems that such an environment-bases approach needs more setup (e.g. it needs the user to explicitly create an environment and install libraries into it), while the simplier "libraries subdir in the sketch directory" approach is more implicit. The way I envison that, it only requires distributing library files along with the sketch, which will be used automatically, without any specific action required on the part of the user that compiles the sketch. Something similar could be achieved with an environment-based approach, but that does require some additional thought.

[federicobond]

I have been looking into this topic for a while in order to improve the tooling we use and get a nice reproducible environment for an Arduino project I am working on, and I think there is a key insight here:

You can already have something really close to an env by setting up ARDUINO_DATA_DIR and ARDUINO_SKETCHBOOK_DIR env vars to different values for different envs.

arduino-cli is 90% of the way there with regards to having usable separate environments. As long as you prefix each command with ARDUINO_SKETCHBOOK_DIR=., libraries are installed in ./libraries relative to the current directory and sketches are built using those libraries only. What would be really helpful is for the CLI to recognize that we are on a project laid out in such a way and set the sketchbook directory to the project root automatically.

This can be done by placing a special file in the project root directory. We could call this file project.json or similar. The most important thing this file does is to put the CLI into this new "project" mode so the sketchbook directory is configured appropriately and libraries are installed locally by default. We could also list those libraries and their versions in the file (so you can install them all easily by running a single command like arduino-cli install), set a default board fqbn (so you don't need to add the --fqbn arg to each compile program) and other default compile settings.

I think this solves a bunch of common problems previously reported by users and requires little change to the CLI. I would love to hear your opinions on it!

[c4deszes]

I started working on a CI/CD pipeline for embedded systems and Arduino seemed like an easy target to start with. What I'd like is a stateless environment, if the developer has arduino-cli he should be able to build the project with minimal setup required. What npm gets right is that it reduces everything down to 2-3 commands to fully build and deploy the project.

This change should allow at least install, compile and upload to be used without any parameters, all of them relying on the file present in the current workspace (like project.json, arduino-cli.json, etc.).

Install should pull all the cores and libraries necessary to build the project. You shouldn't have to specify which cores, rather potential targets.

Compile should create binaries for all target boards, right now the build output is straight into the project folder and upload depends on the output file name and location which is somewhat annoying.

Upload should try to upload to a currently available board if there's a build for it. If the target is ambiguous it should prompt a port or FQBN specification.

A draft of what the .json might look like.

{
    name: "ProjectName"
    libraries: {
        "FTDebouncer": "1.3.1"
    },
    targets: [
        "arduino:avr:uno",
        "arduino:samd:zero"
    ],
    outputDir: "/Project/output"
}

So after these changes in a never before used environment all I need is:

1. pull repository
2. arduino-cli install
3. arduino-cli compile
4. arduino-cli upload

I want to dockerize the build but also allow local builds, if I have to change the scripts I'll have to change the Dockerfile, not to mention that I want to use Github Actions as well. So simplifying these commands would make the process consistent no matter what the build environment is.

[federicobond]

Apparently, there is already a way to attach a board target to a sketch via the arduino-cli board attach command, which generates a sketch.json file inside the sketch directory. This allows for per-sketch target board configuration. The targets defined in a sketch.json file should override any sketchbook global configuration.

[MacroYau]

I have been looking for similar things for a while too. Just implemented a proof-of-concept Dockerized arduino-cli compile that also enables per-project dependency (core and library versioning) management via a YAML file: https://github.com/MacroYau/arduino-cli-compile-docker

[netik]

When people have to resort to docker to solve this problem, we've failed as a community. Why can't the cli just support things the way C has been handling libraries for decades? INCLUDES= and LDFLAGS= would be a nice start, same goes for -I and -L.

The same goes for 'environments'. There is little to no reason to have environments if standard library support is permitted.

It really feels like a lot of the work on the arduino side of things is 'reinventing makefiles'.

[drstevenhale]

My solution to this is to leverage the existing options in arduino-cli.yaml, and the ARDUINO_DATA_DIR environment variable, as mentioned above. I use a makefile where "make env" installs the arduino-cli, and the specified core and libraries. Everything is downloaded and cached within the project, essentially making it an isolated environment. The entire pipeline is reduced to four commands. Clone, make env, make, make upload.

If anyone is interested, I've got a simple example here.

[ianfixes]

@c4deszes if you're looking for CI, check out arduino_ci or the accompanying Arduino CI github action (full disclosure, I maintain them).

@scls19fr could this be solved by placing multiple libraries/ directories in your filesystem? For example, if you did mkdir -p /path/to/environments/myenv/libraries and then put the following in an arduino-cli.yaml:

directories:
  user: /path/to/environments/myenv

I do this in the arduino_ci test suite to enable tests against dummy libraries.