Platformio-core: Cannot use shared library storage with lib_deps

[mcspr]

---

Configuration

Operating system:
Linux

PlatformIO Version (platformio --version):
PlatformIO, version 4.0.0b2

Description of problem

Having a project with multiple environments that re-use same lib_deps variable, I encountered a confusing behaviour with global library storage:

1. Global library storage is ignored with specific versions from lib_deps variable (library @ version or github links)
2. It is seemingly impossible to specify shared library storage for multiple environments. libdeps_dir is following new rules for workspace_dir, forcing each environment to create a subdirectory {libdeps_dir}/{env}/...library...

Steps to Reproduce

The content of platformio.ini:

[env:a]
platform = espressif32
board = lolin32
framework = arduino
lib_deps =
    [email protected]

[env:b]
platform = espressif32
board = lolin32
framework = arduino
lib_deps =
    ArduinoJson

The content of src/dummy.cpp

#include <Arduino.h>
void setup() {}
void loop() {}

$ pio lib -g install [email protected]
Library Storage: /home/builder/.platformio/lib
Looking for ArduinoJson library in registry
Found: https://platformio.org/lib/show/64/ArduinoJson
LibraryManager: Installing id=64 @ 5.13.4
Using cache: /home/builder/.platformio/.cache/7e/ba586a56b4024a92728654afd0862c7e
Unpacking  [####################################]  100%
ArduinoJson @ 5.13.4 has been successfully installed!

$ pio run -e a
Processing a (platform: espressif32; board: lolin32; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/lolin32.html
PLATFORM: Espressif 32 1.8.0 > WEMOS LOLIN32
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: toolchain-xtensa32 2.50200.80 (5.2.0), framework-arduinoespressif32 2.10002.190416 (1.0.2), tool-esptoolpy 1.20600.0 (2.6.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Looking for ArduinoJson library in registry
Found: https://platformio.org/lib/show/64/ArduinoJson
LibraryManager: Installing id=64 @ 5.13.4
Using cache: /home/builder/.platformio/.cache/7e/ba586a56b4024a92728654afd0862c7e
Unpacking  [####################################]  100%
ArduinoJson @ 5.13.4 has been successfully installed!
Found 58 compatible libraries
Scanning dependencies...
^CBuild interrupted.

$ pio run -e b
Processing b (platform: espressif32; board: lolin32; framework: arduino)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/lolin32.html
PLATFORM: Espressif 32 1.8.0 > WEMOS LOLIN32
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: toolchain-xtensa32 2.50200.80 (5.2.0), framework-arduinoespressif32 2.10002.190416 (1.0.2), tool-esptoolpy 1.20600.0 (2.6.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 57 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ArduinoJson> 5.13.4
^CBuild interrupted.

Examining libdeps contents:

$ ls .pio/libdeps/*
.pio/libdeps/a:
ArduinoJson_ID64

.pio/libdeps/b:

I have also tried adding [env:...] lib_extra_dirs = ./my_shared_libs and installing the library there.

Actual Results

lib_deps = ... library gets installed in the .pio/libdeps/env/... despite being in the global storage

Expected Results

Library presence is checked in global or lib_extra storage dir before installing it in the {workspace_dir}/libdeps storage.
Which I think I solved locally for me via mcspr@c231535e

It would also be extremely useful to still be able to use pio run -e ... with old-style .piolibdeps storage when there is no practical reason of having multiple copies of the same library in .pio/libdeps/.../library
For example, when environments only differ in src_build_flags values or without internet connection present.
I wonder if this can be solved by making default libdeps_dir actually reference that it uses current environment as a part of the path (e.g {workspace_dir}/libdeps/{pioenv})?

[ivankravets]

Please re-test with pio upgrade --dev.

[mcspr]

Yes, thanks!

What about the use of global storage with lib_deps? i.e. my second question was, is it possible to place $PIOENV here?
https://github.com/platformio/platformio-core/blob/4123aa4c234ed99f9b4550f94d9d0dc6036b426c/platformio/project/helpers.py#L130-L133
And change all relevant places to substitute in on-demand (either from cli option, config or scons env)
That way lib_deps can be global when it is possible, keeping the fix for #1696

Do I need to open another issue for this? Or is there some other way this will / can be done? The goal is to still use lib_deps variable for dependencies in case there are no conflicts (and since it is not possible to do pio lib -e default_env_with_shared_lib_deps -g install, but even if that were the case, that is still a separate step to manually do). afaik, the code right now only installs in libdeps_dir storage.
For example, like ini above but all having global [env]:lib_deps and [env]:platform=espressif32,framework=arduino

[ivankravets]

Just add global storage to the top of list using http://docs.platformio.org/en/latest/projectconf/section_env_library.html#lib-extra-dirs

[mcspr]

Yes, that does work with already installed libs. But there are no libraries yet, the installation will only happen in the project's .pio/libdeps/...

[ivankravets]

But this is the right behavior? If you don't like the default behavior, you can install packages manually and extend lib_extra_dirs. Another option is to use Pre Script and call pio lib -g manually.

env.GetProjectOption("lib_deps")

[mcspr]

It depends, as I have explained. My hope was it could be resolved via ini variable pointing to the exact installation directory. I will try extra script, thanks for the pointer.

[mcspr]

BTW, I did try extra script just now. Mini template project with subprocess and LibraryManager (probably OK too?)
https://gist.github.com/mcspr/80829350513faf37d3154ab67c40ae8b
Do you think this can be added to the extra_scripts examples? Being both an example of the GetProjectOption (which I did not notice before!) and general library management.

[ivankravets]

I like your implementation! Let keep this in mind when someone will ask. We don't want to support different workflows in official docs. It will help us to save resources when developers will have any issues.

Extra scripting was specially created for these cases when you don't like default PlatformIO Core behavior. So, everyone can tune it for own needs.