Platformio-core: Library manager fails to install libraries from git URLs in certain environments

[zenflow]

What kind of issue is this? Bug in PlatformIO Core

Configuration

Operating system: Windows 10 Home

PlatformIO Version (platformio --version): 5.0.1

Description of problem

It seems that in certain environments (such as mine) the library manager fails to install libraries from git URLs.
For example, the project I am trying to run has added https://github.com/earlephilhower/ESP8266Audio#22b52e0ed5aa86a5e5704c5c86d435c8e3e233a0 under lib_deps, and now I get an error when installing libraries for the project. I assume that this normally works, since it was committed upstream, and comes with changes in the project that use that library, so it must have been working for them. See "The contents of platformio.ini" below.

Steps to Reproduce

1. Use a system with Windows 10 Home. Having cygwin installed may also be necessary.
2. Create a folder and in it create a platformio.ini file with the contents from below ("The contents of platformio.ini").
3. In that folder, run pio lib install (in PowerShell or cmd.exe, doesn't matter which)

Actual Results

Once it gets to installing (or trying to install) the library with the git URL, we see these error messages:

Library Manager: Installing git+https://github.com/earlephilhower/ESP8266Audio#22b52e0ed5aa86a5e5704c5c86d435c8e3e233a0
git version 2.17.0
Cloning into 'C:\Users\Matt\.platformio\.cache\tmp\pkg-installing-zahhj8v6'...
fatal: Invalid path '/cygdrive/c/Users/Matt/.platformio/.cache/tmp/pkg-installing-zahhj8v6/C:\Users\Matt\.platformio\.cache\tmp\pkg-installing-zahhj8v6': No such file or directory
Error: VCS: Could not process command ['git', 'clone', '--recursive', 'https://github.com/earlephilhower/ESP8266Audio', 'C:\\Users\\Matt\\.platformio\\.cache\\tmp\\pkg-installing-zahhj8v6']

And the command exits with a non-zero exit code.

Expected Results

I expect it to successfully install the library specified with a git URL.

If problems with PlatformIO Build System:

The content of platformio.ini:

; PlatformIO Project Configuration File
;
;   Build options: build flags, source filter
;   Upload options: custom upload port, speed and extra flags
;   Library options: dependencies, extra library storages
;   Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; https://docs.platformio.org/page/projectconf.html

[env:ttgo-t-watch]
platform = espressif32
board = ttgo-t-watch
framework = arduino
lib_archive = true
board_build.f_flash = 80000000L
board_build.flash_mode = qio
board_build.embed_txtfiles = 
    src/gui/mainbar/setup_tile/time_settings/timezones.json
monitor_speed = 115200
monitor_filters = 
    default
    esp32_exception_decoder
build_flags = 
    -DCORE_DEBUG_LEVEL=3
    -mfix-esp32-psram-cache-issue
src_filter = 
    +<*>
lib_deps = 
    TTGO TWatch Library@=1.3.0
    ESP Async WebServer@>=1.2.0
    AsyncTCP@>=1.1.1
    ArduinoJson@>=6.15.2
    gianbacchio/ESP8266Spiram@^1.0
    luc-github/ESP32SSDP@>=1.1.1
    IRremoteESP8266@>=2.7.10
    PubSubClient@>=2.8
    https://github.com/earlephilhower/ESP8266Audio#22b52e0ed5aa86a5e5704c5c86d435c8e3e233a0
    earlephilhower/ESP8266SAM@^1.0

[zenflow]

When I run pio run the same error occurs, but includes a stack trace:

C:\Users\Matt\Documents\dev\My-TTGO-Watch>pio run
Processing ttgo-t-watch (platform: espressif32; board: ttgo-t-watch; framework: arduino)
------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/ttgo-t-watch.html
PLATFORM: Espressif 32 (2.0.0) > TTGO T-Watch
HARDWARE: ESP32 240MHz, 320KB RAM, 16MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 3.10004.201016 (1.0.4)
 - tool-esptoolpy 1.20600.0 (2.6.0)
 - toolchain-xtensa32 2.50200.80 (5.2.0)
Converting my-ttgo-watch.ino
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Library Manager: Installing git+https://github.com/earlephilhower/ESP8266Audio#22b52e0ed5aa86a5e5704c5c86d435c8e3e233a0
git version 2.17.0
Cloning into 'C:\Users\Matt\.platformio\.cache\tmp\pkg-installing-35j8vz3p'...
fatal: Invalid path '/cygdrive/c/Users/Matt/.platformio/.cache/tmp/pkg-installing-35j8vz3p/C:\Users\Matt\.platformio\.cache\tmp\pkg-installing-35j8vz3p': No such file or directory
VCSBaseException: VCS: Could not process command ['git', 'clone', '--recursive', 'https://github.com/earlephilhower/ESP8266Audio', 'C:\\Users\\Matt\\.platformio\\.cache\\tmp\\pkg-installing-35j8vz3p']:
  File "C:\Users\Matt\.platformio\penv\Lib\site-packages\platformio\builder\main.py", line 169:
    env.SConscript("$BUILD_SCRIPT")
  File "C:\Users\Matt\.platformio\packages\tool-scons\scons-local-4.0.1\SCons\Script\SConscript.py", line 598:
    return _SConscript(self.fs, *files, **subst_kw)
  File "C:\Users\Matt\.platformio\packages\tool-scons\scons-local-4.0.1\SCons\Script\SConscript.py", line 287:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "C:\Users\Matt\.platformio\platforms\espressif32\builder\main.py", line 223:
    target_elf = env.BuildProgram()
  File "C:\Users\Matt\.platformio\packages\tool-scons\scons-local-4.0.1\SCons\Environment.py", line 219:
    return self.method(*nargs, **kwargs)
  File "C:\Users\Matt\.platformio\penv\Lib\site-packages\platformio\builder\tools\platformio.py", line 62:
    env.ProcessProjectDeps()
  File "C:\Users\Matt\.platformio\packages\tool-scons\scons-local-4.0.1\SCons\Environment.py", line 219:
    return self.method(*nargs, **kwargs)
  File "C:\Users\Matt\.platformio\penv\Lib\site-packages\platformio\builder\tools\platformio.py", line 140:
    project_lib_builder = env.ConfigureProjectLibBuilder()
  File "C:\Users\Matt\.platformio\packages\tool-scons\scons-local-4.0.1\SCons\Environment.py", line 219:
    return self.method(*nargs, **kwargs)
  File "C:\Users\Matt\.platformio\penv\Lib\site-packages\platformio\builder\tools\piolib.py", line 1062:
    project.install_dependencies()
  File "C:\Users\Matt\.platformio\penv\Lib\site-packages\platformio\builder\tools\piolib.py", line 898:
    lm.install(spec)
  File "c:\users\matt\.platformio\penv\lib\site-packages\platformio\package\manager\_install.py", line 48:
    pkg = self._install(
  File "c:\users\matt\.platformio\penv\lib\site-packages\platformio\package\manager\library.py", line 81:
    return super(LibraryPackageManager, self)._install(
  File "c:\users\matt\.platformio\penv\lib\site-packages\platformio\package\manager\_install.py", line 97:
    pkg = self.install_from_url(spec.url, spec, silent=silent)
  File "c:\users\matt\.platformio\penv\lib\site-packages\platformio\package\manager\_install.py", line 142:
    assert vcs.export()
  File "c:\users\matt\.platformio\penv\lib\site-packages\platformio\package\vcsclient.py", line 176:
    assert self.run_cmd(args)
  File "c:\users\matt\.platformio\penv\lib\site-packages\platformio\package\vcsclient.py", line 115:
    raise VCSBaseException("VCS: Could not process command %s" % e.cmd)
============================================== [FAILED] Took 1.30 seconds ==============================================

[vortigont]

Looks like that cywin and windows path-names mix that causes the issue

/cygdrive/c/Users/Matt/.platformio/.cache/tmp/pkg-installing-35j8vz3p/C:\Users\Matt.platformio.

Have you tried to uninstall cygwin git and use git for windows instead?

[ivankravets]

1. We don't support cygwin
2. Could you try the latest pio upgrade --dev?

[zenflow]

Thank you both for your helpful responses.. really great!

[ivankravets]

Happy coding with PlatformIO!

[zenflow]

Looks like that cywin and windows path-names mix that causes the issue

/cygdrive/c/Users/Matt/.platformio/.cache/tmp/pkg-installing-35j8vz3p/C:\Users\Matt.platformio.

@vortigont Yeah I noticed that "fatal: Invalid path" but was a little confused since the command on the next line looks fine (using a Windows path however). I thought that error was coming from pio, but now it's obvious that the error is generated by Cygwin's git command which can't handle the normal Windows path it was given.

Have you tried to uninstall cygwin git and use git for windows instead?

I was reluctant to uninstall Cygwin since some projects rely on it (they may or may not work in WSL, I haven't checked), and also there's no automatic uninstaller. I'm sure that would work though.
Instead I just worked around the problem with Git by replacing the git url (https://github.com/earlephilhower/ESP8266Audio#22b52e0ed5aa86a5e5704c5c86d435c8e3e233a0) with a tarball url (https://codeload.github.com/earlephilhower/ESP8266Audio/legacy.tar.gz/22b52e0ed5aa86a5e5704c5c86d435c8e3e233a0).
I think this is the more portable way to include a GitHub project in lib_deps anyway, since it doesn't depend on an external program (i.e. git).