Platformio-core: Unable to upload or repackage the upstream package

[mcspr]

Configuration

Operating system:

Windows 10 20H2

PlatformIO Version (platformio --version):

PlatformIO Core, version 5.0.4b1`

Description of problem

As previously mentioned in the https://github.com/platformio/platformio-core/issues/3612

ESP8266 Core publishes current development version of the toolchain here:
https://github.com/earlephilhower/esp-quick-toolchain

I was trying to upload it to the registry, however this shows up:

> C:\Users\maxim\.platformio\penv\Scripts\pio.exe package publish .\x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz
Error: Traceback (most recent call last):
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\package.py", line 87, in package_publish
    archive_path = p.pack()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\pack.py", line 120, in pack
    with FileUnpacker(src) as fu:
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 152, in __enter__
    self._archiver = self._init_archiver()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 144, in _init_archiver
    with open(self.path, "rb") as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz'

============================================================

As mentioned in the previous issue, I did the pio package pack after this failed, which created another tarball in the same directory:

> C:\Users\maxim\.platformio\penv\Scripts\pio.exe package pack .\x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz
Wrote a tarball to "C:\Users\maxim\xtensa\toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz"

However, the contents are no longer the same:

/m/c/U/m/xtensa> tar vtf x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz | grep g++
hrwxr-xr-x 1000/1000         0 2020-12-23 08:10 xtensa-lx106-elf/bin/xtensa-lx106-elf-g++ link to xtensa-lx106-elf/bin/xtensa-lx106-elf-c++
-rw-r--r-- 1000/1000   1331843 2020-12-23 08:10 xtensa-lx106-elf/share/man/man1/xtensa-lx106-elf-g++.1
/m/c/U/m/xtensa> tar vtf toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz | grep g++
hrw-rw-rw- 0/0               0 2020-12-23 08:10 bin/xtensa-lx106-elf-g++ link to bin/xtensa-lx106-elf-c++
-rw-rw-rw- 0/0         1331843 2020-12-23 08:10 share/man/man1/xtensa-lx106-elf-g++.1

Files are defaulted to rw permissions and UID/GID 0, executable bit lost as well.
edit: although, UID & GID 0 may still be ok i.e. those should not be used anyway without 'numeric owner' setting (?)

So, the issues are:

• permissions are lost when unpacking and then packing back when using Windows-based PIO installation
  previously I was able to upload and use the version 5.100200.200918, but that was done using Linux installation
• it is impossible to upload the original tarball, despite it being compatible with standalone installation method
• mcspr/toolchain-xtensa linux build is broken, but I'd assume it will be fixed with re-upload using a different OS (...publishing needs unpublish first, which is not checked before the upload, but only on moderation step :/)

cc @jason2866 via https://github.com/esp8266/Arduino/issues/7792#issuecomment-751125057

[mcspr]

Having just tried via WSL, where

• original x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz
• pio-package-pack'ed toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz

~/xtensa> pio --version
PlatformIO Core, version 5.0.4b1

~/xtensa> pio package publish toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz
Error: Traceback (most recent call last):
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/commands/__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/commands/package.py", line 89, in package_publish
    archive_path = p.pack()
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/package/pack.py", line 120, in pack
    with FileUnpacker(src) as fu:
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/package/unpack.py", line 152, in __enter__
    self._archiver = self._init_archiver()
  File "/home/maxim/.platformio/penv/lib64/python3.9/site-packages/platformio/package/unpack.py", line 144, in _init_archiver
    with open(self.path, "rb") as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz'

~/xtensa> pio package publish (readlink -f toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz)
The package has been accepted for moderation. We will notify you on email when process it.

 ~/xtensa> pio package publish (readlink -f x86_64-linux-gnu.xtensa-lx106-elf-dba3cc8.201223.tar.gz)
The package has been accepted for moderation. We will notify you on email when process it.

Surprisingly, using the full path to the archive made this work

[ivankravets]

publish .\x86_64

What is it /\? Just use a file name if you are in the current directory.

[mcspr]

Windows slashes are backwards, autocomplete puts it when TAB'ing the file in the current directory. It is irrelevant, you can still get the exception without it (c/p was from the 2nd attempt, without the dot-slash)

[Jason2866]

@ivankravets okay about the /\ but this is NOT the main issue here.
If you use Windows to do a package from a Linux tar ball the file permissions get lost.
See the linked issue to Arduino ESP8266 -> New Gcc toolchain packages for the actual Arduino ESP8266 stage

[mcspr]

I've uploaded correct tarballs already, trying to upload the one from the bintray:

PS C:\Users\maxim\xtensa> dir

    Directory: C:\Users\maxim\xtensa

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          26.12.2020    12:06       74117637 d10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz

PS C:\Users\maxim\xtensa> C:\Users\maxim\.platformio\penv\Scripts\pio.exe package publish d10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz
Error: Traceback (most recent call last):
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\__main__.py", line 109, in main
    cli()  # pylint: disable=no-value-for-parameter
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\__init__.py", line 44, in invoke
    return super(PlatformioCLI, self).invoke(ctx)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\commands\package.py", line 87, in package_publish
    archive_path = p.pack()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\pack.py", line 120, in pack
    with FileUnpacker(src) as fu:
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 152, in __enter__
    self._archiver = self._init_archiver()
  File "c:\users\maxim\.platformio\penv\lib\site-packages\platformio\package\unpack.py", line 144, in _init_archiver
    with open(self.path, "rb") as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'd10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz'

[ivankravets]

Could you help to investigate this issue? The code is here

• https://github.com/platformio/platformio-core/blob/develop/platformio/package/unpack.py
• https://github.com/platformio/platformio-core/blob/develop/platformio/package/pack.py

Thanks!

[mcspr]

A quick summary:

• https://github.com/platformio/platformio-core/commit/adf9ba29df7ccc1da1ec6c434e6a187af239b937 introduced fs.cd(tmp_dir) while running pio package publish command, it only works with full paths (unless copy happens, somehow, but I don't see it anywhere though)
• tests do not cover PackagePacker case for archives, only directories
• for this issue specifically, packer can preserve [filepath, TarInfo] from the original archive and utilize TarFile::add(..., filter=map_tar_info) to replace TarInfo with the orginal one

[ivankravets]

@Jason2866 thanks, we have reproduced this issue! Downgrading issue to bug.

[ivankravets]

Thanks! Fixed, you can publish tar.gz directly from Windows.

[mcspr]

@ivankravets Thanks!

Should the pio package pack have a warning or just fail on archives?
btw I noticed that the owner inside of d10244e-toolchain-xtensa-linux_x86_64-5.100200.201223.tar.gz is pioapp/pioapp, does the remote API repackage yet again? (or I again misunderstood the package flow, sry but I have not really followed the code too deep)

[ivankravets]

Should the pio package pack have a warning or just fail on archives?

What do you mean?

is pioapp/pioapp, does the remote API repackage yet again?

I downloaded the package mentioned above and the permissions are good. Yes, we repack EVERY package on the server side and remove any trash from it (pictures, movies, etc). People can control package contents using "package.json" and "export" field as described in docs.

[mcspr]

What do you mean?

The other issue after my tests - I was able to do pio package pack $archive on the upstream .tar.gz effectively breaking it. I meant, should it notify the user (or just straight do nothing) about the possible breakage of permissions when running FS without an ability to store unix metadata stuff?

[ivankravets]

Yes, we had the same thoughts to show warning message or raise an error. It's not our issue and we can't fix it. Windows does not support symlinks, and other UNIX FS features.

We will raise an exception and ask user to extra archive manually before running "pack".

[ivankravets]

Thanks, resolved!