Conda-build: Windows builds failing if archive contains symlink

[effigies]

Actual Behavior

In Windows-based builds on AppVeyor, the extraction step crashes if the archive contains a symlink.

For example:

Extracting download
Traceback (most recent call last):
  File "C:\Miniconda36-x64\Scripts\conda-build-script.py", line 10, in <module>
    sys.exit(main())
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\cli\main_build.py", line 456, in main
    execute(sys.argv[1:])
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\cli\main_build.py", line 447, in execute
    verify=args.verify, variants=args.variants)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\api.py", line 208, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\build.py", line 2311, in build_tree
    notest=notest,
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\build.py", line 1407, in build
    try_download(m, no_download_source=False, raise_error=True)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\render.py", line 617, in try_download
    source.provide(metadata)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\source.py", line 653, in provide
    timeout=metadata.config.timeout, locking=metadata.config.locking)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\source.py", line 148, in unpack
    tar_xf(src_path, tmpdir)
  File "C:\Miniconda36-x64\lib\site-packages\conda_build\utils.py", line 718, in tar_xf
    libarchive.extract_file(tarball, flags)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\extract.py", line 70, in extract_file
    extract_entries(archive, flags)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\extract.py", line 50, in extract_entries
    write_header(write_p, entry._entry_p)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\ffi.py", line 91, in check_int
    raise archive_error(args[0], retcode)
  File "C:\Miniconda36-x64\lib\site-packages\libarchive\ffi.py", line 75, in archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Can't create '\\?\C:\bld\tmp5lgmn3gr\nibabel-2.3.2\doc\downloads' (errno=22, retcode=-25, archive_p=429904390704)

Note that nibabel/doc/downloads is a symlink to nibabel/doc/source/downloads.

Expected Behavior

I would expect any of the following:

1) Symlinks replaced with empty files
2) Symlinks replaced with copies of the linked files/directories (probably cleaned to only permit targets within the archive)

Steps to Reproduce

I expect, on Windows:

git clone https://github.com/conda-forge/nibabel-feedstock.git
cd nibabel-feedstock
conda build recipe

Output of conda info

     active environment : base
    active env location : C:\Miniconda36-x64
            shell level : 1
       user config file : C:\Users\appveyor\.condarc
 populated config files : C:\Users\appveyor\.condarc
          conda version : 4.6.0
    conda-build version : 3.17.5
         python version : 3.6.5.final.0
       base environment : C:\Miniconda36-x64  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Miniconda36-x64\pkgs
                          C:\Users\appveyor\.conda\pkgs
                          C:\Users\appveyor\AppData\Local\conda\conda\pkgs
       envs directories : C:\Miniconda36-x64\envs
                          C:\Users\appveyor\.conda\envs
                          C:\Users\appveyor\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.6.0 requests/2.18.4 CPython/3.6.5 Windows/2012ServerR2 Windows/6.3.9600
          administrator : True
             netrc file : None
           offline mode : False

Ref: conda-forge/conda-smithy#1002, #3163.

Known affected builds: conda-forge/nibabel-feedstock#14, conda-forge/openpmd-api-feedstock#20

[mingwandroid]

Hey.

We enable 'developer mode' in our internal CI which is the correct thing to do.

Our team will not have time to look into this unfortunately. The options are:

1. Ask AppVeyor if they can enable developer mode.
2. Move to Azure if it allows / enables developer mode.
3. Implement a fallback in conda-build so that should unpacking fail with a symlink related permissions error, the python tarfile module gets used instead.
4. Implement a fallback in libarchive so that should unpacking fail with a symlink related permissions error, the C code therein falls back to copying instead. This is trickier than it would initially appear because at the point when a symlink is created it is not necessarily the case that the target of the symlink exists due to the ordering of the tarfile, so you may not have any file to copy. This gets worse with symlinks to symlinks to files so must be handled carefully as a post-process. This would be a continuation of my PR at libarchive/libarchive#1030

Cheers!

[mingwandroid]

Overall, I believe if 1 & 2 are not possible, 3 would be pretty easy to implement.

[ax3l]

Thanks for reporting! Since the latest updates a week ago, this breaks my windows builds.

Is it possible to work-around this or is someone able to implement suggestion 3? :)

[effigies]

We worked around by making our Python 2-only dependency a regular dependency and going back to noarch, so it could build on Linux and install on Windows. It's... not ideal.

[ax3l]

I currently work-around by downgrading, which I luckily saw today from others linking here.

Still, won't be a solution for too long and has to be refreshed every rerender.

[ax3l]

Fix shipped in conda-build 3.17.8+
@dbast @chrisburr @dtip @effigies you can remove your work-arounds now :)

[xylar]

I'm still seeing what I think is this same issue on appveryor builds from conda-forge with conda-build 3.17.8. See https://ci.appveyor.com/project/conda-forge/staged-recipes/builds/22586040

[effigies]

It seems to be a different symlink-related bug... I think we ran into it as well. Our fix remains to include our lightweight python 2 dependency as a full dependency and build as noarch.

[xylar]

@effigies, okay, thanks. I noticed the same error message on one of your branches but I agree it's different from the one above.

Yes, I'm also working around this by building noarch on linux. I just figured I'd mention that the problems with symlinks seem to be ongoing...

[lhofmann]

I'm still getting the same error here with conda-build 3.18.11: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=91309
My current workaround is to downgrade to conda-build=3.15.

@effigies Could you please reopen this issue?

[effigies]

@lhofmann I don't have permissions to reopen issues in this repository. We're still using the workaround, FWIW.

[effigies]

Looks like it's now working for me for Python 3+ (see https://github.com/conda-forge/nibabel-feedstock/pull/18), so not really sure what's going on on your end, and the Python 2 issue seems to be pip-related.