Conda-build: conda index 3.18.* crashes on corrupt package with only an obscure error message

[y2kbugger]

Actual Behavior

hash & extract packages for win-64:   0%|          | 0/1 [00:00<?, ?it/s]Exception in thread Thread-2:
Traceback (most recent call last):
  File "C:\ProgramData\CooperConda\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\ProgramData\CooperConda\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\ProgramData\CooperConda\lib\concurrent\futures\process.py", line 272, in _queue_management_worker
    result_item = reader.recv()
  File "C:\ProgramData\CooperConda\lib\multiprocessing\connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
TypeError: __init__() missing 1 required positional argument: 'msg'

Expected Behavior

Not crash, and give error message similar to the 3.17.* series. Downgrading is how I finally determined that their was a problem package.

Package win-64/bokeh-1.3.4-py37_0.tar.bz2 appears to be corrupt.  Please remove it and re-download it
ERROR:conda_build.index:Package` win-64/bokeh-1.3.4-py37_0.tar.bz2 appears to be corrupt.
Please remove it and re-download it

Steps to Reproduce

Ex.zip

1. Extract the zip file.
2. $ cd Ex
   $ conda index .

Output of conda info

$ conda info

     active environment : base
    active env location : C:\ProgramData\CooperConda
            shell level : 1
       user config file : C:\Users\zdkohler\.condarc
 populated config files : C:\ProgramData\CooperConda\.condarc
          conda version : 4.7.11
    conda-build version : 3.18.9
         python version : 3.6.9.final.0
       virtual packages : __cuda=9.1
       base environment : C:\ProgramData\CooperConda  (writable)
           channel URLs : http://technicalwebdev/CooperConda/Internal/win-64
                          http://technicalwebdev/CooperConda/Internal/noarch
                          http://technicalwebdev/CooperConda/External/win-64
                          http://technicalwebdev/CooperConda/External/noarch
          package cache : C:\ProgramData\CooperConda\pkgs
                          C:\Users\zdkohler\.conda\pkgs
                          C:\Users\zdkohler\AppData\Local\conda\conda\pkgs
       envs directories : C:\ProgramData\CooperConda\envs
                          C:\Users\zdkohler\.conda\envs
                          C:\Users\zdkohler\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.7.11 requests/2.22.0 CPython/3.6.9 Windows/10 Windows/10.0.17134
          administrator : True
             netrc file : None
           offline mode : False

[prusse-martin]

I got the same problem.
After some hacking (add a debug=True in the api update_index) it displayed:

$ conda index esss --no-progress --verbose -t 1

...

DEBUG:conda_build.index:hashing, extracting, and caching tbb-2018_20170726-vc10h72acbbc_1.tar.bz2
Traceback (most recent call last):
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_package_handling/api.py", line 49, in extract
    SUPPORTED_EXTENSIONS[ext].extract(fn, dest_dir, components=components)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_package_handling/tarball.py", line 88, in extract
    _tar_xf(fn, dest_dir)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_package_handling/tarball.py", line 77, in _tar_xf
    libarchive.extract_file(tarball, flags)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/libarchive/extract.py", line 70, in extract_file
    extract_entries(archive, flags)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/libarchive/extract.py", line 49, in extract_entries
    for entry in entries:
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/libarchive/read.py", line 26, in __iter__
    r = read_next_header2(archive_p, entry_p)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/libarchive/ffi.py", line 112, in check_int
    raise archive_error(args[0], retcode)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/libarchive/ffi.py", line 96, in archive_error
    raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
libarchive.exception.ArchiveError: Damaged tar archive (errno=22, retcode=-10, archive_p=94044958495392)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jenkins/miniconda3/bin/conda-index", line 11, in <module>
    sys.exit(main())
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/cli/main_index.py", line 91, in main
    return execute(sys.argv[1:])
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/cli/main_index.py", line 87, in execute
    current_index_versions=args.current_index_versions_file)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/api.py", line 401, in update_index
    subdirs=ensure_list(subdir), current_index_versions=current_index_versions)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/index.py", line 295, in update_index
    current_index_versions=current_index_versions)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/index.py", line 805, in index
    subdir, verbose=verbose, progress=progress)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/index.py", line 944, in index_subdir
    fn, mtime, size, index_json = future.result()
  File "/home/jenkins/miniconda3/lib/python3.7/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/home/jenkins/miniconda3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/index.py", line 82, in submit
    result = fn(*args, **kwargs)
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_build/index.py", line 1052, in _extract_to_cache
    conda_package_handling.api.extract(abs_fn, dest_dir=tmpdir, components="info")
  File "/home/jenkins/miniconda3/lib/python3.7/site-packages/conda_package_handling/api.py", line 51, in extract
    raise InvalidArchiveError(fn, str(e))
conda_package_handling.exceptions.InvalidArchiveError: Error with archive /home/jenkins/conda_index_test/conda-channel/esss/win-64/vtk-6.2.0-py27_vc10_debug_esss_dist_3.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:

Damaged tar archive (errno=22, retcode=-10, archive_p=94044958495392)

Whit this change I proceeded to move the reported files to some "trash" folder (one by one) and running again index (without the hack) and it succeeded.

It looks like concurrent.futures is shadowing the error or at least the current code failing to propagate it.

cc @nicoddemus

[marcelotrevisani]

Thanks, @y2kbugger and @prusse-martin for reporting it!
I might take a look at it!

[prusse-martin]

Note that also the debug argument is not originally a command line option and so not forwarded to the ChannelIndex class when it is constructed since when calling conda_build.update_index this is not used.

[msarahan]

@marcelotrevisani this is on my plate for this sprint (started Monday, ends next Friday), so go ahead and investigate if you have time, but I will fix it soon otherwise.

[marcelotrevisani]

That's fine! No problem at all for me!

If it is already in your plans you can take it then! :)