Conda: CondaOSError: OS error: failed to link

[jbednar]

I'm having trouble installing packages using conda 4.2.7 that work fine with conda 4.1.11:

0172-jbednar:~> cat envt.yml 
name: test
channels:
  - ioam
dependencies:
  - param

0172-jbednar:~> conda env create -f envt.yml
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata .........
Solving package specifications: ..........
Linking packages ...
CondaOSError: OS error: failed to link (src=u'/Users/jbednar/anaconda/pkgs/param-1.4.1-py35_0/lib/python3.5/site-packages/numbergen/__pycache__/__init__.cpython-35.pyc', dst='/Users/jbednar/anaconda/envs/test/lib/python3.5/site-packages/numbergen/__pycache__/__init__.cpython-35.pyc', type=1, error=OSError(2, 'No such file or directory'))


0172-jbednar:~> conda install conda=4.1.11
Fetching package metadata .......
Solving package specifications: ..........

Package plan for installation in environment /Users/jbednar/anaconda:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-env-2.5.2            |           py27_0          26 KB

The following packages will be DOWNGRADED due to dependency conflicts:

    conda:     4.2.7-py27_0 --> 4.1.11-py27_0
    conda-env: 2.6.0-0      --> 2.5.2-py27_0 

Proceed ([y]/n)? y

Pruning fetched packages from the cache ...
Fetching packages ...
conda-env-2.5. 100% || Time: 0:00:00 578.53 kB/s
Extracting packages ...
[      COMPLETE      ]|| 100%
Unlinking packages ...
[      COMPLETE      ]|| 100%
Linking packages ...
[      COMPLETE      ]|| 100%

I'm using OS X 10.11.6.

[kalefranz]

Thanks!

[jbednar]

Not sure if it's the same as this issue from Intel: https://software.intel.com/en-us/forums/intel-distribution-for-python/topic/685331

[ivan-kalev]

Related: https://github.com/bokeh/bokeh/issues/5196

Basically some packages in the defaults channel are in an inconsistent state: one or more files listed in the package metadata (info/files) are missing. Conda attempts to link them anyway and fails. I guess the real issue here is that reporting linking problems at installation time is simply too late. One solution would be change conda-build to validate the contents of each package as part of the build process. Or conda itself could try to validate the package it is about to install - but before making any changes to site-packages. Ideally, we could have both checks in place.

[jbednar]

It does seem to me that such validation should be done at build time, not installation time, as the person installing it has no control over the package contents, and the problems reported clearly aren't always errors.

It's not clear to me why info/files would have listed .pyc files; we presumably would not have included those explicitly when building the package!

[ivan-kalev]

Validation at build time seems like something conda-build must absolutely do. However, some sort of package validation at installation time seems necessary to me, for the simple fact that a package manager should refuse to install unsupported package formats or corrupt packages. A package may be corrupt for many different reasons - a bug in the build system, bad post-build manipulation of the build artifact, or a dodgy download. Unfortunately the only objective way to decide if a missing file is an actual error is to ask the package maintainers. I would therefore suggest that conda should take a conservative approach and refuse to install packages with inconsistent metadata.

[kalefranz]

@soapy1 Let's making sure the files we're about to link actually exist in the extracted package as part of the CHECK_LINK step you're working on...

[kalefranz]

I agree that conda-build shouldn't be building broken packages. But we'll add checks into conda itself also as part of https://github.com/conda/conda/pull/3301.

[jbednar]

Does anyone have a theory for how .pyc files could end up in the manifest of files promised by the package? It seems important to address that issue before getting too strict on comparing the files promised and delivered.

[kalefranz]

Dug up some more history. The issue fundamentally relates to https://github.com/conda/conda/pull/2639.

Also, https://github.com/conda/conda/issues/3519 is a duplicate of https://github.com/ContinuumIO/anaconda-issues/issues/1103

[arg0]

during my test, conda version: 4.2.11 works fine, maybe update to 4.2.11 fix it.
conda: 4.2.9-py35_0 --> 4.2.11-py35_0

[kalefranz]

Leaving this open until our "Check Link" PR #3571 is merged for 4.3

[BKJackson]

FYI- I'm having a similar "failed to link" error with qt:

CondaOSError: OS error: failed to link (src='/Applications/anaconda/pkgs/qt-4.8.7-1/bin/Linguist.app/Contents/Info.plist', dst='/Applications/anaconda/bin/Linguist.app/Contents/Info.plist', type=1, error=FileNotFoundError(2, 'No such file or directory'))

[fperez]

Same here, but six is the one crashing:

Linking packages ...
[six                 ]|                                                   |   0%

CondaOSError: OS error: failed to link 
(src='/Users/fperez/usr/conda/pkgs/six-1.10.0-py35_0/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO', 
dst='/Users/fperez/usr/conda/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO', 
type=1, error=NotADirectoryError(20, 'Not a directory'))

Eventually the only thing that worked for me was to completely nuke my conda install and start from scratch.

[BKJackson]

Oh, wow. Pretty much the whole anaconda package doesn't want to update. Here is the command and error message.

conda update --all
Fetching package metadata .......
Solving package specifications: ..........

Package plan for installation in environment /Applications/anaconda:

The following NEW packages will be INSTALLED:

anaconda-client: 1.5.3-py35_0
ipython:         5.1.0-py35_1
pyqt:            4.11.4-py35_4
qt:              4.8.7-1
qtconsole:       4.2.1-py35_0
scipy:           0.17.1-np110py35_1
smart_open:      1.3.5-py35_0

Proceed ([y]/n)? y

Linking packages ...
[qt                  ]|                                                                                |   0%

CondaOSError: OS error: failed to link (src='/Applications/anaconda/pkgs/qt-4.8.7-1/bin/Linguist.app/Contents/Info.plist', dst='/Applications/anaconda/bin/Linguist.app/Contents/Info.plist', type=1, error=FileNotFoundError(2, 'No such file or directory'))

[orchestor]

same here.

[ParsaAkbari]

I get this error when running the following:
conda create -n py36 python=3.5

CondaOSError: OS error: failed to link (src='..../miniconda3/pkgs/openssl-1.0.2j-0/bin/c_rehash', dst='..../miniconda3/envs/py36/bin/c_rehash', type=3, error=OSError(5, 'Input/output error'))

[ddofer]

I'm getting the same error.

`Linking packages ...

CondaOSError: OS error: failed to link (src=u'/Users/d/anaconda/pkgs/qt-4.8.7-4/mkspecs/default', dst='/Users/d/anaconda/mkspecs/default', type=3, error=OSError(17, 'File exists'))
`

[ozak]

I'm, getting the same error:

CondaOSError: OS error: failed to link (src='/Users/omer/Anaconda3/pkgs/geos-3.5.1-1/lib/libgeos.dylib', dst='/Users/omer/Anaconda3/envs/GeoPython/lib/libgeos.dylib', type=3, error=FileExistsError(17, 'File exists'))

After trying to update other packages I get the same error...it seems that it is not unlinking/deleting old dylibs(?). Here's another error

CondaOSError: OS error: failed to link (src='/Users/omer/Anaconda3/pkgs/gsl-2.1-2/lib/libgsl.dylib', dst='/Users/omer/Anaconda3/envs/GeoPython/lib/libgsl.dylib', type=3, error=FileExistsError(17, 'File exists'))

[wilmerhenao]

I am getting the same error. Not only that, but python stops working completely. I have reinstalled anaconda3 several times and still, doesn't work.

CondaOSError: OS error: failed to link (src='/home/wilmer/local/anaconda3/pkgs/tcl-8.6.4-intel_15/lib/libtcl.so', dst='/home/wilmer/local/anaconda3/lib/libtcl.so', type=1, error=FileExistsError(17, 'File exists'))

$ conda info
-bash: /home/wilmer/local/anaconda3/bin/conda: No such file or directory

[nukolas]

I got the same error when trying to downgrade PyQt5 to PyQt4:

CondaOSError: OS error: failed to link (src=u'/home/nick/anaconda2/pkgs/qt-4.8.7-4/mkspecs/default', dst='/home/nick/anaconda2/mkspecs/default', type=3, error=OSError(17, 'File exists'))

I fixed the error by simply deleting the destination file in the error message:

cd /home/nick/anaconda2/mkspecs/
sudo rm default

After that, conda updated just fine, using either conda update --all, or specifically for my case conda install -c anaconda pyqt=4.11.4

This is a bit of a hack solution, and I really don't understand what is going on with the file linking procedure (but possibly an underlying permissions issue?), but it works for me.

[wilmerhenao]

That's interesting, I fixed it too but I forgot to post.
I think my problem was that I accidentally installed intel Distribution for Python, and on top of that I had two parallel anaconda3 installations. It seems like they all were added to the path automatically in the ~/.bashrc file.

After deleting all the redundant entries I got it to work.

I hope that helps.

[nukolas]

I just checked my $PATH, but nothing untoward as you described for yours Wilmer. I do think there was an anaconda update installed recently though..

[wilmerhenao]

Hmm. You know, I just double checked and it turns out that my path is now pointing to the Intel distribution for python instead of anaconda.
I guess the bug still exists.

[kalefranz]

We're working hard and doing our best, I promise.

On Nov 29, 2016, at 3:01 AM, Szabolcs Horvát [e-mail] wrote:

Yes, the bug exists. It has been there for three months. Lots of people are affected even though most of us didn't comment here because we just couldn't believe that Continuum would let this lapse for this long. It's a shame.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[szhorvat]

@kalefranz Thanks! The reason why I deleted my comment is that I was in fact wrong. nukolas's solution (which should have been something pretty obvious to try anyway...) did work for me.

[dhirschfeld]

I just ran into a similar bug with a pretty new Miniconda install on Windows Server 2016. In my case I get a OS error: win32 hard link failed error:

(gateway) C:\Users\dhirschf>conda update --all
Fetching package metadata ...........
Solving package specifications: ..........

Package plan for installation in environment C:\Miniconda\envs\gateway:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    decorator-4.0.10           |           py35_1          14 KB
    jupyter_core-4.2.1         |           py35_0         106 KB
    ------------------------------------------------------------
                                           Total:         119 KB

The following packages will be UPDATED:

    decorator:    4.0.10-py35_0 --> 4.0.10-py35_1
    jupyter_core: 4.2.0-py35_0  --> 4.2.1-py35_0

Proceed ([y]/n)? y

Pruning fetched packages from the cache ...
Fetching packages ...
decorator-4.0. 100% |###############################| Time: 0:00:00   6.96 MB/s
jupyter_core-4 100% |###############################| Time: 0:00:00   8.34 MB/s
Extracting packages ...
[      COMPLETE      ]|##################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|##################################################| 100%
Linking packages ...
[decorator           ]|                                                  |   0%

CondaOSError: OS error: failed to link (src='C:\\Miniconda\\pkgs\\decorator-4.0.10-py35_1\\Lib/site-packages/__pycache__/decorator.cpython-35.pyc', dst='C:\\Miniconda\\envs\\gateway\\Lib/site-packages/__pycache__/decorator.cpython-35.pyc', type=1, error=CondaOSError: OS error: win32 hard link failed
)

[dhirschfeld]

It might be a permissions issue - I got it to work by running as Administrator...

[uduwage]

I am on el capitan (10.11.16) and just did $conda update --allended up with the same error:

CondaOSError: OS error: failed to link (src=u'/Users/a/anaconda2/pkgs/futures-3.0.5-py27_0/lib/python2.7/site-packages/concurrent/__init__.py', dst='/Users/a/anaconda2/lib/python2.7/site-packages/concurrent/__init__.py', type=1, error=OSError(17, 'File exists'))

in addition to that after upgrade I am unable to run any conda commands and get fallowing error:

$ conda list
Traceback (most recent call last):
  File "/Users/a/anaconda2/bin/conda", line 6, in <module>
    sys.exit(conda.cli.main())
  File "/Users/a/anaconda2/lib/python2.7/site-packages/conda/cli/main.py", line 152, in main
    return conda_exception_handler(_main)
  File "/Users/a/anaconda2/lib/python2.7/site-packages/conda/exceptions.py", line 497, in conda_exception_handler
    print_unexpected_error_message(e)
  File "/Users/a/anaconda2/lib/python2.7/site-packages/conda/exceptions.py", line 426, in print_unexpected_error_message
    from conda.base.context import context
  File "/Users/a/anaconda2/lib/python2.7/site-packages/conda/base/context.py", line 20, in <module>
    from ..common.disk import try_write, conda_bld_ensure_dir
  File "/Users/a/anaconda2/lib/python2.7/site-packages/conda/common/disk.py", line 16, in <module>
    from ..utils import on_win
  File "/Users/a/anaconda2/lib/python2.7/site-packages/conda/utils.py", line 11, in <module>
    from .common.url import path_to_url
  File "/Users/a/anaconda2/lib/python2.7/site-packages/conda/common/url.py", line 21, in <module>
    from requests.packages.urllib3.exceptions import LocationParseError
ImportError: No module named requests.packages.urllib3.exceptions

[uduwage]

I have opened a new issue since it breaks entire conda environment:
https://github.com/conda/conda/issues/4020

[fperez]

Any progress on this? It's a completely paralyzing bug... I'm trying to create a new conda env and it dies b/c of this (in my case, on six).

[fperez]

To add some info, the error message is:

CondaOSError: OS error: failed to link 
(src='/Users/fperez/usr/conda/pkgs/six-1.10.0-py35_0/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO', 
dst='/Users/fperez/usr/conda/envs/jlab/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO', 
type=1, error=NotADirectoryError(20, 'Not a directory'))

and if I look in the source directory, indeed that egg-info path is a file, not a directory:

Kerbin[site-packages]> d
/Users/fperez/usr/conda/pkgs/six-1.10.0-py35_0/lib/python3.5/site-packages
total 72
drwxr-xr-x  3 fperez    102 Nov  2 09:15 __pycache__/
-rw-r--r--  1 fperez   1422 Oct  8  2015 six-1.10.0-py3.5.egg-info
-rw-r--r--  1 fperez  30098 Oct  8  2015 six.py

I don't know who, but someone in the chain thinks there's a directory there that can be populated with files, whereas in reality it's a file...

I've tried multiple workarounds, removing six as a base package, etc, and nothing works so far... Any ideas?

This basically makes conda completely unusable once the bug hits.

[ryanriopelle]

It happened to me as well in the Conda for Python 2.7 version. I had to completely delete all evidence of Conda ever being installed on my computer. Including any links in Bashrc, backup files, ect. Then I re-installed Conda for Python 3, and created new environments for Python 2.7. PyCharm was unable to detect the changes until I re-installed it as well.

[kalefranz]

We released conda 4.3.0 to the conda-canary channel today. Here's the changelog
https://github.com/conda/conda/releases/tag/4.3.0

It's got some rough edges in terms of user-messaging, but the core logic is all there. I'm very interested in feedback on how it performs on this issue.

[fperez]

Well, same error, just that now it's a bare traceback instead of a conda summary error msg:

denali[~]> conda --version
conda 4.3.0

denali[~]> conda create -n jlab scikit-image
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /Users/fperez/usr/conda/envs/jlab:

The following NEW packages will be INSTALLED:

    cycler:          0.10.0-py35_0

[...]

    six:             1.10.0-py35_0
    sqlite:          3.13.0-0
    tk:              8.5.18-0
    wheel:           0.29.0-py35_0
    xz:              5.2.2-0
    zlib:            1.2.8-3

Proceed ([y]/n)? y

ERROR conda.core.link:_execute_actions(223): Something bad happened, but it's okay because I'm going to roll back now.
An unexpected error has occurred.
Please consider posting the following information to the
conda GitHub issue tracker at:

    https://github.com/conda/conda/issues



Current conda install:

               platform : osx-64
          conda version : 4.3.0
       conda is private : False
      conda-env version : 4.3.0
    conda-build version : not installed
         python version : 3.5.2.final.0
       requests version : 2.12.4
       root environment : /Users/fperez/usr/conda  (writable)
    default environment : /Users/fperez/usr/conda
       envs directories : /Users/fperez/usr/conda/envs
          package cache : /Users/fperez/usr/conda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/osx-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/osx-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/osx-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /Users/fperez/.condarc
           offline mode : False
             user-agent : conda/4.3.0 requests/2.12.4 CPython/3.5.2 Darwin/15.6.0 OSX/10.11.6
                UID:GID : 33212:20665



`$ /Users/fperez/usr/conda/bin/conda create -n jlab scikit-image`




    Traceback (most recent call last):
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/exceptions.py", line 516, in conda_exception_handler
        return_value = func(*args, **kwargs)
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/cli/main.py", line 145, in _main
        exit_code = args.func(args, p)
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/cli/main_create.py", line 68, in execute
        install(args, parser, 'create')
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/cli/install.py", line 389, in install
        execute_actions(actions, index, verbose=not context.quiet)
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/plan.py", line 903, in execute_actions
        execute_instructions(plan, index, verbose)
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/instructions.py", line 258, in execute_instructions
        cmd(state, arg)
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/instructions.py", line 119, in UNLINKLINKTRANSACTION_CMD
        txn.execute()
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/core/link.py", line 179, in execute
        pkg_data, actions)
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/core/link.py", line 216, in _execute_actions
        action.execute()
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/core/path_actions.py", line 224, in execute
        force=context.force)
      File "/Users/fperez/usr/conda/lib/python3.5/site-packages/conda/gateways/disk/create.py", line 226, in create_link
        os.link(src, dst)
    NotADirectoryError: [Errno 20] Not a directory: '/Users/fperez/usr/conda/pkgs/six-1.10.0-py35_0/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO' -> '/Users/fperez/usr/conda/envs/jlab/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO

So, still 100% broken, unfortunately.

I'm going to nuke my entire anaconda install and start from scratch again, will report back if that helps.

[fperez]

ps, this is my condarc:

denali[~]> cat ~/.condarc
channels:
  - defaults

[fperez]

Just to report, a full nuke-and-reinstall helped. We'll see for how long (since I'd already seen the problem before).

[kalefranz]

In this case I'm glad you got a full stack trace. I'm going to try to jump on this tomorrow now that I have the tools built in to help.

On Dec 14, 2016, at 7:12 PM, Fernando Perez [e-mail] wrote:

ps, this is my condarc:

denali[~]> cat ~/.condarc
channels:

• defaults
  —
  You are receiving this because you were mentioned.
  Reply to this email directly, view it on GitHub, or mute the thread.

[fperez]

Great, thanks! Can't offer to keep testing since I nuked and reinstalled, but if I see the problem again (which already came back once), I'll report.

[kalefranz]

Just ran conda create -n jlab scikit-image -vvv, on OSX, resulting in package plan

The following NEW packages will be INSTALLED:

    cycler:          0.10.0-py27_0
    decorator:       4.0.10-py27_1
    freetype:        2.5.5-1
    icu:             54.1-0
    jbig:            2.1-0
    jpeg:            8d-2
    libpng:          1.6.22-0
    libtiff:         4.0.6-2
    matplotlib:      1.5.3-np111py27_1
    mkl:             11.3.3-0
    networkx:        1.11-py27_0
    numpy:           1.11.2-py27_0
    openssl:         1.0.2j-0
    pillow:          3.4.2-py27_0
    pip:             9.0.1-py27_1
    pyparsing:       2.1.4-py27_0
    pyqt:            5.6.0-py27_1
    python:          2.7.12-1
    python-dateutil: 2.6.0-py27_0
    pytz:            2016.10-py27_0
    qt:              5.6.2-0
    readline:        6.2-2
    scikit-image:    0.12.3-np111py27_1
    scipy:           0.18.1-np111py27_0
    setuptools:      27.2.0-py27_0
    sip:             4.18-py27_0
    six:             1.10.0-py27_0
    sqlite:          3.13.0-0
    tk:              8.5.18-0
    wheel:           0.29.0-py27_0
    xz:              5.2.2-0
    zlib:            1.2.8-3

The error didn't trigger. Still investigating...

[RatanRSur]

also seeing this issue on Arch Linux when attempting to make env from .yml provided by CNTK
conda env create --file ./scripts/install/linux/conda-linux-cntk-py35-environment.yml

[fperez]

@kalefranz, thanks for testing! Indeed it doesn't always happen: once I nuked my entire conda install and reinstalled anew, it worked fine. So evidently there's some state that goes south at some point, for reasons unknown to me. The mysteries of the universe :)

[kalefranz]

With all the errors and stack traces in this thread, there are actually multiple things happening I think. It's not just one bug to fix. This issue is actually a huge part of the motivation for the prioritized feature work in the 4.3 release. All of the work dealing with "safety."

I honestly don't think there's anything I can do with conda to help with the errors here where broken environments have been created. There are just too many things that can go wrong. (In each case, I could of course poke around in the guts of the environment, figure out what went wrong, and then manually edit some of it to pull you back out. But that's just not practical.)

In 4.3 I'm working damn hard to make sure we don't break your environment in the first place.

For all of the environments and installs that have been hosed, I'm sorry. It sucks. It probably hurts me just as much or more than you.

For all of the issues reported here, I'm making sure I at least have a response to it in the code, even if I can't go back and fix something that's already been hosed. The PR https://github.com/conda/conda/pull/4090 is more work in that direction.

[kalefranz]

For a lot of these "failed to link because file exists errors", if you want to manually hack your environment, just delete the file and try again. Conda will actually start to do that for you, but it'll make you use the --force flag. It'll also provide as much information as possible about what's actually happening.

[kalefranz]

@fperez your NotADirectoryError...

The six-1.10.0-py35_0 package has the file lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info. It's a file, not a directory. I see that file when trying to reproduce this.

kfranz@0283:~/continuum/conda *more-link-pre ❯ ls -al /usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/envs/jlab/lib/python3.5/site-packages/
...
-rw-r--r--    2 kfranz  admin   1.4K Oct  8  2015 six-1.10.0-py3.5.egg-info

Definitely a file. Now, how is conda being told that there is this file it should link lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO? I'm stumped so far. It's not listed in the info/files manifest for that six package for me.

kfranz@0283:~/continuum/conda *more-link-pre ❯ cat /usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/pkgs/six-1.10.0-py35_0/info/files
lib/python3.5/site-packages/__pycache__/six.cpython-35.pyc
lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info
lib/python3.5/site-packages/six.py

I'm really confused right now. I grok all of the various issues reported on this thread except for this one.

[fperez]

Thanks for digging further into this, @kalefranz. I've already nuked my old conda env, so all I can offer to help you debug would be to see if I can get my ~/usr/conda/ directory back from a TimeMachine backup, tar it up and send it your way.

If you think that would be useful let me know and I can try.

[kalefranz]

@fperez I've beefed up all of this transaction safety code even more and in several places. I'm ok waiting until (or hopefully even if) it happens again, and at that point the tarballed env would definitely be helpful.

[fperez]

@kalefranz, it happened again on my freshly installed miniconda :(

I've updated to conda 4.3.1 from conda-canary and uploaded a full log here, I hope that gives you something to work with.

I've also uploaded here a tarball of my entire conda directory, which I keep at ~/usr/conda, for you to play with. I hope this helps you replicate the problem, and thanks for any help you can provide. This particular bug has really been a thorn on my side over the last few months, as every time it hits me, my only option is to delete my entire conda installation and start from scratch.

BTW, if in the meantime you have any tips on possible workarounds, I'd love to hear. Since my current solution is so disruptive, even a temporary band-aid would be good to know about.

And thanks again for your persistent help!

[desilinguist]

I've found that adding --copy usually solves the problem for me. Not ideal but it seems to work.

[fperez]

Thanks for the tip, @desilinguist! Unfortunately it didn't help in this case, same error:

denali[~]> conda create -n iptest ipython --copy
Fetching package metadata .......
Solving package specifications: ..........

Package plan for installation in environment /Users/fperez/usr/conda/envs/iptest:

The following NEW packages will be INSTALLED:

    appnope:          0.1.0-py35_0  (copy)
    decorator:        4.0.10-py35_1 (copy)
    ipython:          5.1.0-py35_1  (copy)
    ipython_genutils: 0.1.0-py35_0  (copy)
    openssl:          1.0.2j-0      (copy)
    path.py:          8.2.1-py35_0  (copy)
    pexpect:          4.0.1-py35_0  (copy)
    pickleshare:      0.7.4-py35_0  (copy)
    pip:              9.0.1-py35_1  (copy)
    prompt_toolkit:   1.0.9-py35_0  (copy)
    ptyprocess:       0.5.1-py35_0  (copy)
    pygments:         2.1.3-py35_0  (copy)
    python:           3.5.2-0       (copy)
    readline:         6.2-2         (copy)
    setuptools:       27.2.0-py35_0 (copy)
    simplegeneric:    0.8.1-py35_1  (copy)
    six:              1.10.0-py35_0 (copy)
    sqlite:           3.13.0-0      (copy)
    tk:               8.5.18-0      (copy)
    traitlets:        4.3.1-py35_0  (copy)
    wcwidth:          0.1.7-py35_0  (copy)
    wheel:            0.29.0-py35_0 (copy)
    xz:               5.2.2-0       (copy)
    zlib:             1.2.8-3       (copy)

Proceed ([y]/n)? y

Linking packages ...
[six                 ]|####################################                       |  62%

CondaOSError: OS error: failed to link (src='/Users/fperez/usr/conda/pkgs/six-1.10.0-py35_0/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO', dst='/Users/fperez/usr/conda/envs/iptest/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO', type=3, error=NotADirectoryError(20, 'Not a directory'))

[ozak]

Is this all the same underlying cause for the CondaOSError: OS error: failed to link error, although with various different symptoms? Because it seems the actual OSERROR is different for different people. In @fperez cases it seems to be 'Not a directory'. In @uduwage and own my case it was a 'File exists' error. For @dhirschfeld win32 hard link failed, for @BKJackson 'No such file or directory'.

[warlck]

@fperez can you try creating folder at /Users/fperez/usr/conda/pkgs/six-1.10.0-py35_0/lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info and put file named PKG-INFO in it? This way you will have folder and the file needed to continue the creation of environment... If that helps, then it is the issue with the python egg format.

[fperez]

First, I can't create that as a directory, as it's already a file. So fundamentally there's a problem with the contents of this package vs. what conda expects.

More importantly: the specifics of the egg format are besides the point: this came from a conda package with conda-only operations. That sequence of events should not lead to a "wedged" installation. If the packages were mis-built at the source, it's still a conda issue (whether this package itself needs its build changed or the conda tool needs to cope with a new edge case, I don't know).

So yes, maybe there's issues with eggs. But from the perspective of a conda user, this still manifests as a conda problem, and we'll need to figure out a conda-layer solution. Happy to do any extra testing that could help move the issue forward...

[gkanarek]

Any recent progress on this?

[kalefranz]

@fperez I just circled back. Conda 4.3 was doing the right thing here

The package for six located at /Users/fperez/usr/conda/pkgs/six-1.10.0-py35_0
appears to be corrupted. The path 'lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/SOURCES.txt'
specified in the package manifest cannot be found.

If the package is bad, there's no reliable way for conda to "guess" what it should be. The only thing conda can do is refuse to install it, but leave your current install the way it is. That's what happened.

I looked at the conda.tgz file you uploaded, and the corrupt package is pretty apparent. The contents of pkgs/six-1.10.0-py35_0/info/files is

lib/python3.5/site-packages/__pycache__/six.cpython-35.pyc
lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/PKG-INFO
lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/SOURCES.txt
lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/dependency_links.txt
lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info/top_level.txt
lib/python3.5/site-packages/six.py

but in that package those four egg-info files don't exist, and lib/python3.5/site-packages/six-1.10.0-py3.5.egg-info is instead a file.

I just checked the package as it exists right now at https://repo.continuum.io/pkgs/free/osx-64/six-1.10.0-py35_0.tar.bz2. It appears the package has been fixed, so you shouldn't have this problem with six again.

Conda does cache packages, and if @ilanschnell opts to update a package without bumping the build number, conda probably won't download the new package if it's already cached. conda clean --packages --tarballs can help here, but if conda sees the package as "in use," it won't right now remove it from the cache. rm -rf $PREFIX/pkgs is a perfectly good solution if you don't mind fresh downloads to populate the cache.

[kalefranz]

From all the evidence I have here and elsewhere, conda 4.3 has solved the issue of vague failures when encountering corrupt packages. I think it's safe to consider this issue closed. If anybody encounters a problem they think is related, please open a new issue, but feel free to reference this issue in the new one.

[ilanschnell]

I can verify that all six packages on the default channel are fine (the content of info/files corresponds to the content of the tar archive. I did this using anaconda-verify.

From our logs (the default channel is under revision control), I can see that osx-64/six-1.10.0-py35_0.tar.bz2 was added on Oct 8 2015, and never changed after that. It is possible that the bad six package came from another channel?

[kalefranz]

Yeah I have some ideas to help sort that out in the future. We can write channel URL and other repodata package information to the info directory as a new file immediately after we extract the tarball. That would help with some of this accounting.

On Jan 22, 2017, at 9:12 PM, Ilan Schnell [e-mail] wrote:

I can verify that all six packages on the default channel are fine (the content of info/files corresponds to the content of the tar archive. I did this using anaconda-verify.

From our logs (the default channel is under revision control), I can see that osx-64/six-1.10.0-py35_0.tar.bz2 was added on Oct 8 2015, and never changed after that. It is possible that the bad six package came from another channel?

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

[fperez]

Thanks, folks! I have no idea how I ended up with that corrupt package, as I was using only the default channels (not even conda-forge). But I'm happy to at least know that either a full clean, or if need be, a manual rm -rfof the package cache, can help out.

I'll still report if I see the problem again so at least you guys know how it emerges...

[rdv-chio]

Hi, just wanted to let you know that I am having similar issues. I was trying to install orange; the only solution I found was to force changing the ownership from all the anaconda3 folders to my user.