Conda-build: "[Errno 36] File name too long" while building PyPI package.

[kalefranz]

_From @AndrewPashkin on October 28, 2016 20:33_

Here is the complete terminal output:

$ conda skeleton pypi logbook
logbook not found, trying Logbook
Warning, the following versions were found for Logbook
0.1
0.2
0.2.1
0.3
0.4
0.4.1
0.4.2
0.5.0
0.6.0
0.7.0
0.8.0
0.8.1
0.9.0
0.9.1
0.10.0
0.10.1
0.11.0
0.11.1
0.11.2
0.11.3
0.12.0
0.12.1
0.12.2
0.12.3
0.12.4
0.12.5
1.0.0
Using 1.0.0
Use --version to specify a different version.
Using url https://pypi.python.org/packages/34/e8/6419c217bbf464fe8a902418120cccaf476201bd03b50958db24d6e90f65/Logbook-1.0.0.tar.gz (174 KB) for Logbook.
Downloading Logbook
Using cached download
Unpacking Logbook...
done
working in /tmp/tmp6ukt2ro0conda_skeleton_Logbook-1.0.0.tar.gz

The following NEW packages will be INSTALLED:

    openssl:    1.0.2j-0      (soft-link)
    pip:        8.1.2-py35_0  (soft-link)
    python:     3.5.2-0       (soft-link)
    pyyaml:     3.12-py35_0   (soft-link)
    readline:   6.2-2         (soft-link)
    setuptools: 27.2.0-py35_0 (soft-link)
    sqlite:     3.13.0-0      (soft-link)
    tk:         8.5.18-0      (soft-link)
    wheel:      0.29.0-py35_0 (soft-link)
    xz:         5.2.2-0       (soft-link)
    yaml:       0.1.6-0       (soft-link)
    zlib:       1.2.8-3       (soft-link)

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.5/site-packages/conda/history.py", line 86, in update
    self.init_log_file()
  File "/opt/anaconda3/lib/python3.5/site-packages/conda/history.py", line 76, in init_log_file
    self.write_dists(linked(self.prefix))
  File "/opt/anaconda3/lib/python3.5/site-packages/conda/history.py", line 250, in write_dists
    os.makedirs(self.meta_dir)
  File "/opt/anaconda3/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/anaconda3/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
OSError: [Errno 36] File name too long: '/home/andrei/conda-bld/skeleton_1477686723967/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/anaconda3/bin/conda-skeleton", line 6, in <module>
    sys.exit(conda_build.cli.main_skeleton.main())
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/cli/main_skeleton.py", line 65, in main
    return execute(sys.argv[1:])
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/cli/main_skeleton.py", line 61, in execute
    api.skeletonize(package, args.repo, config=config)
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/api.py", line 192, in skeletonize
    recursive=recursive, config=config, **kwargs)
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/skeletons/pypi.py", line 406, in skeletonize
    noprompt, packages, config=config, setup_options=setup_options)
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/skeletons/pypi.py", line 664, in get_package_metadata
    config=config)
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/skeletons/pypi.py", line 924, in get_pkginfo
    run_setuppy(src_dir, tempdir, python_version, config=config, setup_options=setup_options)
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/skeletons/pypi.py", line 957, in run_setuppy
    config=config)
  File "/opt/anaconda3/lib/python3.5/site-packages/conda_build/build.py", line 431, in create_env
    plan.execute_actions(actions, index, verbose=config.debug)
  File "/opt/anaconda3/lib/python3.5/site-packages/conda/plan.py", line 642, in execute_actions
    with History(actions[inst.PREFIX]):
  File "/opt/anaconda3/lib/python3.5/site-packages/conda/history.py", line 67, in __enter__
    self.update('enter')
  File "/opt/anaconda3/lib/python3.5/site-packages/conda/history.py", line 105, in update
    raise CondaFileIOError(self.path, "Can't write the history file %s" % e)
conda.exceptions.CondaFileIOError: IO error: Couldn't read or write to file. '/home/andrei/conda-bld/skeleton_1477686723967/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/conda-meta/history'. Can't write the history file [Errno 36] File name too long: '/home/andrei/conda-bld/skeleton_1477686723967/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl'

_Copied from original issue: conda/conda#3765_

[msarahan]

Duplicate of #1331 #1482

suggested resolution:

1. install miniconda to a location that is not encrypted
2. pass the --croot argument to conda-build, with the value being some folder in a path that is not encrypted.

[AndreiPashkin]

@msarahan, alright, thanks for the info.

[gabrielboucher]

I fail to see how this workaround can be considered an acceptable resolution. A lot of people have encrypted homes and dismissing support for it by adding a notice to the readme is far from an ideal solution.

[msarahan]

I agree that it is not optimal, but time is finite, and the workaround is easy, though far from desirable.

PRs are welcome if you understand how this encryption works and can recommend an actual fix.

[gabrielboucher]

I'd be glad to take a look, the main thing I'm missing is what is that placeholder thing and why is it there in the first place.

[msarahan]

Sure, good question. It's for relocatability. When conda installs a package on a user's system, it is replacing prefixes wherever they are embedded in the package. These can be plain text, or they can be embedded in binaries. This helps binaries know where to find their libraries. On Mac and Linux, you can't (to the best of my knowledge) replace a prefix that is longer than the one that was used at build time. They get baked in at that size, and then you're stuck. For a long time, the standard was 80 characters. Many people had problems with that, such as HPC users with deep nested paths. To enable them to use conda at all, we increased the standard to 255 characters in https://github.com/conda/conda-build/pull/877

Now, the long prefix is only necessary for packages that embed this binary prefix. To simplify the code, I apply that long prefix everywhere, rather than trying to manage some times when it is short, and others long.

You can readily change the length of the prefix that conda-build tries to create so that it remains 255 characters after encryption. This is presently only a config setting, meaning you can only use it through the python api. https://github.com/conda/conda-build/blob/master/conda_build/config.py#L236-L242 It would be easy to make it available as a command line option, though. The tricky parts keeping me from doing much about this is understanding whether the encryption scheme is deterministic in terms of how it changes the path length, and whether we can detect somehow when encryption is turned on.

[AndreiPashkin]

When conda installs a package on a user's system, it is replacing prefixes wherever they are embedded in the package. These can be plain text, or they can be embedded in binaries. This helps binaries know where to find their libraries.

What could possibly go wrong? :smile:

[kalefranz]

@msarahan that's worthy of being pasted in an FAQ somewhere

[kalefranz]

@AndrewPashkin This is one of conda's foundational techniques, being used from the very beginning. On millions of installations on basically (if not just yet) every CPU architecture and OS that's in anyway relevant in 2016. It's battle-tested and legit. Along with other less-conventional techniques conda uses to great effect.

[msarahan]

@AndrewPashkin if you have other ideas, please tell us. I agree that this may seem hacky, but I also don't see another way. Most programs are not written to be able to be relocatable. GCC is an example that copes remarkably well in this regard (and actually Conda's relocation stuff breaks it, so we have to turn it off for GCC packages), but if you move programs around, and particularly if you have libraries in non-standard (i.e. non-system) locations, then things break pretty quickly.

One of conda's huge perks is its ability to manage local environments without requiring any elevation in permissions, and the relocatability stuff is key to this perk.

It certainly has its issues, but I think the juice is worth the squeeze.

[pedrombmachado]

Hi everyone,
I have a suggestion to work around this bug. Install anaconda on an external HDD/flash drive. Do not update the bashrc with the new path.
After the installation do:
$ cd anaconda3/bin
/anaconda3/bin$ ./conda install conda-build
/anaconda3/bin$ ./conda install conda-build
/anaconda3/bin$ ./conda skeleton pypi
I hope it works for you because it worked for me.
/anaconda3/bin$ ./conda build

[jcfr]

For the record, also just encountered that problem.

After creating my kitchen environment in a folder without encryption, everything worked well :+1:

details

The error reported below occurred while trying to run conda skeleton pypi pytest-shutil from an environment named kitchen having conda-build installed.

Traceback (most recent call last):
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/core/link.py", line 250, in _prepare
    mkdir_p(target_prefix)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/gateways/disk/__init__.py", line 60, in mkdir_p
    makedirs(path)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
OSError: [Errno 36] File name too long: '/home/jcfr/miniconda3/envs/kitchen/conda-bld/skeleton_1532842988548/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jcfr/miniconda3/envs/kitchen/bin/conda-skeleton", line 11, in <module>
    sys.exit(main())
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/cli/main_skeleton.py", line 65, in main
    return execute(sys.argv[1:])
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/cli/main_skeleton.py", line 61, in execute
    version=args.version, config=config)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/api.py", line 281, in skeletonize
    recursive=recursive, config=config, **kwargs)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/skeletons/pypi.py", line 290, in skeletonize
    setup_options=setup_options)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/skeletons/pypi.py", line 694, in get_package_metadata
    config=config)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/skeletons/pypi.py", line 994, in get_pkginfo
    setup_options=setup_options)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/skeletons/pypi.py", line 1031, in run_setuppy
    subdir=config.host_subdir, clear_cache=False, config=config)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda_build/environ.py", line 853, in create_env
    execute_actions(actions, index, verbose=config.debug)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/common/io.py", line 46, in decorated
    return f(*args, **kwds)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/plan.py", line 390, in execute_actions
    execute_instructions(plan, index, verbose)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/plan.py", line 600, in execute_instructions
    cmd(state, arg)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/instructions.py", line 70, in UNLINKLINKTRANSACTION_CMD
    unlink_link_transaction.execute()
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/core/link.py", line 223, in execute
    self.verify()
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/common/io.py", line 46, in decorated
    return f(*args, **kwds)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/core/link.py", line 200, in verify
    self.prepare()
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/core/link.py", line 192, in prepare
    stp.remove_specs, stp.update_specs)
  File "/home/jcfr/miniconda3/envs/kitchen/lib/python3.7/site-packages/conda/core/link.py", line 255, in _prepare
    "" % target_prefix)
conda.CondaError: Unable to create prefix directory '/home/jcfr/miniconda3/envs/kitchen/conda-bld/skeleton_1532842988548/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_'.
Check that you have sufficient permissions.

[ericrpatton]

suggested resolution:

1. install miniconda to a location that is not encrypted
2. pass the --croot argument to conda-build, with the value being some folder in a path that is not encrypted.

I just encountered the same error as the OP trying to install pdal. I had forgotten that my home folder was encrypted.

Can the --croot argument be used in a typical installation command like

conda create --name pdalenv --channel conda-forge pdal ?

[msarahan]

--croot is conda-build. You want something like

conda create -p /path/to/pdalenv

-p / --path can be used instead of -n / --name, not at the same time.

[tall-josh]

Sorry, I don't see how;

1. install miniconda to a location that is not encrypted
   
   
   
   1. pass the --croot argument to conda-build, with the value being some folder in a path that is not encrypted.
   
   

helps. I'm not calling conda-build. The command is conda skeleton pypi <package>. What am I missing?

EDIT:

Never mind: That's how #1331 :-(

EDIT EDIT:

Sooo, --croot is all well and good for running conda build but what about conda skeleton? After a bunch more digging I found export CONDA_BLD_PATH=<somewhere not encrypted> worked, ie:

export CONDA_BLD_PATH=/tmp/condabuild
conda skeleton pypi --version 7.0 click

[marcelotrevisani]

Sorry, I don't see how;

1. install miniconda to a location that is not encrypted
2. pass the --croot argument to conda-build, with the value being some folder in a path that is not encrypted.

helps. I'm not calling conda-build. The command is conda skeleton pypi <package>. What am I missing?

EDIT:

Never mind: That's how #1331 :-(

EDIT EDIT:

Sooo, --croot is all well and good for running conda build but what about conda skeleton? After a bunch more digging I found export CONDA_BLD_PATH=<somewhere not encrypted> worked, ie:

export CONDA_BLD_PATH=/tmp/condabuild
conda skeleton pypi --version 7.0 click

If you want one alternative you can generate the recipes looking from pypi using grayskull :skull:
and you can specify a path where the folder will be created, something like

grayskull pypi click=7.0 --output=/tmp/your_folder

[tall-josh]

I'll give it a crack @marcelotrevisani