Conda-build: `conda build .` or `conda build` command fails and generates a [Errno 36] File name too long: '/home/krishnab/anaconda3/conda-bld/conda_pyugend_1473224912779/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place'

[00krishna]

I am trying to build a package with conda build. My build script has worked all along until today.

I use the conda build . command as usual to build the package and load it to the Anaconda cloud. But now I am getting an error like this posted below. Not sure what the problem is.

Removing old build environment
BUILD START: pyugend-0.5-py35_0
updating index in: /home/krishnab/anaconda3/conda-bld/linux-64
updating index in: /home/krishnab/anaconda3/conda-bld/noarch
Using Anaconda Cloud api site https://api.anaconda.org

The following NEW packages will be INSTALLED:

openssl:    1.0.2h-1      (soft-link)
pip:        8.1.2-py35_0  (soft-link)
python:     3.5.2-0       (soft-link)
readline:   6.2-2         (soft-link)
setuptools: 26.1.1-py35_0 (soft-link)
sqlite:     3.13.0-0      (soft-link)
tk:         8.5.18-0      (soft-link)
wheel:      0.29.0-py35_0 (soft-link)
xz:         5.2.2-0       (soft-link)
zlib:       1.2.8-3       (soft-link)

 Traceback (most recent call last):
 File "/home/krishnab/anaconda3/bin/conda-build", line 6, in <module>
 sys.exit(conda_build.cli.main_build.main())
 File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda_build/cli/main_build.py", line    
234, in main
execute(sys.argv[1:])
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda_build/cli/main_build.py", line 
226, in execute
already_built=None, config=config)
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda_build/api.py", line 81, in build
need_source_download=need_source_download, config=config)
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda_build/build.py", line 942, in 
 build_tree
config=config)
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda_build/build.py", line 546, in build
create_env(config.build_prefix, specs, config=config)
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda_build/build.py", line 440, in    
create_env
plan.execute_actions(actions, index, verbose=config.debug)
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda/plan.py", line 599, in  
execute_actions
inst.execute_instructions(plan, index, verbose)
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda/instructions.py", line 135, in 
execute_instructions
cmd(state, arg)
File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda/instructions.py", line 78, in 
LINK_CMD
 link(state['prefix'], dist, lt, index=state['index'])
 File "/home/krishnab/anaconda3/lib/python3.5/site-packages/conda/install.py", line 1060, in link
os.makedirs(dst_dir)
File "/home/krishnab/anaconda3/lib/python3.5/os.py", line 231, in makedirs
makedirs(head, mode, exist_ok)
File "/home/krishnab/anaconda3/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
OSError: [Errno 36] File name too long: '/home/krishnab/anaconda3/conda-  
  bld/conda_pyugend_1473224912779/_b_env_placehold_placehold_placehold_placehold_placehold_pl acehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place'

[patricksnape]

Which OS version and preferably which file system as well?

[00krishna]

I am using ubuntu 14.04 x64. Filesystem wise I am using ext4. Does that answer your question.

Also, here is my conda info:

Current conda install:

         platform : linux-64
    conda version : 4.1.11
conda-env version : 2.5.2

conda-build version : 2.0.0
python version : 3.5.2.final.0
requests version : 2.9.1
root environment : /home/XXX/anaconda3 (writable)
default environment : /home/XXX/anaconda3
envs directories : /home/XXX/anaconda3/envs
package cache : /home/XXX/anaconda3/pkgs
channel URLs : https://repo.continuum.io/pkgs/free/linux-64/
https://repo.continuum.io/pkgs/free/noarch/
https://repo.continuum.io/pkgs/pro/linux-64/
https://repo.continuum.io/pkgs/pro/noarch/
config file : /home/XXX/.condarc
offline mode : False
is foreign system : False

[msarahan]

Conda-build 2 extended the prefix length from 80 to 255 characters. The confusing thing about your report is that Linux's filename/folder length limit is 255 characters:

http://serverfault.com/questions/9546/filename-length-limits-on-linux

It may be that makedirs is somehow treating that whole string as a single "filename"? I don't know how else we would be violating the 255 char limit here. That limit is a per-folder limit, not a whole string limit.

[00krishna]

Good question. I was just wondering whether this "placehold..." folder was legit, or if it was something temporary. I just checked and the letter count on this filename is 255 exactly. Is there any plan to shorten the length of this name--if the current name is temporary? I am trying to think of what else I can do to get around this issue.

It might also be that the name is really longer than what is displayed. I think that the name got truncated at 255 characters, but it seems like it is actually longer.

Note, I just tried it again. It seems to keep adding a ._1473270073183 directory to the conda-build directory and then tries to load the long name folder to that location during the actual build process.

[msarahan]

The placehold folder is legitimate, and it is intentionally exactly 255 characters. It is intentionally truncated. There are no plans to shorten it, unless it becomes a widespread problem. We are using this on our linux machines, but you are the first bug reporter here.

at a shell, can you create that folder?

mkdir -p /home/krishnab/anaconda3/conda-bld/conda_pyugend_1473224912779/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place

[00krishna]

Okay cool. I just tried it and am getting the filename too long error. Here is a picture of the screen.

Here is a copy of the actual text in case the screenshot test is too small.

(py35_gu) [krishnab@sankayana build_script]$ mkdir -p /home/krishnab/anaconda3/conda-bld/conda_pyugend_1473224912779/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
mkdir: cannot create directory ‘/home/krishnab/anaconda3/conda-bld/conda_pyugend_1473224912779/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place’: File name too long

[msarahan]

Great. That takes Python out of the equation. What happens if you delete just one character off the end there, making it 254 characters?

[00krishna]

I was still getting the same error message with removing just one character. I tried a few different combinations and finally found the magic filename and length:

mkdir -p /home/krishnab/anaconda3/conda-bld/conda_pyugend_1473224912779/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh

I calculated this out and it came to 216 characters.

That is really weird. I suppose technically the filename limit is 255 bytes. Are any characters like "/" or "_" consuming more than one byte?

[msarahan]

What locale are you? It's definitely possible that some characters are occupying more than one byte. We could handle that better, definitely. It's too easy to forget this stuff on a US system.

[00krishna]

I am in Los Angeles, so United States. But I think I might know what the issue is. My hard drive is encrypted, so when the new filename is created, I bet the encryptfs is generating an encrypted string with characters that require more than one byte.

I found an post about a similar issue--check the very bottom post:
https://ubuntuforums.org/showthread.php?t=2258374

Hmm, I imagine a bunch of folks must have encrypted home directories through, so I am surprised that no one else has mentioned the issue. Have you ever run into this before? Can you think of any other possible reasons?

[msarahan]

That's very interesting. This is the first I've encountered. I have disk encryption turned on, but I have a Mac. I guess a workaround (probably easier) would be to install miniconda/anaconda in a non-encrypted location.

If you have better suggestions for how I can handle this better in conda-build, I'm happy to explore them. Without a clear example to follow, I'm afraid this is too involved for me right now.

[00krishna]

Yeah, no prob. FYI though, I do think the problem was related to the encryption.
I just reinstalled anaconda to a new unencrypted drive I am no longer getting that error. The package not builds correctly.

Perhaps it might be good idea to include a note in the README or something. Perhaps something to the effect like:

"Please note that if you are receiving an error about a filename being too long, please check whether Anaconda is installed to an encrypted folder or drive. In such cases, a filename with a proper length might have a different length when encrypted--thus causing the error. The solution is to install Anaconda or Miniconda to a drive that is not encrypted."

[msarahan]

I agree - that belongs in a FAQ of some sort. I'll try to see about a place that is more visible than our wiki. Perhaps the README.

[msarahan]

Thanks for closing. I added the info at https://github.com/conda/conda-build/blob/master/README.rst#gotchasfaq

[patricksnape]

Yes I had read that encrypted paths must be of shorter length.

[marcelm]

My homedir is also encrypted and I get the same error message. Any chance this could be fixed properly?

[msarahan]

I'm open to pull requests, but cannot prioritize this issue right now given that there is an easy (if ugly) workaround. I'll add it to the long-term backlog for if there's a spare moment in the future.

[moorepants]

I'd also like to see this fixed. I have an encrypted home directory on ubuntu 16.10 and just hit this error even though I've built plenty of conda packages before.

[msarahan]

@moorepants based on your thumbs-up in #1502, I'm assuming you've seen that discussion. I don't have time to research and fix things that have such easy workarounds. I'll gladly accept PRs or even just explanations of how encryption works on Ubuntu, but otherwise won't be able to devote development time to this for quite a while - currently ~2-3 months.

[moorepants]

I did read through some of the issues. I don't have any knowledge about how encrypted directories work on linux, but the main issue is that I've been using conda build for a long time now and with the upgrade to 2, it no longer works. If I understand correctly, the work around is to install anaconda/miniconda to somewhere other than, for example my encrypted home directory. One of the nice features of conda is that you can install in your home directory without other permissions, so this isn't that ideal in my case.

Isn't the simple fix just to set the character limit to a number that works instead of 255? Then it works on encrypted directories. Maybe this can be a user option? Or selected if an encrypted directory is detected.

This post:

http://unix.stackexchange.com/questions/32795/what-is-the-maximum-allowed-filename-and-folder-size-with-ecryptfs

says:

Empirically, we have found that character filenames longer than 143 characters start requiring >255 characters to encrypt. So we (as eCryptfs upstream developers) typically recommend you limit your filenames to ~140 characters.

So it looks like 140 could be a nice balance for users that need longer filenames and those who run conda from encrypted directories. Is 255 really necessary if 80 was working for the majority of cases pre-2.0?

[msarahan]

the work around is to install anaconda/miniconda to somewhere other than, for example my encrypted home directory

This is not completely accurate. The only thing that needs to be outside of the encrypted home folder is the temporary build folders created by conda-build. You can pass the --croot argument to conda-build to change this location to something else. On CI, I often change it to /tmp/cbtmp to avoid issues with legacy packages that only have 80 character prefixes. It will create folders as necessary - you don't need to create them for it.

Changing the prefix length once was painful enough. I'm not going back on that decision. Also, this statement:

character filenames longer than 143 characters start requiring >255 characters to encrypt.

implies to me that the prefix lengths depend on the value of the path, not strictly the length of the input path. That concerns me greatly. We can't have variable length prefixes depending on who has built a given package. The prefix length must be fixed for sanity.

[moorepants]

The only thing that needs to be outside of the encrypted home folder is the temporary build folders created by conda-build. You can pass the --croot argument to conda-build to change this location to something else.

Ok, that seems like a reasonable work around. Can this be set in .condarc?

[msarahan]

Not currently, but I'll add some kind of config file for the next release. Issue at https://github.com/conda/conda-build/issues/1512

[msarahan]

I was wrong. This is actually pretty much the only condarc setting that conda-build has: http://conda.pydata.org/docs/config.html#specify-conda-build-output-root-directory-conda-build

However, there appear to be some issues with conda and that setting: https://github.com/conda/conda/issues/3744#issuecomment-260044918

[jwnelson]

Just adding another data point. Experiencing the same issue on Ubuntu 16.04.1 LTS with an encrypted home folder with conda 4.2.9 and conda-build 2.0.2.

Can using --croot as a workaround still work without root privileges if the entire home folder is encrypted?

[msarahan]

--croot will always work, but you must choose a location that is not encrypted. /tmp/<whatever you want> is a good candidate.

[msarahan]

For anyone encountering this, as of #1579 you'll be able to change the prefix length with a command line argument. I still don't recommend it - it's much better to set croot to some non-encrypted folder - but the option is there.