Docker-stacks: conda install fails with permission denied errors

[nils-werner]

Right after starting a container, I am not able to install any packages in my running Docker containers using conda. It fails due to a permissions error.

What docker image you are using?

jupyter/scipy-notebook:621b96ed75cb

What complete docker command do you run to launch the container (omitting sensitive values)?

sudo docker run -it --rm --name condatest --user root -e NB_USER=$(whoami) -e NB_GROUP=$(id -ng) -e NB_UID=$(id -u) -e NB_GID=$(id -g) -p 8888:8888 -v work:$HOME/jupyterhub jupyter/scipy-notebook:621b96ed75cb start.sh

What steps do you take once the container is running to reproduce the issue?

1. Start the container, it should open a Bash shell

2. Run install (it should fail)

   conda install librosa
   

3. Run install again (it should succeed)

   conda install librosa
   

4. Clean up the cache

   conda clean -tipsy
   

5. Run install again (it should fail again)

   conda install librosa
   

What do you expect to happen?

Conda installs packages the first time I try.

What actually happens?

After starting the container or clearing the cache, installation fails with the following error:

/usr/local/bin/start.sh: ignoring /usr/local/bin/start-notebook.d/*

Set username to: nwerner
Relocating home dir to /home/nwerner
Setting CWD to /home/nwerner/
Set nwerner UID to: 5005
Add nwerner to group: 5000
Executing the command: bash
nwerner@fbf0041a687d:~$ conda install librosa
Solving environment: failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.6/site-packages/conda/exceptions.py", line 819, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.6/site-packages/conda/cli/main.py", line 78, in _main
        exit_code = do_call(args, p)
      File "/opt/conda/lib/python3.6/site-packages/conda/cli/conda_argparse.py", line 77, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/opt/conda/lib/python3.6/site-packages/conda/cli/main_install.py", line 11, in execute
        install(args, parser, 'install')
      File "/opt/conda/lib/python3.6/site-packages/conda/cli/install.py", line 235, in install
        force_reinstall=context.force,
      File "/opt/conda/lib/python3.6/site-packages/conda/core/solve.py", line 506, in solve_for_transaction
        force_remove, force_reinstall)
      File "/opt/conda/lib/python3.6/site-packages/conda/core/solve.py", line 439, in solve_for_diff
        final_precs = self.solve_final_state(deps_modifier, prune, ignore_pinned, force_remove)
      File "/opt/conda/lib/python3.6/site-packages/conda/core/solve.py", line 180, in solve_final_state
        index, r = self._prepare(prepared_specs)
      File "/opt/conda/lib/python3.6/site-packages/conda/core/solve.py", line 580, in _prepare
        self.subdirs, prepared_specs)
      File "/opt/conda/lib/python3.6/site-packages/conda/core/index.py", line 174, in get_reduced_index
        subdir_datas = tuple(SubdirData(Channel(url)) for url in channel_urls)
      File "/opt/conda/lib/python3.6/site-packages/conda/core/index.py", line 174, in <genexpr>
        subdir_datas = tuple(SubdirData(Channel(url)) for url in channel_urls)
      File "/opt/conda/lib/python3.6/site-packages/conda/core/subdir_data.py", line 68, in __call__
        subdir_data_instance = super(SubdirDataType, cls).__call__(channel)
      File "/opt/conda/lib/python3.6/site-packages/conda/core/subdir_data.py", line 130, in __init__
        self.cache_path_base = join(create_cache_dir(),
      File "/opt/conda/lib/python3.6/site-packages/conda/core/subdir_data.py", line 618, in create_cache_dir
        mkdir_p_sudo_safe(cache_dir)
      File "/opt/conda/lib/python3.6/site-packages/conda/gateways/disk/__init__.py", line 81, in mkdir_p_sudo_safe
        os.chown(path, uid, gid)
    PermissionError: [Errno 1] Operation not permitted: '/opt/conda/pkgs/cache'

`$ /opt/conda/bin/conda install librosa`

  environment variables:
                 CIO_TEST=<not set>
                CONDA_DIR=/opt/conda
               CONDA_ROOT=/opt/conda
                     PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin
                          :/bin
               PYTHONPATH=
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
             SUDO_COMMAND=/bin/bash
                 SUDO_GID=0
                 SUDO_UID=0
                SUDO_USER=root

     active environment : None
       user config file : /home/nwerner/.condarc
 populated config files : /opt/conda/.condarc
          conda version : 4.5.6
    conda-build version : not installed
         python version : 3.6.5.final.0
       base environment : /opt/conda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /opt/conda/pkgs
                          /home/nwerner/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/nwerner/.conda/envs
               platform : linux-64
             user-agent : conda/4.5.6 requests/2.19.1 CPython/3.6.5 Linux/4.15.0-24-generic ubuntu/18.04 glibc/2.27
                UID:GID : 5005:100
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?

As you can see, conda picks out the right cache directory /opt/conda/pkgs/cache, but fails trying to write to it.

[parente]

Could be #665.

[nils-werner]

I am not certain it really is #665. However you are right the owner of the files in /opt/conda isn't me, but some uid 1000.

nwerner@2ad214156521:~$ ls -aln /opt/conda/
total 72
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 .
drwxr-xr-x 1    0   0 4096 Jul  9 22:55 ..
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 bin
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 conda-bld
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 conda-meta
-rw-rw-r-- 1 1000 100   88 Jul  9 22:56 .condarc
drwsrwsr-x 2 1000 100 4096 Jul  9 22:56 envs
drwsrwsr-x 1 1000 100 4096 Jul 19 21:11 etc
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 include
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 lib
drwsrwsr-x 3 1000 100 4096 Jul 19 21:13 libexec
-rw-rw-r-- 1 1000 100 4134 Mar  2 07:18 LICENSE.txt
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 pkgs
drwsrwsr-x 1 1000 100 4096 Jul 19 21:13 share
drwsrwsr-x 4 1000 100 4096 Jul  9 22:57 ssl
drwsrwsr-x 1 1000 100 4096 Jul 19 21:11 var
drwsrwsr-x 3 1000 100 4096 Jul  9 22:55 x86_64-conda_cos6-linux-gnu

While I am not uid 1000 but 5005

nwerner@2ad214156521:~$ id
uid=5005(nwerner) gid=100(users) groups=100(users),5000(nwerner)

I believe this issue is mainly caused by /opt/conda/pkgs/cache/ only appearing after conda install

nwerner@2ad214156521:~$ ls -aln /opt/conda/pkgs/cache/
ls: cannot access '/opt/conda/pkgs/cache/': No such file or directory
nwerner@2ad214156521:~$ conda install librosa
[...]
nwerner@2ad214156521:~$ ls -aln /opt/conda/pkgs/cache/
total 12
drwxr-sr-x 2 5005 100 4096 Jul 22 11:20 .
drwsrwsr-x 1 1000 100 4096 Jul 22 11:20 ..

And disappears again after conda clean -tipsy

nwerner@2ad214156521:~$ conda clean -tipsy
[...]
nwerner@2ad214156521:~$ ls -aln /opt/conda/pkgs/cache/
ls: cannot access '/opt/conda/pkgs/cache/': No such file or directory

[nils-werner]

I just realized conda is trying to run os.chown() which usually doesn't work when you're not root, and then I realized conda and env are reporting some strange environment variables:

SUDO_COMMAND=/bin/bash
    SUDO_GID=0
    SUDO_UID=0
   SUDO_USER=root

but I am not root:

nwerner@319fb5503012:/opt/conda/pkgs$ whoami
nwerner
nwerner@319fb5503012:/opt/conda/pkgs$ touch blabla
nwerner@319fb5503012:/opt/conda/pkgs$ ls -al blabla
-rw-r--r-- 1 nwerner users 0 Jul 22 11:37 blabla

And removing these variables fixes the problem:

nwerner@06924ecc8ddc:~$ env | grep SUDO
SUDO_GID=0
SUDO_COMMAND=/bin/bash
SUDO_USER=root
SUDO_UID=0
nwerner@06924ecc8ddc:~$ unset SUDO_GID
nwerner@06924ecc8ddc:~$ unset SUDO_UID
nwerner@06924ecc8ddc:~$ unset SUDO_USER
nwerner@06924ecc8ddc:~$ unset SUDO_COMMAND
nwerner@06924ecc8ddc:~$ env | grep SUDO
nwerner@06924ecc8ddc:~$ conda install librosa
[...]
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

[nils-werner]

It seems this behaviour in conda has caused many other issues, so it was removed in 4.6.0

[parente]

@nils-werner Thanks for tracking down the root cause. We can update to conda 4.6 when it's out. In the meantime, unset seems like a reasonable workaround.

[nils-werner]

Thinking about this a bit more, another solution would be to replace sudo with su in start.sh:92, as su won't set those variables.

[parente]

We used to use su but switched to sudo because of its own side-effects, running bash as the entrypoint IIRC.

Edit: #509 was the PR

[parente]

The images are now using conda 4.6 so I think this is safe to close. Feel free to open a new issue if the problem persists.

[vtrajan]

I got similar problem when trying to install keras:

(FYI: I have installed anaconda for my use only. Should I have installed it in the root?
I had it installed in the root before.)

Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(568): An error occurred while uninstalling package 'defaults::python.app-2-py37_9'.
PermissionError(1, 'Operation not permitted')
Attempting to roll back.

Rolling back transaction: done

[Errno 1] Operation not permitted: '/Users/vtrajan/anaconda3/python.app/Contents/PkgInfo' -> '/Users/vtrajan/anaconda3/python.app/Contents/PkgInfo.c~'

[vtrajan]

I installed the anaconda today on my Mac. So it must be the latest.

[parente]

@vtrajan make sure you are using the latest docker image by doing a docker pull <image name> and then try your keras install against a new container spawned from that image.

[aishwaryathondapu]

I got permission issue while running the keras and pyinstaller in MacOs. When ran with the sudo it worked. But there are packages in conda which executed perfectly with out using sudo. What might be the reason. My user in Mac has Admin previliges

[parente]

@aishwaryathondapu This is an issue tracker for Project Jupyter and its use with conda in docker images. You might try asking your question about using conda on macOS in an appropriate conda support forum.

[JDOUBLE-U]

I got similar problem when trying to install keras:

(FYI: I have installed anaconda for my use only. Should I have installed it in the root?
I had it installed in the root before.)

Preparing transaction: done
Verifying transaction: done
Executing transaction: failed
ERROR conda.core.link:_execute(568): An error occurred while uninstalling package 'defaults::python.app-2-py37_9'.
PermissionError(1, 'Operation not permitted')
Attempting to roll back.

Rolling back transaction: done

[Errno 1] Operation not permitted: '/Users/vtrajan/anaconda3/python.app/Contents/PkgInfo' -> '/Users/vtrajan/anaconda3/python.app/Contents/PkgInfo.c~'

I had the same problem. I recommend to install Anaconda through the graphical installer. https://www.anaconda.com/distribution/. After that I installed keras in the terminal and it all worked.