Nvidia-docker: Cannot install current latest version of rpm nvidia-docker2 (2.0.3-1.docker18.09.0.ce.noarch)

[elouanKeryell-Even]

1. Issue description

Cannot install current latest version (2.0.3-1.docker18.09.0.ce.noarch) of yum package nvidia-docker2 on CentOS 7.

The cause is that the nvidia-docker2 package has a strict dependency on a very particular version of Docker (18.09.0), which yum does not want to install since it is not anymore the latest available version (latest is now 18.09.1). See here, here, here, here.

+ sudo yum install -y nvidia-docker2
. . .
. . .
. . .
Error: Package: nvidia-docker2-2.0.3-1.docker18.09.0.ce.noarch (nvidia-docker)
           Requires: docker-ce = 3:18.09.0
           Installed: 3:docker-ce-18.09.1-3.el7.x86_64 (@docker-ce-stable)
               docker-ce = 3:18.09.1-3.el7
           Available: docker-ce-17.03.0.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.03.0.ce-1.el7.centos
           Available: docker-ce-17.03.1.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.03.1.ce-1.el7.centos
           Available: docker-ce-17.03.2.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.03.2.ce-1.el7.centos
           Available: docker-ce-17.03.3.ce-1.el7.x86_64 (docker-ce-stable)
               docker-ce = 17.03.3.ce-1.el7
           Available: docker-ce-17.06.0.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.06.0.ce-1.el7.centos
           Available: docker-ce-17.06.1.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.06.1.ce-1.el7.centos
           Available: docker-ce-17.06.2.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.06.2.ce-1.el7.centos
           Available: docker-ce-17.09.0.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.09.0.ce-1.el7.centos
           Available: docker-ce-17.09.1.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.09.1.ce-1.el7.centos
           Available: docker-ce-17.12.0.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.12.0.ce-1.el7.centos
           Available: docker-ce-17.12.1.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 17.12.1.ce-1.el7.centos
           Available: docker-ce-18.03.0.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 18.03.0.ce-1.el7.centos
           Available: docker-ce-18.03.1.ce-1.el7.centos.x86_64 (docker-ce-stable)
               docker-ce = 18.03.1.ce-1.el7.centos
           Available: docker-ce-18.06.0.ce-3.el7.x86_64 (docker-ce-stable)
               docker-ce = 18.06.0.ce-3.el7
           Available: docker-ce-18.06.1.ce-3.el7.x86_64 (docker-ce-stable)
               docker-ce = 18.06.1.ce-3.el7
           Available: 3:docker-ce-18.09.0-3.el7.x86_64 (docker-ce-stable)
               docker-ce = 3:18.09.0-3.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Downgrading+locking docker version works fine as a temporary fix, see here.

2. Steps to reproduce the issue

On a fresh CentOS 7 VM:

################################################################################
# Install docker-ce, version  18.09.1-3.el7
################################################################################

sudo yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
sudo yum -y install docker-ce-18.09.1-3.el7


################################################################################
# Install latest available version of package `nvidia-docker2` --> 2.0.3-1.docker18.09.0.ce.noarch
################################################################################

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \
  sudo tee /etc/yum.repos.d/nvidia-docker.repo

# Install nvidia-docker2
sudo yum install -y nvidia-docker2-2.0.3-1.docker18.09.0.ce.noarch

[turesayer]

Same problem on Linux Mint 19.1 after the last update this morning.

sudo apt-get install nvidia-docker2 
[sudo] password for xxx:             
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nvidia-docker2 : Depends: docker-ce (= 5:18.09.0~3-0~ubuntu-bionic) but 5:18.09.1~3-0~ubuntu-bionic is to be installed or
                           docker-ee (= 5:18.09.0~3-0~ubuntu-bionic) but it is not installable
E: Unable to correct problems, you have held broken packages.

[elouanKeryell-Even]

Would it be conceivable to make the RPM depend more loosely on the Docker version? For example, have it depend on docker 18.09 instead of 18.09.0. It would solve the issue.

[elouanKeryell-Even]

Here is the temporary fix I use in my installation scripts

version_is_greater_than () {
  REFERENCE_VERSION="${2}"
  VERSION_TO_BE_COMPARED="${1}"

  if [[ "${VERSION_TO_BE_COMPARED}" == "${REFERENCE_VERSION}" ]]; then
    return 1
  else
    SMALLEST_VERSION="$(echo -e "${VERSION_TO_BE_COMPARED}\n${REFERENCE_VERSION}" | sort -V | head -n 1)"
    if [[ "${REFERENCE_VERSION}" == "${SMALLEST_VERSION}" ]]; then
      return 0
    elif [[ "${VERSION_TO_BE_COMPARED}" == "${SMALLEST_VERSION}" ]]; then
      return 1
    else
      >&2 echo "Error comparing versions '${VERSION_TO_BE_COMPARED}' & '${REFERENCE_VERSION}', exiting..."
      exit 1
    fi
  fi
}

################################################################################


# Retrieve docker version required by nvidia-docker2
yum -y install yum-utils
NVIDIA_REQUIRED_DOCKER_VERSION="$(repoquery --requires nvidia-docker2 | grep 'docker-ce' | awk -F ' = ' '{ print $2 }' | cut -d ':' -f 2)"

# Retrieve installed docker version
INSTALLED_DOCKER_VERSION="$(repoquery --installed docker-ce --qf "%{ver}")"


if version_is_greater_than "$INSTALLED_DOCKER_VERSION" "$NVIDIA_REQUIRED_DOCKER_VERSION"; then
  # Special case, see issue https://github.com/NVIDIA/nvidia-docker/issues/903

  # Downgrade docker
  yum -y downgrade "docker-ce-$NVIDIA_REQUIRED_DOCKER_VERSION"

  # Lock docker version
  yum -y install yum-plugin-versionlock
  yum versionlock docker-ce-*

fi


# Install nvidia-docker2
sudo yum install -y nvidia-docker2

[Davidnet]

Same thing is happening with Bionic.

[suneeta-mall]

Same issue with Debian Stretch

[jonathonf]

Very easy workaround for deb files:

apt-get download nvidia-docker2
dpkg-deb -R nvidia-docker*deb nvidiatmp

Edit the depends line in debian/control:

Depends: nvidia-container-runtime (= 2.0.0+docker18.09.0-1), docker-ce (>= 5:18.09.0) | docker-ee (>= 5:18.09.0), docker-ce (<< 5:18.10) | docker-ee (<< 5:18.10)

Then repack and install:

dpkg-deb -b nvidiatmp fixed.deb
sudo dpkg -i fixed.deb
sudo apt-get install -f

Maintaining the rest of the control file has the benefit of the "hacked" package being upgraded when the official package is.

[RenaudWasTaken]

Hello!

We've released new packages, this should fix your bug. Unfortunately we can't relax the constraint of being tightly coupled.
We've been working on a pipeline to accelerate this.

A solution is always to downgrade to the latest minor version, this would make more sense than patching the debs...
Using version of the nvidia-container-runtime that's different from the version of docker you have may result in undefined behavior :)