Vault: Using Vault CLI installed via APT in automation flows breaks with an "Operation not permitted"

[josemaia]

Describe the bug
Replacing an installation of the Vault CLI binary in an automation flow (e.g. jenkins agent, rundeck agent) with the new APT installation causes the automation flow to stop working

To Reproduce

1. Write a Dockerfile similar to this

FROM adoptopenjdk/openjdk11:debianslim

RUN apt-get update \
 && apt-get install -y software-properties-common \
    apt-transport-https \
    ca-certificates \
    curl \
    wget \
    gnupg2 \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*

RUN curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add - \
  && apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  && apt-get -y update && apt-get install -y vault=1.3.2 \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

RUN vault --help

1. Run docker build .
2. Observe the error vault: Operation not permitted

Expected behavior
APT is a drop-in replacement for wget/unzip of the .bin file, with additional features optional (for example, if i want to have vault as a systemd service, that should not really affect the scenario in which i only want it for the binary)

Environment:

• Vault Server Version (retrieve with vault status): 1.3.2
• Vault CLI Version (retrieve with vault version): 1.3.2
• Server Operating System/Architecture: tested on debian/ubuntu variants

[kpenfound]

Hi @josemaia , thanks for all of the detail! I believe this is because of the binary capabilities (IPC_LOCK) not being available from within the docker context. More information can be found on that here: https://hub.docker.com/_/vault

I'm not sure if docker build can be run with added capabilities or not, but the link above shows how to allow the container to use those capabilities with docker run.

If you're running vault with disable_mlock and do not need these capabilities on the installed binary, a potential workaround would be to add RUN setcap cap_ipc_lock= /usr/bin/vault before executing vault. More information on disable_mlock and it's implications here: https://www.vaultproject.io/docs/configuration#disable_mlock

[josemaia]

Is there any reason why the binary-only version did not require adding IPC_LOCK, though? Assuming I don't want to run vault server from my docker image, and only use it to interact with the server's API (for convenience's sake, it's often easier to write a vault read instead of writing an API call), I don't see why this difference would exist.

[kpenfound]

The installed package takes care of the necessary setup steps previously required to be done manually - such as placing the binary in the appropriate path, creating a service user, creating a default configuration, and providing a systemd unit file. In addition to this, part of the required steps for a manual installation are to set the binary's capabilities documented in this guide.

In your case, are you able to run the container with the --cap-add=IPC_LOCK flag? And is there a specific reason that vault --help needs to be executed in the Dockerfile?

If you're not able to run with that flag, I would suggest adding RUN setcap cap_ipc_lock= /usr/bin/vault to the dockerfile since this will not be run as a vault server.

[josemaia]

vault --help was simply an example, and yeah, with IPC_LOCK or the container running as privileged things work. we'll see on a case-by-case basis whether to keep the wget or add IPC_LOCK, I suppose. Thanks!

[flixx]

Hello,

we also run into this issue since we are using vault as CLI in a gitlab CI with an alpine base image.
In that image, setcomp needed to be installed first (called libcap). This is what solved it:

apk add --no-cache vault libcap
setcap cap_ipc_lock= /usr/sbin/vault

Felix