Vault: status=213/Securebits on systemctl start

[Zwags]

Hello All,

My team recently deployed a copy of vault, hooked into a consul backend on a Debian server using Terraform. The deployment of the servers and Consul goes off without a hitch, however when we get to Vault, it fails on start up.
When I do a systemctl status vault.services I get an error:
Process: 16657 ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl (code=exited, status=213/SECUREBITS)

I don't think its a permissions issue, because in desperation my team just did 777 on anything vault and this process still seemed to fail. If the command vault server -dev or vault server -config /etc/vault.hcl is run, then the server comes up fine. We're thinking its something to do OS level, but we're stuck at this point. Anyone able to help out?

This is our vault.hcl file:
```ui = true
disable_mlock = true

listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}

storage "consul" {
address = "127.0.0.1:8500"
path = "vault/"
}

api_addr = "http://10.10.8.150:8200"
cluster_addr = "http://10.10.8.150:8201"

[ncabatoff]

I believe this is a symptom of vault running as a non-root user and not having been granted the cap_ipc_lock (nor having mlock disabled.) See Server Configuration and the section on disable_mlock.

[briankassouf]

Anything in the vault logs?

[Zwags]

So I am running everything through an elevated system user named Vault. If I jump onto the server and elevate myself to root, I get the same issue doing anything through systemctl

Also if I do disable_mlock=true in the vault.hcl file, the same 213 error shows up.

Only error I seem to get is running a journal after trying to start it as root

Dez 13 08:12:08 ebvault1 systemd[1]: Started a tool for managing secrets.
-- Subject: Unit vault.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit vault.service has finished starting up.
-- 
-- The start-up result is done.
Dez 13 08:12:08 ebvault1 systemd[17220]: vault.service: Failed at step SECUREBITS spawning /usr/local/bin/vault: Operation not permitted
-- Subject: Process /usr/local/bin/vault could not be executed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The process /usr/local/bin/vault could not be executed and failed.
-- 
-- The error number returned by this process is 1.

However, if I take what its trying to run and run the command as root or vault user:

/usr/local/bin/vault  server -config /etc/vault.hcl
==> Vault server configuration:

             Api Address: http://10.10.8.150:8200
                     Cgo: disabled
         Cluster Address: https://10.10.8.150:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: (not set)
                   Mlock: supported: true, enabled: false
                 Storage: consul (HA available)
                 Version: Vault v1.0.0
             Version Sha: c19cef14891751a23eaa9b41fd456d1f99e7e856

==> Vault server started! Log data will stream in below:

Which is why I'm kinda looking at systemd as the issue here. I tried repeating this problem in Ubuntu Server 16 and systemd seems to have no issues with it, so I'm wondering if this is an issue where my team just has to bite the bullet and have our vault cluster run in Ubuntu.

[Zwags]

Oh as an aisde, this is the vault.service file

[Unit]
Description=a tool for managing secrets
Documentation=https://vaultproject.io/docs/
After=network.target
ConditionFileNotEmpty=/etc/vault.hcl

[Service]
User=vault
Group=vault
ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl
ExecReload=/usr/local/bin/kill --signal HUP $MAINPID
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
SecureBits=keep-caps
NoNewPrivileges=yes
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

[ncabatoff]

I was able to reproduce the error ("Failed at step SECUREBITS spawning /usr/local/bin/vault: Operation not permitted") on a debian/jessie box running Vault in systemd.

To do so I had to delete the following line from the vault.service file you gave:

Capabilities=CAP_IPC_LOCK+ep

I'm not sure how to account for the fact that you're getting the error with that line in your vault.service file. Is it possible you changed it and forgot to run systemctl daemon-reload before restarting the service?

The repo https://github.com/ncabatoff/minvault demonstrates Vault working on Debian Jessie using a vault.service file much like the one you gave above.

I'm closing this issue because I don't believe there's a bug here, if you need more help with this I suggest you try the mailing list (https://groups.google.com/forum/#!forum/vault-tool).

[Zwags]

Thanks for looking into it! My team has decided instead to move towards just disabling (or later on encrypting) swap on the Vault server in order to get this working for testing. I'll look into that repo though and test removing that Capabilities line from the vault.service file.
Thanks again!

[ncabatoff]

test removing that Capabilities line from the vault.service file

I think you may have misunderstood me. I was saying that I could only reproduce your problem by removing that line. With the line present, everything just worked.

[ncabatoff]

On the off chance this helps someone else: once I moved to stretch I experienced the same problem. I saw errors in the logs saying "Support for option Capabilities= has been removed and it is ignored". Replacing the Capabilities= line with AmbientCapabilities=CAP_IPC_LOCK did the trick.

[jwaiswa7]

Thanks @ncabatoff , Your solution was of great help

[johnataravir]

I ran across this issue trying to do a test install of Vault on CentOS 8. None of the above helped, but after struggling with it for awhile, I remembered that SELinux likes to make things like this difficult. It's enabled by default on Cent 8, and disabling it did the trick.

[gtbromen]

Hello @johnataravir
I have same issue on rhel 8 please johnataravir could you explain what did you exactly do on centos to solve the issue