Vault: mlock should be supported on FreeBSD but apparently isn't

[dch]

Describe the bug

since https://github.com/hashicorp/vault/pull/1297 mlock should be supported on FreeBSD, however trying to run this fails with Error initializing core: Failed to lock memory: cannot allocate memory

To Reproduce

Steps to reproduce the behavior:

1. On latest std FreeBSD 12.0p3, install vault from FreeBSD packages or download the 1.0.3 Hashicorp binary
2. make a simple config file that is not dev mode* (as mlock warning is not shown in dev mode)
3. run vault and get spanked by mlock error

# vault version
Vault v1.0.3 ('85909e3373aa743c34a6a0ab59131f61fd9e8e43')

# vault server -config=/usr/local/etc/vault/vault.hcl
Error initializing core: Failed to lock memory: cannot allocate memory

This usually means that the mlock syscall is not available.
Vault uses mlock to prevent memory from being swapped to
disk. This requires root privileges as well as a machine
that supports mlock. Please enable mlock on your system or
disable Vault from using it. To disable Vault from using it,
set the `disable_mlock` configuration option in your configuration
file.

Expected behavior

vault runs with mlock support and no warnings, whether using official HC binary or ports-compiled FreeBSD version.

Environment:

• vault 1.0.3 official release & also FreeBSD security/vault port.
• FreeBSD 12.0-RELEASE-p3 amd64

Vault server configuration file(s):

# /usr/local/etc/vault/vault.hcl
# disable_mlock = true

listener "tcp" {  address = "127.0.0.1:8200"
  tls_disable = 1
}

storage "file" {
  path = "/var/db/vault"
}

Additional context

I've not had this work in the past before, so I can't say whether it has ever worked or not since the original PR.

Running with disable_mlock=true uncommented, vault logs that mlock is supported but actually disabled. huh.

vault server -config=/usr/local/etc/vault/vault.hcl
==> Vault server configuration:

                     Cgo: disabled
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
                 Storage: file
                 Version: Vault v1.0.3
             Version Sha: 85909e3373aa743c34a6a0ab59131f61fd9e8e43

[gem-pete]

I can confirm I am seeing this on my FreeBSD systems as well. Would appreciate any insight, or areas of the code we should look at to help move this forward.

[jefferai]

It should be supported so it's not clear why it's not working. My guess is that it needs a different syscall. Anyone interested that wants to track down the issue, we'd be happy to have a fix!

[gem-pete]

Hey there @dch I did some testing on my end. Setting this sysctl knob allows me to to run vault as a non-root user:

vm.old_mlock=1

otherwise the process will need to be started as root (which is not desirable). I believe the root cause of this is due to a new interaction between mlockall() and RLIMIT_MEMLOCK resource limits. Check out the manpage for mlockall(2) and you'll see:

If vm.old_mlock is set to 1 the per-process RLIMIT_MEMLOCK resource limit 
will not be applied for mlockall() calls.

It would be interesting to see what has changed recently to trigger this new behavior - I may hit up the freebsd mailing lists and see if I can find a better answer. But for now this workaround works on my end.

[dch]

Thanks @gem-pete looks like this change landed a while back, https://github.com/freebsd/freebsd/commit/15b6949. It's not clear to me why this also doesn't work when vault is run directly as root without this fall-back sysctl. That shouldn't be necessary. I've enabled kern.racct.enable=1 as well, and even as root this still doesn't work.

This is what we want:

• run as non-root user
• set login class to (e.g. daemon) where this would pick up memorylocked=128M from /etc/login.conf
• this could be applied to the default port and updated vault docs

I whipped up this hack for helper/mlock/mlock_unix.go to clarify what errors we are hitting:

diff --git a/helper/mlock/mlock_unix.go b/helper/mlock/mlock_unix.go
index af0a69d48..a83c801e3 100644
--- a/helper/mlock/mlock_unix.go
+++ b/helper/mlock/mlock_unix.go
@@ -3,16 +3,29 @@
 package mlock

 import (
-       "syscall"
+    "fmt"
+    "syscall"

-       "golang.org/x/sys/unix"
+    "golang.org/x/sys/unix"
 )

 func init() {
-       supported = true
+    supported = true
 }

 func lockMemory() error {
-       // Mlockall prevents all current and future pages from being swapped out.
-       return unix.Mlockall(syscall.MCL_CURRENT | syscall.MCL_FUTURE)
+    // Mlockall prevents all current and future pages from being swapped out.
+    result := unix.Mlockall(syscall.MCL_CURRENT | syscall.MCL_FUTURE)
+    switch result {
+    case syscall.ENOMEM:
+        fmt.Println("mlock: ENOMEM user has insufficient RLIMIT_MEMLOCK - adjust /etc/login.conf")
+    case syscall.EPERM:
+        fmt.Println("mlock: EPERM user has insufficient privileges - rtry running as different user class")
+    case nil:
+        fmt.Println("mlock: granted")
+    default:
+        err := result.(syscall.Errno)
+        fmt.Printf("mlock: unhandled error %v:%s", uintptr(err), result)
+    }
+    return result
 }

And it appears that simply running vault as root is not sufficient - the login class needs to be applied, an in our vault instance, seting memorylocked=128M in login.conf is also not sufficient - but 1024M is. The error messages are important!

@gem-pete I will adjust the FreeBSD port to use a daemon login class, which would allow small vault installs to use the defaults, and add a comment about this to the port docs too for larger setups.

@jefferai I'll send a PR adding more useful error message to https://github.com/hashicorp/vault/blob/master/vault/core.go#L655-L669 and mentioning something in the docs about this.

[jefferai]

Rather than update that message in core.go, it might be better to split out logic for the helper using build flags so that freebsd isn't used for mlock_unix.go and instead you make an mlock_freebsd.go file. Then the mlock helper can be responsible for returning a verbose enough error message (while still including the underlying error) rather than having it hardcoded in core.

[gem-pete]

And it appears that simply running vault as root is _not_ sufficient - the login class needs to be applied, an in our vault instance, seting memorylocked=128M in login.conf is _also_ not sufficient - but 1024M is. The error messages are important!

@gem-pete I will adjust the FreeBSD port to use a daemon login class, which would allow small vault installs to use the defaults, and add a comment about this to the port docs too for larger setups.

sounds great!

if you happen to remember when submitting a PR for the ports tree it would be rad if you could add me as a watcher:
petenomadlogicorg

i'll keep my eyes peeled on the mailing list as well :)

[dch]

Looking through the code you can have the same issues on other UNIXes and Linux too so having a generic reminder error in core IMO is a Good Thing. First a pr anyway ;-)

[jefferai]

You're right, but as those UNIXes are found, they can be split off into OS-specific helper files. Up until that point moving the generic message to mlock_unix means that when those are created we don't have to keep modifying the core message with OS-specific info.

[dch]

see https://reviews.freebsd.org/D20025 for addressing this upstream

[swills]

see https://reviews.freebsd.org/D20025 for addressing this upstream

Taken care of.

[aphorise]

Interestingly the issue is more root vs non-root access to mlock and behaviour specific to FreeBSD - you can circumvent the issue by running Vault via:

sudo su -l root -c 'vault server -config=/etc/vault.d/vault.cfg'

==> Vault server configuration:

             Api Address: https://192.168.178.253:8200
                     Cgo: disabled
         Cluster Address: https://192.168.178.253:8201
              Go Version: go1.14.4
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "192.168.178.253:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: trace
                   Mlock: supported: true, enabled: true
           Recovery Mode: false
                 Storage: file
                 Version: Vault v1.5.0

PS - This attempt was with using the binaries from releases.hashicorp.com.

[aphorise]

Thanks to the support & guidance provided on IRC-Freenode #freebsd (kudos to :crown: @swills :point_left: ) - using pkg manager binaries and the steps below I am able to confirm that mlock works as with a service / account as expected.

sudo pkg install vault ;

Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    vault: 1.5.0

Number of packages to be installed: 1

The process will require 93 MiB more space.
20 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching vault-1.5.0.txz: 100%   20 MiB   6.9MB/s    00:03
Checking integrity... done (0 conflicting)
[1/1] Installing vault-1.5.0...
===> Creating groups.
Creating group 'vault' with gid '471'.
===> Creating users
Creating user 'vault' with uid '471'.
[1/1] Extracting vault-1.5.0: 100%
=====
Message from vault-1.5.0:

--
The vault user created by the vault package is now a member of the daemon
class, which will allow it to use mlock() when started by the rc script. This
will not be reflected in systems where the user already exists. Please add the
vault user to the daemon class manually by running:

pw usermod -L daemon -n vault

or delete the user and reinstall the package.

You may also need to increase memorylocked for the daemon class in
/etc/login.conf to 256M or more and run:

cap_mkdb /etc/login.conf

Or to disable mlock, add:

disable_mlock = 1

to /usr/local/etc/vault.hcl

# // entries in: /etc/rc.conf to be made via sysrc
VUSER=$(whoami) ;  # // VAULT EXECUTING USER - will use my own account.
sudo sysrc vault_user=${VUSER} ;
sudo sysrc vault_login_class=root ;
sudo sysrc vault_enable=yes ;
sudo sysrc vault_syslog_output_enable=yes ;

# // ensure all syslog stuff are in place:
sudo nano /etc/syslog.conf ; # add:
#'daemon.info                                     /var/log/daemon.log'
sudo touch /var/log/daemon.log ;
sudo service syslog restart ;
sudo service vault restart ;
sudo service vault status ;

sudo cat /var/log/daemon.log ;

… vault[3388]: ==> Vault server configuration:
… vault[3388]:
… vault[3388]:              Api Address: https://192.168.178.253:8200
… vault[3388]:                      Cgo: enabled
… vault[3388]:          Cluster Address: https://192.168.178.253:8201
… vault[3388]:               Go Version: go1.14.6
… vault[3388]:               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "192.168.178.253:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
… vault[3388]:                Log Level: trace
… vault[3388]:                    Mlock: supported: true, enabled: true
… vault[3388]:            Recovery Mode: false
… vault[3388]:                  Storage: file
… vault[3388]:                  Version: Vault v1.5.0
… vault[3388]:
… vault[3388]: ==> Vault server started! Log data will stream in below:

[timotheosh]

Just a side note for posterity, since I was just trying to run vault on FreeNAS, if you try to run vault in a jail, also be sure to add "allow.mlock" = 1 to jail parameters

[aphorise]

@dch Can you kindly confirm that this issue is resolved as described / detailed earlier.

[dch]

yep, works here