Server: Simplify self-hosting for seasoned admins

[blackwind]

Clearly, care has been taken to make self-hosting Bitwarden easy for anyone, but the approach used introduces needless complexity for seasoned admins.

Anyone using Docker is going to be familiar with docker-compose, and GUI users will be leveraging Portainer or their NAS's custom frontend to manage their containers. Given that, I have a number of concerns about the current bitwarden.sh setup:

• It clutters my existing Docker management tools with numerous Bitwarden containers, when it should be one appliance, one container.
• I need to use a shell script to manage Bitwarden instead of my existing tools.
• The shell script routinely pulls other shell scripts from the internet -- a positively anus-puckering prospect indeed, requiring constant manual review.
• I need to manually update my database after upgrades.

It'd be great if the distribution were condensed into one image that automatically updated my database at startup when needed, and I could then simply toss this in docker-compose.yaml and move on with my life:

bitwarden:
  container_name: bitwarden
  image: bitwarden/all
  environment:
    - UID=123
    - GID=123
  ports:
    - 443:443
  volumes:
    - /var/bitwarden:/var/bitwarden

If simplicity is the goal, nothing could be simpler than that -- especially the current bitwarden.sh setup. Sure, keep bitwarden.sh around for users who have no clue what they're doing, but don't let it get in the way of those who do.

[kspearrin]

Duplicate feature request from here: https://community.bitwarden.com/t/non-scripted-deployments/383

[DomiStyle]

I would also appreciate just a simple docker-compose.yml with current versions, necessary environment variables, links, etc. so I don't need to look through shell scripts every time I want to update Bitwarden.

That would make deploying Bitwarden with an orchestrator much easier, currently you have to update the tags, run the updater/installer manually, check for changed config files (nginx, etc.), check for new ports used, check for new environment variables and then incorporate all of that manually into your orchestrator.

[wohlben]

...please don't consolidate the image.
official documentations

what isn't mentioned here is that you'll also lose the ability to utilize health checks and properly configure the logging driver with a consolidated image.

[seffyroff]

The crux of the issue for me is the reliance on external scripting, which is an additional burden to maintain, rather than having it simply deployable (like many other complex microcontainer service mesh based apps). Please don't consolidate - I want to be able to scale out and use placement constraints (so the db deploys to the server I have db-optimized storage on for example). Move any scripting logic into the binaries themselves, so we can control them with envs or config templating, or containerize the scripting so that the Bitwarden environment is built within the existing tools a Docker hacker is already using. It seems to me that all of the scripting could be accomplished with a Dockerfile?

[kspearrin]

We don't have any plans to consolidate the containers. Docker's own documentation states that a best practice for a multi-service application like Bitwarden should have one process per container. This is why Docker Compose is used.

As far as the scripting, if you are a seasoned Docker admin I don't see a reason why you couldn't reproduce the output of the setup scripts yourself, however you want. Using the scripts certainly aren't a requirement. All the setup script/container does is create certs, setup docker-compose.yml and .env files, and generate a few config files that the containers need like nginx config. If you want to create and manage these dependencies yourself, you certainly can. Just run through the scripted setup process once and observe the output. Of course, if you have any questions, feel free to ask.

The only exception would be reliance on the setup container for doing database migrations. We can look into ways to make that automatically run on one of the container's (ex. the API) startup each time as suggested. That's probably a good idea.

[DomiStyle]

We don't have any plans to consolidate the containers.

I agree with this, docker-compose makes it really easy to create the stack, no matter how many containers there are.

As far as the scripting, if you are a seasoned Docker admin I don't see a reason why you couldn't reproduce the output of the setup scripts yourself, however you want.

Which is why just a plain docker-compose.yml with all necessary images, ports and environment variables would be great. Currently you need to grab these by going through the regular update process to make sure nothing changed. Which isn't the hardest thing to do but unnecessary since you need to repeat this process for every update instead of just looking at a single docker-compose file.

We can look into ways to make that automatically run on one of the container's (ex. the API) startup each time as suggested. That's probably a good idea.

I also prefer it when each container can take care of itself. The migrations should run from the API container and the nginx config should be provided by the nginx container if necessary. I don't really like the idea of having to use the setup container. It's great for use with an install script but horrible when using it with an orchestrator.

The ideal scenario would be to switch out the tags on the images, restart the stack and the stack takes care of updating itself.

[seffyroff]

Thanks for the replies. Indeed, the point I'm stuck on right now with my own adaptation for Swarm is the database migration. Deploying the setup container as a Service like this:

docker service create \
--restart-condition=none \
--name setup \
--network bw_default \
--mount type=bind,source=/var/data/bw/bwdata/,destination=/bitwarden \
bitwarden/setup:latest dotnet Setup.dll -update 1 -db 1

I get the following output from the setup container:

Migrating database.

Unhandled Exception: System.Data.SqlClient.SqlException: CREATE FILE encountered operating system error 2(The system cannot find the file specified.) while attempting to open or create the physical file '/var/opt/mssql/data/vault.mdf'.

CREATE DATABASE failed. Some file names listed could not be created. Check related errors.

at Bit.Setup.Program.MigrateDatabase(Int32 attempt) in /Users/kyle/Projects/bitwarden/core/util/Setup/Program.cs:line 285

at Bit.Setup.Program.Main(String[] args) in /Users/kyle/Projects/bitwarden/core/util/Setup/Program.cs:line 57

I'll iterate over that next and see if I can hit my mark. Any advice would be appreciated.

[seffyroff]

aaand I got it working by checking my file ownership and adding in the envs for the PGID and PUID to the setup container service deployment. Looks like it's working now, I'll do some testing then post my initial findings for others, after a bit of cleanup!

[seffyroff]

So I'm seeing a few initial problems. The Icons container terminates after a while with this error:

warn: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[28]

  The response could not be cached for this request because the 'Content-Length' did not match the body length.

Process is terminating due to StackOverflowException.

I'm unable to import my premium license, I get "An unhandled server error has occurred." in the vault when I attempt to do so.

And I don't see any logs in the nginx container, although this could be intentional?

[MarcusWichelmann]

I would really appreciate to not have to use these setup scripts, too.
I have already a running docker environment with an own nginx proxy and letsencrypt and want to keep my setup straight-forwarded without adding a secondary nginx-proxy behind the proxy and probably mess some things up with my subdomains.

The best solution to make the life of seasoned admin's a lot easier would be IMO to add some pages to the docs about

• how a docker-compose.yml could look like
• how the config-files should look like (and what each configuration parameter means, if not obvious)
• what the nginx-proxy does (which subdomains exists and to which docker container should these point to. does it do SSL?)
• how database migrations work

Then every admin could easily use these information to adopt the samples to his own server setups and not everybody has to dig around with the bash scripts every time there is an update.

As an example, I would prefer to add all the http- and database-servers of bitwarden to my existing docker-compose and simply configure some more subdomains in my proxy to make them point to port 80 of the bitwarden containers. SSL is all done by my proxy using a wildcard certificate.

[crazy-max]

Same case for me, i have a running docker swarm environment with an existing reverse proxy (traefik) and i just want a docker-compose file for my bitwarden stack.

[kspearrin]

1.24.0 adds a new ./bwdata/config.yml file that allows you to customize various aspects of the installation in more detail. You can also disable auto-generation of the docker-compose.yml and nginx.conf if you so desire.

[SamuelNitsche]

Are there some news? It should not be too hard to provide a simple and up to date docker-compose.yml file.

[DomiStyle]

Anything new @kspearrin?

The config.yml is no use when running in Kubernetes, I need to know which volumes and which environment variables which container needs so I can set them accordingly. Providing an always up to date docker-compose.yml and/or .env files in the main repository would solve this.

Having to rerun the install script on a separate host every time a update releases to check if any new environment variables or volumes were added is not ideal.

[kspearrin]

docker-compose.yml and other resources are generated by the install scripts based on the templates defined here: https://github.com/bitwarden/server/tree/master/util/Setup/Templates

Would it be helpful if I generated these resources (from the templates) and included them with each release on GitHub?

[DomiStyle]

Would it be helpful if I generated these resources (from the templates) and included them with each release on GitHub?

Yes, that would be really helpful.

[kspearrin]

I added a step to the CI builds to run the install script and stub out all of the templated docker resources (docker-compose.yml, env files, nginx conf, etc). These are then zipped up and attached as an artifact to each build (docker-stub.zip). See here as an example. Upon each release on GitHub this docker-stub.zip will also be attached as an asset.

Does that work?

[DomiStyle]

Does that work?

That looks good to me.

Will the actual releases feature the correct tags instead of latest in the docker-compose.yml? Would be nice to know which core version and web version belong together.

[kspearrin]

The versions for web vault are a bit disconnected from this repo at the moment, so you'll need to maintain tags yourself.

[JoelLinn]

Sorry but it is totally beyond me how anybody could think having an install script you need to run on your docker host is the best and only way one should deploy a software.
It totally breaks what docker (also) is about: Isolating your host from your application.
I will never deploy any software in docker using anything different than compose files.

[evanstucker-hates-2fa]

Joel, a simple "+1 I would like a Docker compose file" is sufficient. Thanks, kspearrin for providing this awesome, free software.

[kspearrin]

This is now available in v1.30.1.

1. As mentioned previously, a docker-stub.zip is now included with the release assets here on GitHub.
2. The database migration script was moved into the application code and gets automatically executed each time the admin container is started.

You can find the "Manual Docker Installation" instructions documented in our help center here: https://help.bitwarden.com/article/install-on-premise/#manual-docker-installations

[Beanow]

For anyone else planning to deploy this on docker swarm, here's my progress in trying to make that happen so far. There are still some issues with it and hopefully when those are worked out I'll get around to writing a full guide with templates and such.

Steps are mostly the same as https://help.bitwarden.com/article/install-on-premise/#manual-docker-installations except you will move all non-secret env vars to the stack file, and secrets to the bitwarden_server_secrets.1.json secret.

Example stack.yml

# Useful references:
# https://docs.docker.com/compose/compose-file/

# You probably want 3.4 as a minimum, because of extension fields and becase update order is relevant.
version: '3.4'

# Shared configuration values.
x-global-env: &global-env
  ASPNETCORE_ENVIRONMENT: Production
  globalSettings__selfHosted: 'true'
  globalSettings__baseServiceUri__vault: https://bitwarden.company.com
  globalSettings__baseServiceUri__api: https://bitwarden.company.com/api
  globalSettings__baseServiceUri__identity: https://bitwarden.company.com/identity
  globalSettings__baseServiceUri__admin: https://bitwarden.company.com/admin
  globalSettings__baseServiceUri__notifications: https://bitwarden.company.com/notifications
  globalSettings__disableUserRegistration: 'false'
  globalSettings__mail__replyToEmail: [email protected]
  globalSettings__mail__smtp__host: smtp.company.com
  globalSettings__mail__smtp__port: 587
  globalSettings__mail__smtp__ssl: 'false'
  # Internal settings
  globalSettings__baseServiceUri__internalNotifications: http://notifications:5000
  globalSettings__baseServiceUri__internalAdmin: http://admin:5000
  globalSettings__baseServiceUri__internalIdentity: http://identity:5000
  globalSettings__baseServiceUri__internalApi: http://api:5000
  globalSettings__baseServiceUri__internalVault: http://web:5000
  globalSettings__pushRelayBaseUri: https://push.bitwarden.com
  globalSettings__installation__identityUri: https://identity.bitwarden.com
  globalSettings__licenseDirectory: /etc/bitwarden/core/licenses

x-admin-settings: &admin-settings
  adminSettings__admins: ''

# These let you run the services under a different user.
# The default represents nobody:nogroup.
x-local-uid: &LOCAL_UID '65534'
x-local-gid: &LOCAL_GID '65534'

# Change the default deploy config.
x-deploy-defaults: &deploy-defaults
  mode: replicated
  restart_policy:
    condition: any
    window: 5s
    delay: 5s

# Shortcuts for ourselves.
# It's not a configuration value.
x-secret-mode: &secret-mode
  uid: *LOCAL_UID
  gid: *LOCAL_GID
  mode: 0400

# Shortcuts for ourselves.
# It's not a configuration value.
x-run-as: &run-as
  LOCAL_UID: *LOCAL_UID
  LOCAL_GID: *LOCAL_GID

services:

  # Several security settings are enforced using settings here.
  # Therefore this is the ONLY container you can expose to the web.
  # Directly if you configured TLS here.
  # Or indirectly through your own TLS termination and/or load-balancer.
  nginx:
    image: bitwarden/nginx:latest
    healthcheck:
      test: wget --quiet --tries=1 --spider http://localhost:8080/ || exit 1
      interval: 5s
      timeout: 3s
      retries: 3
    configs:
      # Configs are easier to scale and update in swarms compared to sharing volumes.
      - source: bitwarden-nginx-default.1.conf
        target: /etc/bitwarden/nginx/default.conf
    volumes:
      - nginx-logs:/var/log/nginx
    # If your swarm does not provide any other web-facing services, you can expose it this way.
    # ports:
    #   - 80:8080
    #   - 443:8443
    networks:
      - default
      # - traefik
    deploy:
      # It's OK to scale this as you like.
      <<: *deploy-defaults
      replicas: 2
      update_config:
        # Start first, because these run fine concurrently.
        order: start-first
      # But if you're like me and use traefik (and it's let's encrypt), you can expose it this way.
      # Note, you will need to add the networks too.
      # labels:
      #   traefik.enable: "true"
      #   traefik.port: 8080
      #   traefik.docker.network: "traefik"
      #   traefik.frontend.rule: "Host: bitwarden.company.com"
      #   traefik.frontend.entryPoints: "http,https"

  mssql:
    image: bitwarden/mssql:latest
    environment:
      <<: *run-as
      # Of course, you should read the EULA before deploying this and change it to Y.
      ACCEPT_EULA: 'N'
      MSSQL_PID: Express
      # This is new and will require a change to this image's entrypoint.
      SA_PASSWORD_FILE: /run/secrets/sa_password
    secrets:
      - <<: *secret-mode
        source: bitwarden_mssql_sa_password.1.txt
        target: sa_password
    volumes:
      - mssql-backups:/etc/bitwarden/mssql/backups
      - mssql-data:/var/opt/mssql/data
      - mssql-logs:/var/opt/mssql/log
    deploy:
      # It's NOT OK to change the scaling here.
      # Find out yourself how you can do high-availibility MSSQL if you need it.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Stop first, because the data volume shouldn't be shared with multiple instances.
        order: stop-first

  web:
    image: bitwarden/web:latest
    environment:
      <<: *global-env
      <<: *run-as
    configs:
      # Configs are easier to scale and update in swarms compared to sharing volumes.
      - source: bitwarden-web-app-id.1.json
        target: /etc/bitwarden/web/app-id.json
    deploy:
      # It's OK to scale this as you like.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Start first, because these run fine concurrently.
        order: start-first

  attachments:
    image: bitwarden/attachments:latest
    environment:
      <<: *global-env
      <<: *run-as
    volumes:
      - attachments:/etc/bitwarden/core/attachments
    deploy:
      # I don't know if this will scale without problems. Needs more looking into.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Stop first, because I don't know if we can run concurrently.
        order: stop-first

  api:
    image: bitwarden/api:latest
    secrets:
      - <<: *secret-mode
        source: bitwarden_server_secrets.1.json
        target: /app/appsettings.Production.json
    volumes:
      - api-core:/etc/bitwarden/core
      - api-logs:/etc/bitwarden/logs
      # - ../ca-certificates:/etc/bitwarden/ca-certificates
    environment:
      <<: *global-env
      <<: *run-as
    deploy:
      # I don't know if this will scale without problems. Needs more looking into.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Stop first, because I don't know if we can run concurrently.
        order: stop-first

  identity:
    image: bitwarden/identity:latest
    secrets:
      - <<: *secret-mode
        source: bitwarden_server_secrets.1.json
        target: /app/appsettings.Production.json
      - <<: *secret-mode
        source: bitwarden_identity_key.1.pfx
        target: /etc/bitwarden/identity/identity.pfx
    volumes:
      - identity-core:/etc/bitwarden/core
      - identity-logs:/etc/bitwarden/logs
      # - ../ca-certificates:/etc/bitwarden/ca-certificates
    environment:
      <<: *global-env
      <<: *run-as
    deploy:
      # I don't know if this will scale without problems. Needs more looking into.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Stop first, because I don't know if we can run concurrently.
        order: stop-first

  admin:
    image: bitwarden/admin:latest
    secrets:
      - <<: *secret-mode
        source: bitwarden_server_secrets.1.json
        target: /app/appsettings.Production.json
    volumes:
      - admin-core:/etc/bitwarden/core
      - admin-logs:/etc/bitwarden/logs
      # - ../ca-certificates:/etc/bitwarden/ca-certificates
    environment:
      <<: *global-env
      <<: *run-as
      <<: *admin-settings
    deploy:
      # I don't know if this will scale without problems. Needs more looking into.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Stop first, because I don't know if we can run concurrently.
        order: stop-first

  icons:
    image: bitwarden/icons:latest
    environment:
      <<: *global-env
      <<: *run-as
    volumes:
      - icons-logs:/etc/bitwarden/logs
    deploy:
      # I don't know if this will scale without problems. Needs more looking into.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Stop first, because I don't know if we can run concurrently.
        order: stop-first

  notifications:
    image: bitwarden/notifications:latest
    secrets:
      - <<: *secret-mode
        source: bitwarden_server_secrets.1.json
        target: /app/appsettings.Production.json
    volumes:
      - notifications-logs:/etc/bitwarden/logs
      # - ../ca-certificates:/etc/bitwarden/ca-certificates
    environment:
      <<: *global-env
      <<: *run-as
    deploy:
      # I don't know if this will scale without problems. Needs more looking into.
      <<: *deploy-defaults
      replicas: 1
      update_config:
        # Stop first, because I don't know if we can run concurrently.
        order: stop-first

volumes:
  # If you need your data to migrate between hosts, you probably want to specify a volume driver.
  # See https://docs.docker.com/compose/compose-file/#volume-configuration-reference
  # If this isn't possible in your configuration and you must use local storage,
  # I would recommend you use labels to pin your deployment to a specific host.
  mssql-backups:
  mssql-data:

  # I don't know if these can work in isolation or are supposed to be shared mutable state.
  attachments:
  api-core:
  identity-core:
  admin-core:

  # Logs may also have issues with scaling/concurrency.
  mssql-logs:
  api-logs:
  identity-logs:
  admin-logs:
  icons-logs:
  notifications-logs:
  nginx-logs:

configs:
  bitwarden-web-app-id.1.json:
    external: true
  bitwarden-nginx-default.1.conf:
    external: true

secrets:
  bitwarden_identity_key.1.pfx:
    external: true
  bitwarden_mssql_sa_password.1.txt:
    external: true
  bitwarden_server_secrets.1.json:
    external: true

networks:
  default:
  # traefik:
  #   external: true

The PR here https://github.com/bitwarden/server/pull/477 will need to be applied to support reading SA_PASSWORD from a file in mssql.

The format for bitwarden_server_secrets.1.json

{
  "globalSettings": {
    "installation": {
      "id": "globalSettings__installation__id",
      "key": "globalSettings__installation__key"
    },
    "identityServer": {
      "certificatePassword": "globalSettings__identityServer__certificatePassword"
    },
    "sqlServer": {
      "connectionString": "globalSettings__sqlServer__connectionString"
    },
    "duo": {
      "aKey": "globalSettings__duo__aKey"
    },
    "mail": {
      "smtp": {
        "username": "globalSettings__mail__smtp__username",
        "password": "globalSettings__mail__smtp__password"
      }
    },
    "internalIdentityKey": "globalSettings__internalIdentityKey"
  }
}

This essentially replaces the /app/appsettings.Production.json file in order to load secrets from a file instead of ENV.

It _may_ break things in the future doing it this way, because some important value could be set exclusively in the production variant. _Currently_ though, it's only base URLs (which we override anyway) and payment providers switching to production mode. Which you're probably not using for your self-hosted instance.

Other secrets are:

• bitwarden_mssql_sa_password.1.txt the value of SA_PASSWORD just placed in a text file.
• bitwarden_identity_key.1.pfx the identity.pfx you've created in earlier steps.

Config value bitwarden-nginx-default.1.conf

server {
  listen 8080 default_server;
  listen [::]:8080 default_server;
  server_name localhost;
  include /etc/nginx/security-headers.conf;

  location / {
    proxy_pass http://web:5000/;
    include /etc/nginx/security-headers.conf;
    add_header Content-Security-Policy "default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com; child-src 'self' https://*.duosecurity.com; frame-src 'self' https://*.duosecurity.com; connect-src 'self' wss://bitwarden.robin-it.com https://api.pwnedpasswords.com https://twofactorauth.org; object-src 'self' blob:;";
    add_header X-Frame-Options SAMEORIGIN;
    add_header X-Robots-Tag "noindex, nofollow";
  }

  location = /app-id.json {
    proxy_pass http://web:5000/app-id.json;
    include /etc/nginx/security-headers.conf;
    proxy_hide_header Content-Type;
    add_header Content-Type $fido_content_type;
  }

  location = /duo-connector.html {
    proxy_pass http://web:5000/duo-connector.html;
  }

  location = /u2f-connector.html {
    proxy_pass http://web:5000/u2f-connector.html;
  }

  location /attachments/ {
    proxy_pass http://attachments:5000/;
  }

  location /api/ {
    proxy_pass http://api:5000/;
  }

  location /identity/ {
    proxy_pass http://identity:5000/;
  }

  location /icons/ {
    proxy_pass http://icons:5000/;
  }

  location /notifications/ {
    proxy_pass http://notifications:5000/;
  }

  location /notifications/hub {
    proxy_pass http://notifications:5000/hub;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $http_connection;
  }

  location /admin {
    proxy_pass http://admin:5000;
    include /etc/nginx/security-headers.conf;
    add_header X-Frame-Options SAMEORIGIN;
  }
}

This assumes you will do TLS termination elsewhere.

Other configs:

• bitwarden-web-app-id.1.json the contents of web/app-id.json after you've changed the URL.

[rschoenbichler]

I would like this issue to be re-opened, as I just went through the installation process in an working docker environment where I already have a reverse proxy and ssl management set up.
I have to say this was not a pleasant task, due to the various config files I had to change quite drastically (like the nginx config for ssl, the secret keys in various property files, the host names all over the place).

Imho, the docker integration currently seems to be more like a workaround based on the setup script solution. I personally think a much cleaner experience would be a single docker container that just exposes the web port and expects the configuration properties (like secret keys, installation id & key, host name etc) which would allow for further docker based composition down the road.
Still, a setup script could be provided for the less experienced users, that would just generate a docker-compose.yml file introducing configs based on the users choices (like if the user wants a lets-encrypt instance to be running).

[Beanow]

I would like this issue to be re-opened, as I just went through the installation process [...]. I have to say this was not a pleasant task, ...

Having spent the better part of a weekend on the previous comment and related PRs for a deployment, I would agree easier deployments would be helpful.

That said, I would assume @kspearrin graciously hosting bitwarden.com as a freemium service and sharing as oss, will have different scaling concerns than your average 1-5 users first self-hosted installation would.

So I think what this issue needs, is someone not in the nr 1 contributor position by a landslide working on this. Just my 2 cents.

[h-town]

brotha just wants some mod_security and headersmore :/

[Ciantic]

This could be done effectively with Docker in Docker. The aim is to inherit from dind image and run the install script inside that container, so that it starts the all required containers.

I've tried running kubernetes cluster inside a docker container without nested vm (using kind), and it works just fine.