Bitwarden_rs: Websocket Notification Traefik

[budimanjojo]

Hello, I just migrated from Nginx to Traefik. And everything works perfectly unless my Bitwarden. So, I followed the proxy example from https://github.com/dani-garcia/bitwarden_rs/wiki/Proxy-examples and set WEBSOCKET_ENABLED=true in the env file.
This is what I got from the log:
[ws::handler][ERROR] WS Error <Protocol>: Unable to parse WebSocket key.
I can't even access my vault, browser shows this error:
WebSocket Protocol Error: Unable to parse WebSocket key.
If I comment out the line:

      - "traefik.hub.frontend.rule=Host:vault.jojocomptech.com;Path:/notifications/hub"
      - "traefik.hub.port=3012"

in my docker-compose file, then everything is working again except I can't auto sync my data across devices.
This is my docker-compose file (example domain name of course):

version: '3.7'

services:
  bitwarden_rs:
    image: "mprasil/bitwarden:latest"
    container_name: "bitwarden_rs"
    restart: always
    env_file:
      - config.env
    volumes:
      - "app:/data"
      - "db:/database"
      - "log:/log"
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.backend=bitwarden_rs"
      - "traefik.web.frontend.rule=Host:bitwarden.example.com"
      - "traefik.web.port=80"
      - "traefik.hub.frontend.rule=Host:bitwarden.example.com;Path:/notifications/hub"
      - "traefik.hub.port=3012"
      - "traefik.docker.network=traefik_default"
      - "traefik.frontend.headers.SSLRedirect=true"
      - "traefik.frontend.headers.STSSeconds=315360000"
      - "traefik.frontend.headers.browserXSSFilter=true"
      - "traefik.frontend.headers.contentTypeNosniff=true"
      - "traefik.frontend.headers.forceSTSHeader=true"
      - "traefik.frontend.headers.SSLHost=example.com"
      - "traefik.frontend.headers.STSIncludeSubdomains=true"
      - "traefik.frontend.headers.STSPreload=true"
      - "traefik.frontend.headers.frameDeny=true"
      - "traefik.frontend.customFrameOptionsValue=SAMEORIGIN"

networks:
  traefik:
    external:
      name: traefik_default

volumes:
  app:
  db:
  log:

And this is my config.env file (example domain and credentials of course):

SIGNUPS_ALLOWED=true
INVITATIONS_ALLOWED=false
WEBSOCKET_ENABLED=true
SHOW_PASSWORD_HINT=false
DOMAIN=https://bitwarden.example.com
DATABASE_URL=/database/bitwarden_rs.sqlite3
ATTACMENTS_FOLDER=/attachments
ICON_CACHE_FOLDER=/icon_cache
LOG_FILE=/log/bitwarden_rs.log
ROCKET_WORKERS=15
WEB_VAULT_ENABLED=true
SMTP_HOST=mail.example.com
[email protected]
SMTP_FROM_NAME=My Name
SMTP_PORT=587
SMTP_SSL=true
SMTP_USERNAME=myname
SMTP_PASSWORD=123456

Thanks before :+1:

[mprasil]

@fbartels had that issue and the culprit was some headers missing. I see you have some extra labels set for your container, maybe one of those is blocking some headers? I'd try to remove those and strip down the configuration to the example on wiki, if that works, try adding more labels until it breaks again.

[budimanjojo]

Thanks for your fast reply. I just commented out all the "traefik.frontend.header*" label in the compose file. And the problem persists. Here's the entire logs, maybe you can find something:

bitwarden_rs docker-compose up -d && docker-compose logs -f                     bitwarden_rs is up-to-date
Attaching to bitwarden_rs
bitwarden_rs    | /--------------------------------------------------------------------\
bitwarden_rs    | |                       Starting Bitwarden_RS                        |
bitwarden_rs    | |                       Version 1.8.0-3bb46ce4                       |
bitwarden_rs    | |--------------------------------------------------------------------|
bitwarden_rs    | | This is an *unofficial* Bitwarden implementation, DO NOT use the   |
bitwarden_rs    | | official channels to report bugs/features, regardless of client.   |
bitwarden_rs    | | Report URL: https://github.com/dani-garcia/bitwarden_rs/issues/new |
bitwarden_rs    | \--------------------------------------------------------------------/
bitwarden_rs    |
bitwarden_rs    | [2019-04-11 11:06:07][launch][INFO] Configured for staging.
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] address: 0.0.0.0
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] port: 80
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] log: normal
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] workers: 15
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] secret key: generated
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] limits: forms = 32KiB, json* = 10MiB
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] keep-alive: 5s
bitwarden_rs    | [2019-04-11 11:06:07][launch_][INFO] tls: disabled
bitwarden_rs    | [2019-04-11 11:06:07][rocket::fairing::fairings][INFO] Fairings:
bitwarden_rs    | [2019-04-11 11:06:07][_][INFO] 1 response: Application Headers
bitwarden_rs    | [2019-04-11 11:06:07][launch][INFO] Rocket has launched from http://0.0.0.0:80
bitwarden_rs    | [2019-04-11 11:06:07][ws][INFO] Listening for new connections on 0.0.0.0:3012.
bitwarden_rs    | [2019-04-11 11:06:31][ws::io][INFO] Accepted a new tcp connection from 172.18.0.2:56614.
bitwarden_rs    | [2019-04-11 11:06:31][ws::handler][ERROR] WS Error <Protocol>: Unable to parse WebSocket key.
bitwarden_rs    | [2019-04-11 11:06:32][rocket::rocket][INFO] GET /favicon.ico image/webp:
bitwarden_rs    | [2019-04-11 11:06:32][_][INFO] Matched: GET /<p..> [10] (web_files)
bitwarden_rs    | [2019-04-11 11:06:32][_][INFO] Outcome: Success
bitwarden_rs    | [2019-04-11 11:06:32][_][INFO] Response succeeded.
bitwarden_rs    | [2019-04-11 11:11:56][ws::io][INFO] Accepted a new tcp connection from 172.18.0.2:36346.
bitwarden_rs    | [2019-04-11 11:11:56][ws::handler][ERROR] WS Error <Protocol>: Unable to parse WebSocket key.
bitwarden_rs    | [2019-04-11 11:11:57][rocket::rocket][INFO] GET /favicon.ico image/webp:
bitwarden_rs    | [2019-04-11 11:11:57][_][INFO] Matched: GET /<p..> [10] (web_files)
bitwarden_rs    | [2019-04-11 11:11:57][_][INFO] Outcome: Success
bitwarden_rs    | [2019-04-11 11:11:57][_][INFO] Response succeeded.
bitwarden_rs    | [2019-04-11 11:11:58][ws::io][INFO] Accepted a new tcp connection from 172.18.0.2:36516.
bitwarden_rs    | [2019-04-11 11:11:58][ws::handler][ERROR] WS Error <Protocol>: Unable to parse WebSocket key.
bitwarden_rs    | [2019-04-11 11:11:58][rocket::rocket][INFO] GET /favicon.ico image/webp:
bitwarden_rs    | [2019-04-11 11:11:58][_][INFO] Matched: GET /<p..> [10] (web_files)
bitwarden_rs    | [2019-04-11 11:11:58][_][INFO] Outcome: Success
bitwarden_rs    | [2019-04-11 11:11:58][_][INFO] Response succeeded.

Thanks again :)

[budbay]

@budimanjojo I had a similar issue, try adding this label to your docker-compose:
- "traefik.hub.protocol=ws"
I added this in between the other two .hub labels and it resolved the websocket key error for me.
EDIT:
i also removed this line:
- "traefik.backend=bitwarden_rs"

[budimanjojo]

Wow it works. Thanks @budbay :D

[mprasil]

@budimanjojo I assume traefik.hub.protocol=ws helped?

Great help there @budbay! Any info where is this setting documented? I've seen most of the documentation recommending to just set it to http, but it looks like your setting is the correct one. I would like to update the wiki, just need to verify what this really does.

[budimanjojo]

@mprasil Yes it solved my problem. Also, I needed to remove traefik.backend=bitwarden_rs like @budbay mentioned. I was also curious and found out that there are 4 protocols supported, they are http, https, ws, wss (ws = websocket, wss = secure websocket) but I can't find the official documentation for this.
I also found this: https://github.com/containous/traefik/issues/1027

[mprasil]

Yeah removing backend makes sense, I'd expect that might break things and it's not in the sample proxy configuration.

I'm going to add the protocol settings on the wiki. Thanks for info.

[budbay]

Sorry guys I don't have the source for the websocket label that I suggested. I had the same error mentioned and was trying to troubleshoot. I don't have any other containers with websockets being proxied by traefik so I started searching any info I could find on websockets and traefik. I came across setting the protocol somewhere and thought it made sense so I tried that and it worked. glad I could help and sorry I can't provide sources, as mentioned there's not much documented anywhere.

[budimanjojo]

@budbay @mprasil Hello, sorry for bumping this issue. Traefik recently got v2.0 which require routers, services, and middlewares instead of frontend and backend. I'm planning to update my whole stack and got problem with the websocket. Any idea on how to use ws protocol in the new format? Thanks again