Bitwarden_rs: Setup with Fail2Ban

[itr6]

Hello all! First off, thank you for this! It is amazing!
I was talking to u/me-ro on the homelab subreddit and they suggested I raise an issue to see if I can get some help with setting up bitwarden and Fail2Ban. He mentioned it logged all the password attempts but I can't find any of the logs.

Can someone please help?

Thanks!

[mprasil]

Yeah, this was added in PR #152 but keep in mind, that bitwarden_rs logs to stdout, so, you need to configure docker (or however you run the service) to log to file instead.

As for setting up Fail2Ban, it seems like the only missing bit might be setting up the filter. Maybe @Baelyk submitted the PR for this purpose and already has some configuration?

[itr6]

I run it in a standard docker container with all persistent data in a 'data' folder. How would I set it up to log the attempts into a file that Fail2Ban can read?

[mprasil]

You can configure logging driver for container or the daemon. Maybe something like syslog would work to get the logs somewhere where fail2ban can read them?

[itr6]

Sorry if this is really basic, I'm just starting out with docker.
But I need to add:

services:
  bitwarden:
    logging:
      driver: "syslog"

to my docker-compose file?

[mprasil]

I think that should work.

[itr6]

I added that and started the container, now when I run docker inspect --format='{{.LogPath}}' Bitwarden it doesn't return anything. Before I added the log driver it showed a log. I dont know if I have taken a step backward or what. I guess I don't know enough about this yet.

[mprasil]

I guess that's expected as it now logs to syslog. Now you need to figure out where does syslog store the messages.

[itr6]

OK so I've made a little progress. I found where the syslog is and it works. Mostly. It logs attempts made if the username looks like an email address, like [email protected]. But if I try to log in with a random username, like "asdfl" then no logging is made. I'm assuming this is expected behavior?

[mprasil]

Yeah, if it's not valid email, it's rejected on the client side, so there's no request reaching server. However if you called the login API endpoint manually you would see the log.

[itr6]

Gotcha, Now I have the syslog working and made it a little more human friendly with the 'tag' option. Now I just need to create the fail2ban filter.

[itr6]

GOT IT! OK, here we go.
I added the following to my docker-compose file:

    logging:
      driver: "syslog"
      options:
        tag: "Bitwarden"

Which made my syslog at /var/log/syslog report this:
Nov 16 14:05:54 $SERVER Bitwarden[979]: ERROR: Username or password is incorrect. Try again. IP: XX.XX.XX.XX. Username: [email protected].
I wanted to make sure only attempts at Bitwarden were logged against this filter so I struggled but finally found a way for it to include the 'Bitwarden' tag since the '[979]' was throwing it off.
So I created my filter

[INCLUDES]
before = common.conf

[Definition]
failregex = ^%(__prefix_line)s.*Bitwarden.* ERROR: Username or password is incorrect. Try again. IP: <HOST>.*$
ignoreregex =

Then comes the jail

[bitwarden]
enabled = true
port = 80,443,8081
filter = bitwarden
action = iptables-allports[name=bitwarden]
logpath = /var/log/syslog
maxretry = 3
bantime = 14400
findtime = 14400

And it WORKS! This is my first time creating anything like this so if anyone has any suggestions or improvements, PLEASE feel free to say so.

[mprasil]

Maybe a small suggestion there to improve the regex:

failregex = ^%(__prefix_line)s.*Bitwarden.* ERROR: Username or password is incorrect. Try again. IP: <HOST>\. Username:.*$

Note the extra characters after <HOST> that make sure that .* is not eating part of the IP. (Haven't tested this with fail2ban, but the regex should be OK.

Also maybe create a PR to add this documentation if you feel like? (something like a proxy with whole documentation in extra file and then link to it from README.md)

[Biepa]

Hey,

I tried to create the filter by your steps.

• Configured logging in syslog.
• Created the bitwarden.conf and pasted the part (here I have to change common.conf to "paths-debian.conf" for example, right?)
• pasted the second part in my jail config

When i try to start I get
Failed during configuration: Bad value substitution: option 'failregex' in section 'Definition' contains an interpolation key '__prefix_line' which is not a valid option name.
Where am I wrong?

[itr6]

@Biepa

Sorry I just saw this. Do you still need help?

[Biepa]

@itr6 No problem. Yes, please :)

[itr6]

@Biepa
From the error message you pasted. It looks like it could be a syntax error in your filter? Make sure to copy and paste this exactly into your filter.

failregex = ^%(__prefix_line)s.*Bitwarden.* ERROR: Username or password is incorrect. Try again. IP: <HOST>\. Username:.*$

[Biepa]

@itr6 The failregex was right.
Made a little mistake in the "includes" area above. :man_facepalming:

Started now, thanks for the help.

[itr6]

Aw, yea. Make sure those brackets are there. Glad it is working for you!

[mprasil]

I think we can close this now that #274 was merged. Thanks for that @itr6

[dohlin]

@mprasil I know this issue is closed, but I wanted to leave a comment here as it's the most relevant thread. I'm wanting to get fail2ban setup and working on my Bitwarden instance, but this is my first time heavily messing with docker/docker-compose so I'm a bit of a noob. I'm running Ubuntu 18.04, and I cannot find my docker-compose.yml file anywhere...is this something we have to manually create for the first time? I've read that the default path is ./docker-compose.yml.

And if this is indeed something that needs to be created from scratch, then would you be willing to share the whole relevant structure of your docker-compose.yml file? I tried making one from scratch, but since I really don't fully understand what I'm doing yet I just encountered error after error. Thank you!

[mprasil]

Hi @dohlin, I don't use docker compose in my setup, so I don't have tested configuration to share, perhaps @itr6 can share his setup? But yeah, you need to create the file from scratch. Something like this should work:

version: '3'
services:
  bitwarden_web:
    image: "mprasil/bitwarden"
    restart: "always"
    volumes:
      - "/local_bw_data:/data"
    logging:
      driver: "syslog"
      options:
        tag: "$TAG"

Note that you don't necessarily need to use docker compose to setup fail2ban. If you already run bitwarden_rs via docker directly, you can just add --log-driver syslog --log-opt tag=$TAG to your usual docker run setup to achieve the same.

Hope that helps.

[dohlin]

@mprasil Thank you, that's very helpful and explains a lot. I really appreciate the easy explanation. It would appear that after digging into it a little more, since I'm serving this behind an Apache reverse proxy, that I would have to do a bit more work to get this to work as whenever there's a failed login Bitwarden sees it as coming from my reverse proxy server IP address. I might do some further digging down the road but for now I think my best best is a strong master password paired with OTP 2FA. Thanks again!!

[mprasil]

I think bitwarden_rs should honor the X-Forwarded-For header, so maybe you just need to pass that one to fix your issue.

[dohlin]

@mprasil Interesting...should that be as simple as adding a line that says

RemoteIPHeader X-Forwarded-For

to the Apache virtualhost config file that controls the bitwarden site? I tried that and it didn't work. I also found

SetEnvIf X-Forwarded-For "^......." forwarded

elsewhere online, tried that and it didn't work either. I've enabled mod_remoteip in Apache and it doesn't complain about the config or anything so I think I've got it all set up right, but when I fail a login I still see the 192.168.1.x IP that is my reverse proxy IP.

EDIT: I tried the instructions here and can confirm that in the last step when viewing the log my public IP address shows up as expected...unfortunately failed logins on Bitwarden still show as being from the private IP of my reverse proxy server. Also looked through everything else I can find online, from adding different config options to apache.conf and the individual virtual host conf, to modules such as rpaf and no matter what I do the results are still the same. :(

[Algebro7]

Note that you don't necessarily need to use docker compose to setup fail2ban. If you already run bitwarden_rs via docker directly, you can just add --log-driver syslog --log-opt tag=$TAG to your usual docker run setup to achieve the same.

Should we add this to the README for people who don't use docker compose? I am not very experienced with docker and I was very confused following the instructions for fail2ban because it a) assumed I was using docker-compose without providing instructions for how to use docker-compose and b) didn't mention you could do it without docker-compose

[mprasil]

Good point! There are some changes around logging in the latest beta. So we will probably need to revisit this whole setup anyways.

[itr6]

@dohlin Sorry for the confusion. I'm glad @mprasil was able to help you.
@Algebro7 I intend to re-write this to help others that aren't using compose, just haven't had time since the holidays are upon us.

[Algebro7]

No worries @itr6. I think adding @mprasil's advice would be enough, although if using compose is undeniably better it would probably be worth adding a short section laying out how to do that.

[itr6]

@Algebro7 I prefer compose so I can navigate to my folder where my yaml file is (IE /opt/bitwarden) and run:
docker-compose up -d and it's running. I dont have to remember 10 variables and which port goes to which port and all the volume maps. That's just my $.02

[jeslinmx]

Hi there! Sorry to add on to an already closed issue but I'm not sure if this warrants a new one or if I'm only facing this due to some weird setup.

My log line differs slightly from this issue (and in the documentation):

Dec 14 14:25:08 ubuntu-s-1vcpu-1gb-eur1-01 bitwarden[773]: [2018-12-14][14:25:08][bitwarden_rs::api::identity][ERROR] Username or password is incorrect. Try again. IP: 23.154.210.87. Username: [email protected].

No problem, I'll try my hand at writing an appropriate failregex to account for the additional bits:

[INCLUDES]
before = common.conf

[Definition]

_daemon = bitwarden
failregex = ^%(__prefix_line)s.*\[ERROR\] Username or password is incorrect\. Try again\. IP:<HOST>\. Username: .*\s*$
ignoreregex =

No dice. I get 0 matches:

root@ubuntu-s-1vcpu-1gb-eur1-01:/etc/fail2ban# fail2ban-regex -v /var/log/syslog /etc/fail2ban/filter.d/bitwarden.conf

Running tests
=============

Use   failregex file : /etc/fail2ban/filter.d/bitwarden.conf
Use         log file : /var/log/syslog


Results
=======

Failregex: 0 total
|-  #) [# of hits] regular expression
|   1) [0] ^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\] )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?bitwarden(?:\(\S+\))?[\]\)]?:?|[\[\(]?bitwarden(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID \d+ \S+\])?\s*.*\[ERROR\] Username or password is incorrect\. Try again\. IP:<HOST>\. Username: .*\s*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [8252] MONTH Day Hour:Minute:Second
|  [0] WEEKDAY MONTH Day Hour:Minute:Second[.subsecond] Year
|  [0] WEEKDAY MONTH Day Hour:Minute:Second Year
|  [0] WEEKDAY MONTH Day Hour:Minute:Second
|  [0] Year/Month/Day Hour:Minute:Second
|  [0] Day/Month/Year Hour:Minute:Second
|  [0] Day/Month/Year2 Hour:Minute:Second
|  [0] Day/MONTH/Year:Hour:Minute:Second
|  [0] Month/Day/Year:Hour:Minute:Second
|  [0] Year-Month-Day Hour:Minute:Second[,subsecond]
|  [0] Year-Month-Day Hour:Minute:Second
|  [0] Year.Month.Day Hour:Minute:Second
|  [0] Day-MONTH-Year Hour:Minute:Second[.Millisecond]
|  [0] Day-Month-Year Hour:Minute:Second
|  [0] Month-Day-Year Hour:Minute:Second[.Millisecond]
|  [0] TAI64N
|  [0] Epoch
|  [0] ISO 8601
|  [0] Hour:Minute:Second
|  [0] <Month/Day/Year@Hour:Minute:Second>
|  [0] YearMonthDay Hour:Minute:Second
|  [0] Month-Day-Year Hour:Minute:Second
`-

Lines: 8252 lines, 0 ignored, 0 matched, 8252 missed
Missed line(s): too many to print.  Use --print-all-missed to print all 8252 lines

Would be really thankful for another pair of eyes on my regex to see what I missed!

[itr6]

@jeslinmx Did you try the failregex exactly as it is from the readme documentation?

[itr6]

@jeslinmx Scratch the above. Try this:

failregex = ^%(__prefix_line)s.*bitwarden.* Username or password is incorrect. Try again. IP: <HOST>\. Username:.*$

[mprasil]

Note, that if you use latest docker image (or build from master) the logformat changed a bit. The release notes for 1.5.0 (not out yet) warn about that change.

[itr6]

@mprasil Thanks for the heads up. I keep an eye out for the new version and make changes to the setup.

[jeslinmx]

@itr6 oh goodness. The reason mine was failing was because I left out the space after IP:. Yesterday was a long day.

Anyway, I have generalized it to:

[Definition]
_daemon = bitwarden
failregex = ^%(__prefix_line)s.*Username or password is incorrect\. Try again\. IP: <HOST>\. Username:.*$

There is no need for .*bitwarden.* as %(_prefix_line)s already captures Dec 14 14:25:08 ubuntu-s-1vcpu-1gb-eur1-01 bitwarden[773]. Specifically, %(_prefix_line)s expands out into:

^\s*(<[^.]+\.[^.]+>)?\s*(?:\S+ )?(?:kernel: \[ *\d+\.\d+\] )?(?:@vserver_\S+ )?(?:(?:\[\d+\])?:\s+[\[\(]?bitwarden(?:\(\S+\))?[\]\)]?:?|[\[\(]?bitwarden(?:\(\S+\))?[\]\)]?:?(?:\[\d+\])?:?)?\s(?:\[ID \d+ \S+\])?\s*

where the bitwarden in the regex is substituted in from the value of _daemon. Hence, assuming _daemon is set to match $TAG in docker's logging options, the above regex should suffice.

[mprasil]

Sounds like you should submit a PR to update the docs @jeslinmx if you can.

[lordlzh]

GOT IT! OK, here we go.
I added the following to my docker-compose file:

    logging:
      driver: "syslog"
      options:
        tag: "Bitwarden"

Which made my syslog at /var/log/syslog report this:
Nov 16 14:05:54 $SERVER Bitwarden[979]: ERROR: Username or password is incorrect. Try again. IP: XX.XX.XX.XX. Username: [email protected].
I wanted to make sure only attempts at Bitwarden were logged against this filter so I struggled but finally found a way for it to include the 'Bitwarden' tag since the '[979]' was throwing it off.
So I created my filter

[INCLUDES]
before = common.conf

[Definition]
failregex = ^%(__prefix_line)s.*Bitwarden.* ERROR: Username or password is incorrect. Try again. IP: <HOST>.*$
ignoreregex =

Then comes the jail

[bitwarden]
enabled = true
port = 80,443,8081
filter = bitwarden
action = iptables-allports[name=bitwarden]
logpath = /var/log/syslog
maxretry = 3
bantime = 14400
findtime = 14400

And it WORKS! This is my first time creating anything like this so if anyone has any suggestions or improvements, PLEASE feel free to say so.

Hello
I can't find the "syslog" file in /var/log/
I don't know why.
I do it fallowing your steps.
System: Centos 7

[lordlzh]

GOT IT! OK, here we go.
I added the following to my docker-compose file:

    logging:
      driver: "syslog"
      options:
        tag: "Bitwarden"

Which made my syslog at /var/log/syslog report this:
Nov 16 14:05:54 $SERVER Bitwarden[979]: ERROR: Username or password is incorrect. Try again. IP: XX.XX.XX.XX. Username: [email protected].
I wanted to make sure only attempts at Bitwarden were logged against this filter so I struggled but finally found a way for it to include the 'Bitwarden' tag since the '[979]' was throwing it off.
So I created my filter

[INCLUDES]
before = common.conf

[Definition]
failregex = ^%(__prefix_line)s.*Bitwarden.* ERROR: Username or password is incorrect. Try again. IP: <HOST>.*$
ignoreregex =

Then comes the jail

[bitwarden]
enabled = true
port = 80,443,8081
filter = bitwarden
action = iptables-allports[name=bitwarden]
logpath = /var/log/syslog
maxretry = 3
bantime = 14400
findtime = 14400

And it WORKS! This is my first time creating anything like this so if anyone has any suggestions or improvements, PLEASE feel free to say so.

I find it ，wow，thanks