Hi
I have been using my nextcloud VM for some time now and suddenly Bitwarden stopped working (503 Service Unavailable). So I went for checking the docker containers and boom, nothing there. That happened before but back then, a reboot solved it.
Now there are no bitwarden containers, even after a reboot of the VM.
root@files:/# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ffab21dc24e containrrr/watchtower "/watchtower --clean…" 4 weeks ago Up 2 days 8080/tcp watchtower
d648be10c8fe ark74/nc_fts "/tini -- /usr/local…" 4 weeks ago Up 2 days 127.0.0.1:9200->9200/tcp, 127.0.0.1:9300->9300/tcp fts_esror
not sure, whether the /dev/loop* stuff is supposed to be full according to df -h and whether that has anything to do with bitwarden containers not being there anymore.
root@files:/home/ncadmin# df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 797M 2.3M 794M 1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 35G 16G 18G 47% /
tmpfs 3.9G 8.0K 3.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda2 976M 197M 713M 22% /boot
/dev/loop0 48M 48M 0 100% /snap/certbot/500
/dev/loop1 53M 53M 0 100% /snap/certbot/579
/dev/loop3 55M 55M 0 100% /snap/core18/1880
/dev/loop2 56M 56M 0 100% /snap/core18/1885
/dev/loop4 61M 61M 0 100% /snap/core20/634
/dev/loop5 71M 71M 0 100% /snap/lxd/16889
/dev/loop6 30M 30M 0 100% /snap/snapd/8542
/dev/loop7 30M 30M 0 100% /snap/snapd/8790
/dev/loop8 71M 71M 0 100% /snap/lxd/16926
ncdata 985G 17G 968G 2% /mnt/ncdata
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/099c110df378876490f3e99135487d0b9b61ebedc3b87845767b15b17163f9fd/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/5b73477d79a51126c5da21936c66f031b2dc5ccc96f422fe7433e3c41b7eca36/merged
tmpfs 797M 0 797M 0% /run/user/1000
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/d00d0b020e722f4c799efd92bf20a393a961b4f425a31d0cdbc095ad923d8422/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/bd9d3f2149c25b02d28abc1ebfa462c939965182cf30b6b0873645db5aedb437/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/fbbbbd9543d04cfc35ced209359736d0af480dae55efd3282f5f2e920582e647/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/e70002097715720565641d6151b25e439cbd867f6fdbb5897a3a8eebe7ac6e19/merged
shm 64M 0 64M 0% /var/lib/docker/containers/8d2b94bbfc1ce845c11b85e9b8a2f23adcdb7353ec2ab3812baa836e23ab680f/mounts/shm
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/a045d8f3646943ec4cfc89a71526b988845b3c463a9a1e5398a013bd969f599c/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/a88e7f1816ce7dc42addc16653e8c4dcab9c53b76205d9e70d77911c7656725c/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/a4663dca35b98c9231ecac7af7d44d6eb834b08766f109936b9e7978f1a2b05a/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/7999b98796201e34fd3bc0f9afe560e56d15dffa0f58a66dc4a87c9d4890c720/merged
shm 64M 0 64M 0% /var/lib/docker/containers/348ba070df22020678549a272908392623832d3ffbe2ca135e57eb20d2a30363/mounts/shm
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/6c16dd031bf017c0f216e0aff54283fb6ea52f7ca51019458c510c0098bf43e2/merged
shm 64M 0 64M 0% /var/lib/docker/containers/1852a9f4324a40ebe620a1bbf2a8b13891495487c922c4372f123c88f5c413a2/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/49e711f7978a77fe2bc443e000949a372da5dfe21030e2f5b3a9c7f1fc96ded3/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/d23def6762f7ed80e722e09c6d2cbf53242b9a9fe66780de8805b959b1bd0a7c/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/942cfbb99f3babc92306806037d393048ca8ab1bf57dd7caed5a710d2a5c9934/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/fcbe4948474ef3e25f71dedcc35f542d246ae7be2eae2e8cc13df87a767bb944/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/a1fd8035c19b323afe422c8fd8db241fc2e03eb47731c09a1f92b340f00d73d5/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/e9cc81aa876ea848f00fe68aa3382fb61d897a3aa4b3c473f886e65aaa6723b2/mounts/shm
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/932dfa725567cbbfd678175948ff46edaec8b74c6ae4dfa5c6694e091f02da6f/merged
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/b8581bc46c5553085251d7fd2ae1c861ea9e160377cb14d9ceab300eb203cb32/merged
shm 64M 0 64M 0% /var/lib/docker/containers/b68a7504e693b3bb19114fb6a839ce9f1a713d73d7b5d13b58c1b1209b403120/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/aae2f608a3580a6fca38d4c6922ac8d26953c0d42c4478511867cd42047072dc/mounts/shm
overlay 35G 16G 18G 47% /var/lib/docker/overlay2/24de9b585e817f5c00707fa19a4f448d288a6a857c946de2d77bf4d26ce7c0d5/merged
shm 64M 0 64M 0% /var/lib/docker/containers/18e8d8e4a94fc8b5d074ec0dbd89da60746d4bd3653e7fc0df77582388abfb83/mounts/shm
The VM has not been altered in any way, please advise how I can get Bitwarden back working.
BTW, /home/bitwarden/bwdata/mssql/data is empty, however /home/bitwarden/bwdata/mssql/backup has files in it:
root@files:/home/bitwarden/bwdata/mssql/backups# ls -lha
total 195M
drwxrwxr-x 2 bitwarden bitwarden 4.0K Sep 12 02:00 .
drwxrwxr-x 4 bitwarden bitwarden 4.0K Aug 15 18:26 ..
-rw-r----- 1 bitwarden bitwarden 6.7M Aug 16 02:00 vault_FULL_20200816_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.7M Aug 17 02:00 vault_FULL_20200817_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.7M Aug 18 02:00 vault_FULL_20200818_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.8M Aug 19 02:00 vault_FULL_20200819_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.8M Aug 20 02:00 vault_FULL_20200820_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.8M Aug 21 02:00 vault_FULL_20200821_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.8M Aug 22 02:00 vault_FULL_20200822_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.9M Aug 23 02:00 vault_FULL_20200823_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.9M Aug 24 02:00 vault_FULL_20200824_000000.BAK
-rw-r----- 1 bitwarden bitwarden 6.9M Aug 25 02:00 vault_FULL_20200825_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.0M Aug 26 02:00 vault_FULL_20200826_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Aug 27 02:00 vault_FULL_20200827_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Aug 28 02:00 vault_FULL_20200828_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Aug 29 02:00 vault_FULL_20200829_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Aug 30 02:00 vault_FULL_20200830_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Aug 31 02:00 vault_FULL_20200831_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 1 02:00 vault_FULL_20200901_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 2 02:00 vault_FULL_20200902_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 3 02:00 vault_FULL_20200903_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 4 02:00 vault_FULL_20200904_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 5 02:00 vault_FULL_20200905_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 6 02:00 vault_FULL_20200906_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 7 02:00 vault_FULL_20200907_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 8 02:00 vault_FULL_20200908_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 9 02:00 vault_FULL_20200909_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 10 02:00 vault_FULL_20200910_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.1M Sep 11 02:00 vault_FULL_20200911_000000.BAK
-rw-r----- 1 bitwarden bitwarden 7.2M Sep 12 02:00 vault_FULL_20200912_000000.BAK
mamama1
All 22 comments
FTR I did bitwarden.sh start and it magically seems to work again. That's not very trustworthy... This fragile contruction is supposed to be my holy grail of passwords...?!?
root@files:/home/bitwarden# ./bitwarden.sh start
_ _ _ _
| |__ (_) |___ ____ _ _ __ __| | ___ _ __
| '_ \| | __\ \ /\ / / _` | '__/ _` |/ _ \ '_ \
| |_) | | |_ \ V V / (_| | | | (_| | __/ | | |
|_.__/|_|\__| \_/\_/ \__,_|_| \__,_|\___|_| |_|
Open source password management solutions
Copyright 2015-2020, 8bit Solutions LLC
https://bitwarden.com, https://github.com/bitwarden
===================================================
Docker version 19.03.12, build 48a66213fe
docker-compose version 1.25.0, build unknown
Stopping bitwarden-nginx ... done
Stopping bitwarden-portal ... done
Stopping bitwarden-admin ... done
Stopping bitwarden-notifications ... done
Stopping bitwarden-web ... done
Stopping bitwarden-api ... done
Stopping bitwarden-sso ... done
Stopping bitwarden-identity ... done
Stopping bitwarden-mssql ... done
Stopping bitwarden-attachments ... done
Stopping bitwarden-icons ... done
Stopping bitwarden-events ... done
Removing bitwarden-nginx ... done
Removing bitwarden-portal ... done
Removing bitwarden-admin ... done
Removing bitwarden-notifications ... done
Removing bitwarden-web ... done
Removing bitwarden-api ... done
Removing bitwarden-sso ... done
Removing bitwarden-identity ... done
Removing bitwarden-mssql ... done
Removing bitwarden-attachments ... done
Removing bitwarden-icons ... done
Removing bitwarden-events ... done
Removing network docker_default
Removing network docker_public
Pulling mssql ... done
Pulling web ... done
Pulling attachments ... done
Pulling api ... done
Pulling identity ... done
Pulling sso ... done
Pulling admin ... done
Pulling portal ... done
Pulling icons ... done
Pulling notifications ... done
Pulling events ... done
Pulling nginx ... done
Creating network "docker_default" with the default driver
Creating network "docker_public" with the default driver
Creating bitwarden-identity ... done
Creating bitwarden-icons ... done
Creating bitwarden-sso ... done
Creating bitwarden-notifications ... done
Creating bitwarden-events ... done
Creating bitwarden-web ... done
Creating bitwarden-mssql ... done
Creating bitwarden-attachments ... done
Creating bitwarden-api ... done
Creating bitwarden-admin ... done
Creating bitwarden-portal ... done
Creating bitwarden-nginx ... done
1.37.0: Pulling from bitwarden/setup
Digest: sha256:d53aa2bc97c9bfdb5609cf9ec7cbee8cfedac536a7b101e039db2e0d733c60e0
Status: Image is up to date for bitwarden/setup:1.37.0
docker.io/bitwarden/setup:1.37.0
Bitwarden is up and running!
===================================================
visit https://pw.ne-mail.net
to update, run `./bitwarden.sh updateself` and then `./bitwarden.sh update`
Can you post the output of: docker logs watchtower?
Thanks!
enoch85
on 15 Sep 2020
Hi,
sure:
root@files:/home/ncadmin# docker logs watchtower
time="2020-08-15T16:50:42Z" level=info msg="Starting Watchtower and scheduling first run: 2020-08-15 17:50:42 +0000 UTC m=+3601.139143957"
time="2020-08-15T16:51:39Z" level=info msg="Waiting for running update to be finished..."
time="2020-08-15T16:52:29Z" level=info msg="Starting Watchtower and scheduling first run: 2020-08-15 17:52:29 +0000 UTC m=+3605.797470197"
time="2020-08-15T20:40:08Z" level=info msg="Waiting for running update to be finished..."
time="2020-08-15T20:40:48Z" level=info msg="Starting Watchtower and scheduling first run: 2020-08-15 21:40:48 +0000 UTC m=+3600.606668103"
time="2020-08-15T21:13:50Z" level=info msg="Waiting for running update to be finished..."
time="2020-08-15T21:14:30Z" level=info msg="Starting Watchtower and scheduling first run: 2020-08-15 22:14:30 +0000 UTC m=+3600.594082839"
time="2020-08-15T21:18:22Z" level=info msg="Waiting for running update to be finished..."
time="2020-08-15T21:19:00Z" level=info msg="Starting Watchtower and scheduling first run: 2020-08-15 22:19:00 +0000 UTC m=+3601.125400803"
time="2020-08-15T22:30:00Z" level=info msg="Waiting for running update to be finished..."
time="2020-08-15T22:30:40Z" level=info msg="Starting Watchtower and scheduling first run: 2020-08-15 23:30:40 +0000 UTC m=+3601.182699474"
time="2020-08-17T17:40:35Z" level=info msg="Waiting for running update to be finished..."
time="2020-08-17T17:41:12Z" level=info msg="Starting Watchtower and scheduling first run: 2020-08-17 18:41:12 +0000 UTC m=+3600.774780731"
time="2020-08-18T14:42:10Z" level=info msg="Unable to update container /bitwarden-api. Proceeding to next."
time="2020-08-18T14:43:02Z" level=info msg="Unable to update container /bitwarden-mssql. Proceeding to next."
time="2020-08-18T14:43:23Z" level=info msg="Unable to update container /bitwarden-notifications. Proceeding to next."
time="2020-08-18T15:41:27Z" level=info msg="Unable to update container /bitwarden-nginx. Proceeding to next."
time="2020-08-18T15:41:27Z" level=info msg="Unable to update container /bitwarden-admin. Proceeding to next."
time="2020-08-30T01:42:28Z" level=info msg="Unable to update container /bitwarden-api. Proceeding to next."
time="2020-08-30T01:43:14Z" level=info msg="Unable to update container /bitwarden-mssql. Proceeding to next."
time="2020-08-30T01:43:58Z" level=info msg="Unable to update container /bitwarden-web. Proceeding to next."
time="2020-08-30T11:41:22Z" level=info msg="Unable to update container /bitwarden-nginx. Proceeding to next."
time="2020-08-30T11:42:26Z" level=info msg="Unable to update container /bitwarden-api. Proceeding to next."
time="2020-08-30T11:42:56Z" level=info msg="Unable to update container /bitwarden-events. Proceeding to next."
time="2020-08-30T11:43:17Z" level=info msg="Unable to update container /bitwarden-mssql. Proceeding to next."
time="2020-08-30T11:43:28Z" level=info msg="Unable to update container /bitwarden-notifications. Proceeding to next."
time="2020-08-30T11:44:04Z" level=info msg="Unable to update container /bitwarden-identity. Proceeding to next."
time="2020-08-30T11:44:31Z" level=info msg="Unable to update container /bitwarden-icons. Proceeding to next."
time="2020-08-30T11:45:18Z" level=info msg="Unable to update container /fts_esror. Proceeding to next."
time="2020-09-05T20:32:40Z" level=info msg="Waiting for running update to be finished..."
time="2020-09-05T20:33:17Z" level=info msg="Starting Watchtower and scheduling first run: 2020-09-05 21:33:17 +0000 UTC m=+3601.127151641"
time="2020-09-11T15:23:27Z" level=info msg="Waiting for running update to be finished..."
time="2020-09-11T17:37:14Z" level=info msg="Starting Watchtower and scheduling first run: 2020-09-11 18:37:14 +0000 UTC m=+3600.622623014"
time="2020-09-11T20:39:41Z" level=info msg="Waiting for running update to be finished..."
time="2020-09-11T22:23:19Z" level=info msg="Starting Watchtower and scheduling first run: 2020-09-11 23:23:19 +0000 UTC m=+3601.050064849"
time="2020-09-14T19:39:52Z" level=info msg="Waiting for running update to be finished..."
time="2020-09-14T19:40:17Z" level=info msg="Starting Watchtower and scheduling first run: 2020-09-14 20:40:17 +0000 UTC m=+3600.818904536"
OK, so probably watchtower is to blame for this. Not sure though as it always worked for me.
I will add a fix which excludes it from updating Bitwarden anyway. Better safe than sorry.
enoch85
on 15 Sep 2020
Maybe you could try this?
The original command for starting watchtower is this:
docker run -d --restart=unless-stopped --name watchtower -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower "$CmdDocker".
This is the actual code.
I don't want to merge something that is broken and I'm not 100% sure on how to exclude only Bitwarden.
enoch85
on 15 Sep 2020
No response yet @mamama1, Keen to help out here? :)
enoch85
on 19 Sep 2020
btw @mamama1 have you seen this?
https://github.com/nextcloud/vm/blob/9361b3a2970bc19a2141a21b24f4a5b36073ade1/apps/tmbitwarden.sh#L325-L329
szaimen
on 20 Sep 2020
@szaimen that's unrelated.
What we need to figure out here is to disable Watchtower for Bitwarden, and allow everything else.
Preferably also disable Bitwarden for current users.
enoch85
on 20 Sep 2020
Note to self: https://github.com/containrrr/watchtower/issues/169
enoch85
on 20 Sep 2020
We might just run watchtower once when the update script is run:
docker run --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
containrrr/watchtower \
--run-once \
collabora onlyoffice fts
Preferably with a check if those dockers are installed.
I'll prepare a PR soon.
enoch85
on 20 Sep 2020
While we're at it, maybe we can add notify_admin_gui as well. :thinking:
enoch85
on 20 Sep 2020
docker run --rm
-v /var/run/docker.sock:/var/run/docker.sock
containrrr/watchtower
--run-once
collabora onlyoffice fts
Can we add bitwarden_rs to this command, too?
https://github.com/nextcloud/vm/blob/9361b3a2970bc19a2141a21b24f4a5b36073ade1/apps/bitwarden-rs.sh#L196
szaimen
on 20 Sep 2020
Is Bitwarden RS just one container?
enoch85
on 20 Sep 2020
Is Bitwarden RS just one container?
Yes
szaimen
on 20 Sep 2020
@szaimen Could you please list the names of all dockers we are using in the VM?
Prepping a PR now.
enoch85
on 20 Sep 2020
I think it should be:
code, $fts_es_name, onlyoffice, bitwarden_rs
szaimen
on 20 Sep 2020
@mamama1 This is now fixed. Please run the update script:
sudo bash /var/scripts/update.sh
Report back if there's still issues.
enoch85
on 20 Sep 2020
Hi
sorry for not coming back to you earlier - busy times...
bitwarden seems to be gone for good now. my appliances assumably did the scheduled update on sunday(?).
root@files:/home/bitwarden# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d648be10c8fe ark74/nc_fts "/tini -- /usr/local…" 5 weeks ago Up 16 minutes 127.0.0.1:9200->9200/tcp, 127.0.0.1:9300->9300/tcp fts_esror
root@files:/home/bitwarden# ./bitwarden.sh start
_ _ _ _
| |__ (_) |___ ____ _ _ __ __| | ___ _ __
| '_ \| | __\ \ /\ / / _` | '__/ _` |/ _ \ '_ \
| |_) | | |_ \ V V / (_| | | | (_| | __/ | | |
|_.__/|_|\__| \_/\_/ \__,_|_| \__,_|\___|_| |_|
Open source password management solutions
Copyright 2015-2020, 8bit Solutions LLC
https://bitwarden.com, https://github.com/bitwarden
===================================================
Docker version 19.03.13, build 4484c46d9d
docker-compose version 1.25.0, build unknown
ERROR: 2 matches found based on name: network docker_default is ambiguous
at first the containers were not showing again when i typed docker ps so i did bitwarden.sh start again and then it threw errors at me. then i ran the update script and now I'm where I am.
sql backups are there, under /home/bitwarden/bwdata/mssql/backups.
mamama1
on 22 Sep 2020
I got an email from another user with the same issue as you, he ran the update and everything as fine.
Try to remove the network and start with ./bitwarden.sh start again.
enoch85
on 22 Sep 2020
jeez, that was a trip.
bitwarden started when I first docker network rm'd the duplicate networks but then my db was gone/empty. I recall the update script telling me that it had deleted the bitwarden mssql docker volume (yay), so I had to restore it from the last backup file:
docker exec -it bitwarden-mssql /bin/bash
/opt/mssql-tools/bin/sqlcmd -S localhost -U sa -P ${SA_PASSWORD}
ALTER DATABASE vault
SET OFFLINE WITH ROLLBACK IMMEDIATE
GO
RESTORE DATABASE vault FROM DISK = '/etc/bitwarden/mssql/backups/vault_FULL_20200919_000000.BAK' WITH REPLACE
GO
ALTER DATABASE vault
SET ONLINE
Thanks anyway for your help.
So, if I understood correctly, the regular weekly update script now won't touch bitwarden anymore? How is bitwarden supposed to be updated now to get patches and security fixes? Just by running /home/bitwarden/bitwarden.sh update?
mamama1
on 22 Sep 2020
Great that it works again!
So, if I understood correctly, the regular weekly update script now won't touch bitwarden anymore? How is bitwarden supposed to be updated now to get patches and security fixes? Just by running /home/bitwarden/bitwarden.sh update?
Have you seen this?
https://github.com/nextcloud/vm/blob/master/nextcloud_update.sh#L536-L569
szaimen
on 22 Sep 2020
nope didn't see that. cool bananas, so let's hope bitwarden will survive the next update cycle this time xD
thanks!
mamama1
on 22 Sep 2020
Related issues
enoch85
·
8Comments
rakekniven
·
7Comments
lantrix
·
6Comments
smartens
·
6Comments
thejohnha
·
8Comments