Vm: Nextcloud VM 11.0.3 - Collabora 2.1 docker image fills all disk space

[aaaskew]

Steps to reproduce

1. Clean install Nextcloud 11.0.3 VM on Debian Wheezy 64bit VirtualBox 5.0.40 host.
2. Use NAT but redirect port 443 on the modem as well as VirtualBox host.
3. Enable all options during installation. Install completes (with some errors for passman and search).
4. Reboot.
5. Interface comes up and configure email, search and enable passman.
6. Try to edit example text document. Access denied.
7. Logout, wait, try to login. Login fails.

Expected behaviour

Collabora should open.

Actual behaviour

Disk full of files created by Collabora 2.1 docker image.

du -sh /var/lib/docker/aufs/diff/e1062c24c31a7fdb96b04bdfa9240295d7ba022cc63c4e563f0f0a4ad0c96cde/.wh..wh.plnk
14G .wh..wh.plnk

Docker logs showing repeated Docker startup and crash:

Failed to fork child processes.
wsd-00025-00025 00:43:37.982447 [ loolwsd ] FTL Failed to fork child processes.| wsd/LOOLWSD.cpp:2551
Failed to fork child processes.
wsd-00025-00025 00:43:37.983012 [ loolwsd ] WRN Waking up dead poll thread [admin], started: false, finished: false| ./net/Socket.hpp:445
-00025 00:43:37.983672 [ loolwsd ] WRN Waking up dead poll thread [accept_poll], started: false, finished: false| ./net/Socket.hpp:445

Same error reported at: https://help.nextcloud.com/t/restarting-docker-results-in-failed-to-fork-child-processes/12209

New Collabora image only 4 days old. Maybe this is broken? Can the VM use an older image until this is fixed?

Server configuration

Server version: (see your admin page)

Nextcloud 11.0.3 (stable)
PHP 7.0.15
mysql 5.7.18

VM version (master / released version)

Nextcloud 11.0.3 VM

Updated from an older installed version or fresh install:

Fresh install

Network

Do you use DHCP?

Turned off by Nexcloud VM intaller?

Is port 80 and/or 443 open?

Port 443 open

Logs / Screenshots

Log file (/var/ncdata/nextcloud.log)

{"reqId":"kFEUTdDn8oRBk0s0kUhR","remoteAddr":"10.0.2.3","app":"PHP","message":"vsprintf(): Too few arguments at \/var\/www\/nextcloud\/lib\/private\/legacy\/l10n\/string.php#72","level":3,"time":"2017-05-07T21:48:28+00:00","method":"GET","url":"\/apps\/passman\/api\/v2\/language?lang=en","user":"ncadmin","version":"11.0.3.2"}
{"reqId":"kFEUTdDn8oRBk0s0kUhR","remoteAddr":"10.0.2.3","app":"PHP","message":"Method OC_L10N_String::__toString() must return a string value at \/var\/www\/nextcloud\/lib\/private\/L10N\/L10N.php#85","level":3,"time":"2017-05-07T21:48:28+00:00","method":"GET","url":"\/apps\/passman\/api\/v2\/language?lang=en","user":"ncadmin","version":"11.0.3.2"}
{"reqId":"EhAIEsbIafK8MNJ2idTl","remoteAddr":"10.0.2.3","app":"index","message":"Exception: {\"Exception\":\"GuzzleHttp\\\\Exception\\\\ServerException\",\"Message\":\"Server error response [url] https:\\\/\\\/office.home.aaaskew.me.uk\\\/hosting\\\/discovery [status code] 502 [reason phrase] Proxy Error\",\"Code\":502,\"Trace\":\"#0 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/Subscriber\\\/HttpError.php(32): GuzzleHttp\\\\Exception\\\\RequestException::create(Object(GuzzleHttp\\\\Message\\\\Request), Object(GuzzleHttp\\\\Message\\\\Response))\\n#1 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/Event\\\/Emitter.php(108): GuzzleHttp\\\\Subscriber\\\\HttpError->onComplete(Object(GuzzleHttp\\\\Event\\\\CompleteEvent), 'complete')\\n#2 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/RequestFsm.php(91): GuzzleHttp\\\\Event\\\\Emitter->emit('complete', Object(GuzzleHttp\\\\Event\\\\CompleteEvent))\\n#3 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/RequestFsm.php(132): GuzzleHttp\\\\RequestFsm->__invoke(Object(GuzzleHttp\\\\Transaction))\\n#4 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/react\\\/promise\\\/src\\\/FulfilledPromise.php(25): GuzzleHttp\\\\RequestFsm->GuzzleHttp\\\\{closure}(Array)\\n#5 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/ringphp\\\/src\\\/Future\\\/CompletedFutureValue.php(55): React\\\\Promise\\\\FulfilledPromise->then(Object(Closure), NULL, NULL)\\n#6 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/Message\\\/FutureResponse.php(43): GuzzleHttp\\\\Ring\\\\Future\\\\CompletedFutureValue->then(Object(Closure), NULL, NULL)\\n#7 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/RequestFsm.php(134): GuzzleHttp\\\\Message\\\\FutureResponse::proxy(Object(GuzzleHttp\\\\Ring\\\\Future\\\\CompletedFutureArray), Object(Closure))\\n#8 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/Client.php(165): GuzzleHttp\\\\RequestFsm->__invoke(Object(GuzzleHttp\\\\Transaction))\\n#9 \\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/Client.php(125): GuzzleHttp\\\\Client->send(Object(GuzzleHttp\\\\Message\\\\Request))\\n#10 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Http\\\/Client\\\/Client.php(137): GuzzleHttp\\\\Client->get('https:\\\/\\\/office....', Array)\\n#11 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/richdocuments\\\/lib\\\/WOPI\\\/DiscoveryManager.php(84): OC\\\\Http\\\\Client\\\\Client->get('https:\\\/\\\/office....')\\n#12 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/richdocuments\\\/lib\\\/WOPI\\\/Parser.php(41): OCA\\\\Richdocuments\\\\WOPI\\\\DiscoveryManager->get()\\n#13 \\\/var\\\/www\\\/nextcloud\\\/apps\\\/richdocuments\\\/lib\\\/Controller\\\/DocumentController.php(235): OCA\\\\Richdocuments\\\\WOPI\\\\Parser->getUrlSrc('application\\\/vnd...')\\n#14 [internal function]: OCA\\\\Richdocuments\\\\Controller\\\\DocumentController->create('application\\\/vnd...', 'New Document.od...', '\\\/Documents')\\n#15 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(160): call_user_func_array(Array, Array)\\n#16 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/Http\\\/Dispatcher.php(90): OC\\\\AppFramework\\\\Http\\\\Dispatcher->executeController(Object(OCA\\\\Richdocuments\\\\Controller\\\\DocumentController), 'create')\\n#17 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/App.php(114): OC\\\\AppFramework\\\\Http\\\\Dispatcher->dispatch(Object(OCA\\\\Richdocuments\\\\Controller\\\\DocumentController), 'create')\\n#18 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/AppFramework\\\/Routing\\\/RouteActionHandler.php(47): OC\\\\AppFramework\\\\App::main('OCA\\\\\\\\Richdocumen...', 'create', Object(OC\\\\AppFramework\\\\DependencyInjection\\\\DIContainer), Array)\\n#19 [internal function]: OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler->__invoke(Array)\\n#20 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/private\\\/Route\\\/Router.php(299): call_user_func(Object(OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler), Array)\\n#21 \\\/var\\\/www\\\/nextcloud\\\/lib\\\/base.php(1010): OC\\\\Route\\\\Router->match('\\\/apps\\\/richdocum...')\\n#22 \\\/var\\\/www\\\/nextcloud\\\/index.php(40): OC::handleRequest()\\n#23 {main}\",\"File\":\"\\\/var\\\/www\\\/nextcloud\\\/3rdparty\\\/guzzlehttp\\\/guzzle\\\/src\\\/Exception\\\/RequestException.php\",\"Line\":89}","level":3,"time":"2017-05-07T22:44:14+00:00","method":"POST","url":"\/apps\/richdocuments\/ajax\/documents\/create","user":"ncadmin","version":"11.0.3.2"}
{"reqId":"k66dU3BQk4UIkq4\/m75Q","remoteAddr":"10.0.2.3","app":"PHP","message":"Uncaught Error: Access to undeclared static property: OC\\Files\\Filesystem::$normalizedPathCache in \/var\/www\/nextcloud\/lib\/private\/Files\/Filesystem.php:806\nStack trace:\n#0 \/var\/www\/nextcloud\/lib\/private\/Files\/View.php(2018): OC\\Files\\Filesystem::normalizePath('\/ncadmin\/files\/...')\n#1 \/var\/www\/nextcloud\/lib\/private\/Files\/View.php(1156): OC\\Files\\View->unlockFile('\/Nextcloud.mp4', 1)\n#2 [internal function]: OC\\Files\\View->OC\\Files\\{closure}()\n#3 \/var\/www\/nextcloud\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php(109): call_user_func(Object(Closure))\n#4 [internal function]: Icewind\\Streams\\CallbackWrapper->stream_close()\n#5 {main}\n  thrown at \/var\/www\/nextcloud\/lib\/private\/Files\/Filesystem.php#806","level":3,"time":"2017-05-07T22:44:47+00:00","method":"GET","url":"\/ocs\/v2.php\/apps\/notifications\/api\/v2\/notifications","user":"ncadmin","version":"11.0.3.2"}
{"reqId":"FhL2akOPzoY4VFDOvkUa","remoteAddr":"10.0.2.3","app":"PHP","message":"session_write_close(): write failed: No space left on device (28) at \/var\/www\/nextcloud\/lib\/private\/Session\/Internal.php#104","level":3,"time":"2017-05-07T23:03:07+00:00","method":"GET","url":"\/logout?requesttoken=1w4bEOcqPBgK9c7b1ornJPMhUbYTXLSuQFWr6gVEHQ%3D%3A6AR7Ky2qkckHaaNCWhRDqPmJ8yqvf0Xg8WY6xPgtQho%3D","user":"--","version":"11.0.3.2"}
{"reqId":"FhL2akOPzoY4VFDOvkUa","remoteAddr":"10.0.2.3","app":"PHP","message":"session_write_close(): Failed to write session data (files). Please verify that the current setting of session.save_path is correct (\/var\/lib\/php\/sessions) at \/var\/www\/nextcloud\/lib\/private\/Session\/Internal.php#104","level":3,"time":"2017-05-07T23:03:07+00:00","method":"GET","url":"\/logout?requesttoken=1w4bEOcqPBgK9c7b1ornJPMhUbYTXLSuQFWr6gVEHQ%3D%3A6AR7Ky2qkckHaaNCWhRDqPmJ8yqvf0Xg8WY6xPgtQho%3D","user":"--","version":"11.0.3.2"}
{"reqId":"KfH7vu7mti8Ii3K9vlx9","remoteAddr":"10.0.2.3","app":"PHP","message":"session_write_close(): write failed: No space left on device (28) at \/var\/www\/nextcloud\/lib\/private\/Session\/Internal.php#104","level":3,"time":"2017-05-07T23:03:08+00:00","method":"GET","url":"\/login","user":"--","version":"1{"reqId":"dSi5AWqeHOEljwsHT\/K6","remoteAddr":"10.0.2.3","app":"core","message":"Trusted domain error. \"10.0.2.3\" tried to access using \"216.153.158.244\" as host.","level":2,"time":"2017-05-08T05:24:01+00:00","method":"GET","url":"\/","user":"--","version":"11.0.3.2"}

Installation log (command output)

Insert the CLI output here

Screenshoots

[enoch85]

Thanks for reporting. This is due to that docker saves all the old containers. Maybe we should add a cronjob to remove them...

Please run:
docker ps --filter "status=exited" | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty docker rm

And then check the available memory again. Thanks!

[enoch85]

Ignore my last comments. Please test this:
docker system prune -a --force

[Ark74]

El lun, 08-05-2017 a las 11:28 -0700, Daniel Hansson escribió:

Igonore my last comments. Please test this:
docker system prune -a --force`
So, in order to clean old docker images we need to run another?
Or am i missing something
I'm on a old installation.

[aaaskew]

Thanks for the help. The system prune command does clean up the disk space. However it also removes the container and image. To get back to a working system the docker image would need to be re-downloaded and the container setup again. See below for what I have done so far.

However, this does not fix the underlying issue, that is the current collabora docker image appears to crash and restart. This could be down to my setup, although it was a clean install of the VM, or a docker image that does not like the Nextcloud VM. I will try a complete re-install at the weekend but in the meantime I will see if I can understand how the docker container is created and started at present and what is behind the docker image crashing.

root@nextcloud:~# docker system prune -a --force
Deleted Containers:
05b8474397ae6707da6bedb9e878384f939df71166e098345eb25e44cba14c52

Deleted Images:
untagged: collabora/code:latest
untagged: collabora/code@sha256:f8484c89288f9d0571481f03c39823e4a06fc9fbff2760b64941210c4560f514
deleted: sha256:42ce9c66b4e4728d6a68ebe83293a4576e6d44a8aff6e6ce24f1ae9774e41eb3
deleted: sha256:6d14c8717073f101dccf478bd246aaaecce58d40fe16de07ed045bb4c5e94d43
deleted: sha256:eafc63e39582a7b58c00ca17b4b511bdb6ef71c37509f50f5b8413ac5089a320
deleted: sha256:bacd66008c3598243007387732bb044877ba6e1ca13a78fa1bc53e563ac8128c
deleted: sha256:3e2d12b23fafe176cf40429fb25be6572212807f27455b8e3e114c397324446f
deleted: sha256:88a37465e211da3c72acbd999b158ee31c6c7239131f6308f000dcf52d622a7b
deleted: sha256:99be185bee70b39c64096b8d39b96153d28d3caa7764961a9285ad4d189cd536
deleted: sha256:fc7e2c65ec42780443f87ae7d9621cd6fcdb371e2127dd461b449d9e50b7ab7b
deleted: sha256:4f03495a4d7de505ccb8b8e4cfd0a8ac201491e1f67ae54b65584e0012aaab9c

root@nextcloud:~# docker pull collabora/code:latest
latest: Pulling from collabora/code
8aec416115fd: Pull complete
695f074e24e3: Pull complete
946d6c48c2a7: Pull complete
bc7277e579f0: Pull complete
2508cbcde94b: Pull complete
db0e063f8c96: Pull complete
6a091a62febe: Pull complete
2097d1c8ed5e: Pull complete
Digest: sha256:f8484c89288f9d0571481f03c39823e4a06fc9fbff2760b64941210c4560f514
Status: Downloaded newer image for collabora/code:latest
root@nextcloud:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
collabora/code latest 42ce9c66b4e4 4 days ago 1.72 GB

root@nextcloud:~# df -h
Filesystem Size Used Avail Use% Mounted on
…
/dev/mapper/nextcloud--vg-root 18G 5.1G 12G 32% /

root@nextcloud:~# docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

[enoch85]

@aaaskew Hmm, that's strange. I just re-installed my own Collabora setup with the script, and there are no errors. Also, dodcker system prune only removes untagged containers (that are not in use) and shouldn't remove live containers.

This is really a Collabora issue rather than a VM issue tbh, but I'd be happy to try to come up with a solution that safely deletes old containers.

@Ark74 Naah, just a proposal, system prune seems to handle the job quite well, but I'm not sure if it deletes the live container as well. It didn't do it for me, what about for you?

[JSoko]

Here
docker system prune -a --force
just removed untagged containers. CODE is just running as it should...

[aaaskew]

Just a quick update to clarify. As the hard disk was full I had used 'kill ####' on the container process. I always only ever had the one container. This container was the only one adding all the files. However it appears as if the container was starting up and dying lots of times. If I took a look at the files in the aufs diff, the same files appeared multiple times. This looks to me as if each time it started it created a bunch of files but did not finish, restarted and created some more.

I have not used docker before so my understanding of what is going on and the appropriate terminology might well be incorrect. However when running the system prune command, the container was still listed with -a (all containers) but not part of the running list. If I restarted in this state, the container would start up again.

I need to try various full re-installs of the VM using different networking options, guest VM parameters and on different hardware before I am happy I understand why this is going on. That will have to wait until the weekend.

[enoch85]

just removed untagged containers. CODE is just running as it should...

Yup, noticed the same behavior...

Works: 2
Doesn't work: 1

Any one else that can confirm this working?

[aaaskew]

Just so people know the initial problem reported was due to aufs, used by docker crashing:
auplink[2576]: segfault at 7fff324a50a8 ip 00007f57e19009c9 sp 00007fff324a50b0 error 6 in libc-2.23.so

This was on a host with:

• 1.5GHz AMD Turion(tm) II Neo N40L Dual-Core
• Debian 7 64-bit
• VirtualBox 5.0.40

To improve things I removed PAE and dropped the VM down to 1 core and turned on host I/O for the hard disk. Although slow, at least this appears to stop the timing / scheduling issues causing the docker to do things out of order and mess up the aufs file system.

Now to see if this is actually stable.

[danger89]

I'm using Debian Jessie, and my docker container is also crashing over and over again, so I get error 502 of Collabora.

Collabora/code docker container is crashing since the STATUS uptime is always less than a second up and then it the up time resets to Up 1 seconds..

sudo docker ps                                                                                                                                                    
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES 
ff7bb05494f4        collabora/code      "/bin/sh -c 'bash ..."   12 hours ago        Up 34 seconds       127.0.0.1:9980->9980/tcp   peaceful_haibt 
5a0be9cc3e04        mquandalle/wekan    "/bin/sh -c 'bash ..."   14 months ago       Up 2 months         0.0.0.0:8081->80/tcp       wekan_wekan_1 
02cf142a5f3c        mongo               "/entrypoint.sh mo..."   14 months ago       Up 2 months         27017/tcp                  wekan_wekandb_1   

I'm using Docker version 17.05.0-ce.

So what now..!?? I'm not using a VM actually.

[enoch85]

@danger89 In the VM we have a script that takes care of that called dockerprune.sh Check the merged commit.

[danger89]

The solution was to switch aufs file system (that didn't work very well under Debian Jessie) to device mapper. From that moment I could start collabora docker image without crashes..