Docker-selenium: Containers size increases filling up disk space

[bizmate]

Meta -

Image(s):
selenium/node-chrome latest 7ec01c7dcfb0 2 weeks ago 809.5 MB
selenium/node-firefox latest bf49590cb0d1 2 weeks ago 634 MB
selenium/hub latest 2e3bc5bc3dd0 2 weeks ago 319.8 MB
Docker Version:
Docker version 1.12.0, build 8eab29e

Expected Behavior -

An acceptable or workable garbage collection mechanism, ie is there a way to limit the maxim size and force containers to do garbage collection?

Actual Behavior -

Containers become huge, see this is 14GB but I have had containers reaching 300+ GB, this is only with chrome and firefox

docker ps -as
fd7c31eb5d33        selenium/node-firefox                 "/opt/bin/entry_point"   22 hours ago        Up 14 seconds                                     sitecrawler_firefox_1    14 GB (virtual 14.63 GB)

[ddavison]

i believe this could be something to do with the /tmp directory, and the profiles being created, but not cleaned up. I agree this is something that should be covered.

[sheerun]

also because of excessive logging

[joelchrist]

Any update on this? Would really like to know which files are growing so big.

[rnemeth1980]

core-dumps make containers full, so that no selenium grid tests can be executed on any firefox pod in k8s anymore.

I get this error while test execution org.openqa.selenium.WebDriverException: java.io.IOException: No space left on device.

entries in ./home
-rw------- 1 seluser seluser 507M Sep 27 23:03 core.1211
-rw------- 1 seluser seluser 496M Sep 24 23:02 core.12299
-rw------- 1 seluser seluser 628M Sep 21 23:00 core.17331
-rw------- 1 seluser seluser 627M Sep 7 12:24 core.17928
-rw------- 1 seluser seluser 584M Sep 9 23:01 core.1847
-rw------- 1 seluser seluser 536M Sep 23 23:00 core.19056
-rw------- 1 seluser seluser 611M Sep 15 23:00 core.23882
-rw------- 1 seluser seluser 584M Sep 30 23:00 core.23942
-rw------- 1 seluser seluser 667M Sep 19 23:00 core.24496
-rw------- 1 seluser seluser 644M Sep 26 23:01 core.27137
-rw------- 1 seluser seluser 364M Okt 1 23:01 core.27953
-rw------- 1 seluser seluser 515M Sep 28 23:03 core.28882
-rw------- 1 seluser seluser 644M Sep 10 23:01 core.2943

is there any way to prevent these core-dumps or delete them automatically?

[slmzig]

Have the same issue, when I close browser - new core dump created.
Any update on this?

[diemol]

Aside from the logging, it seems that Firefox was having an issue where dumps were being created on every session. Some improvements were done there by the Firefox/GeckoDriver people, so this should not be an issue anymore.

For the ones who have commented in this issue? What is the status you see now with the most recent release?

[isaulv]

My org is seeing this issue now since we have always on containers running in docker on a AWS instance. We don't use firefox, but only Chrome, and it seems that chromedriver doesn't clean up profiles (anymore?).

I think a script that runs every 24 hour can reap directories that match a pattern (chrome*, whatever firefox uses in case of crashes) that checks the ctime would suffice to keep the rubbish down.

Another factor to consider is that we should update the docs to advise mounting the /tmp directory to tmpfs or /tmp for those who don't have autoscaling solutions.

[ddavison]

Another factor to consider is that we should update the docs to advise mounting the /tmp directory to tmpfs or /tmp for those who don't have autoscaling solutions.

that's definitely a good idea, too. we already do -v /dev/shm:/dev/shm perhaps we could recommend -v /tmp:/tmp

[exic]

I recently got this with selenium/node-chrome:3.141.59-dubnium, creating a lot of profile (?) directories in its /tmp folder (/var/lib/docker/overlay2/43d80c...abbe/diff/tmp):

[...]
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 15:55 .org.chromium.Chromium.1DdWfD/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 15:55 .org.chromium.Chromium.9uUpHM/
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 15:55 .org.chromium.Chromium.2x16R7/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 15:55 .org.chromium.Chromium.ESIsob/
[...]
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 16:10 .org.chromium.Chromium.euyXVK/
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 16:14 .org.chromium.Chromium.m70XNj/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 16:14 .org.chromium.Chromium.5E24aP/
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 16:16 .org.chromium.Chromium.Wy4wOU/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 16:16 .org.chromium.Chromium.NWyI1u/
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 16:18 .org.chromium.Chromium.wMsBjA/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 16:18 .org.chromium.Chromium.j203Rs/
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 16:18 .org.chromium.Chromium.n4ALqh/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 16:18 .org.chromium.Chromium.8I5iD3/
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 16:19 .org.chromium.Chromium.VCEEdI/
drwx------    3 xxxxadm xxxxadm   4096 Dec 27 16:19 .org.chromium.Chromium.DOasni/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 16:20 .org.chromium.Chromium.bzaxTe/
drwx------    4 xxxxadm xxxxadm   4096 Dec 27 16:20 .org.chromium.Chromium.4RkT5a/

they all contained those:

total 16
-rw-r--r-- 1 xxxxadm xxxxadm   78 Dec 27 16:19 Local State
-rw-r--r-- 1 xxxxadm xxxxadm    0 Dec 27 16:19 First Run
drwx------ 2 xxxxadm xxxxadm 4096 Dec 27 16:19 Default/
drwx------ 2 xxxxadm xxxxadm 4096 Dec 27 16:20 BrowserMetrics/
-rw-r--r-- 1 xxxxadm xxxxadm  103 Dec 27 16:20 chrome_debug.log

BrowserMetrics then contains 4.1M of data in files like BrowserMetrics-5C24EDA3-244FDB.pma.

This can't be intended behavior, is it? Is this the same issue?

I'd rather not solve the symptom by creating cronjobs that clean this up...

EDIT: Details. However, this only happened on one of two nodes that were running node-chrome containers and doesn't seem to reproduce right now. I'll return here if it occurs again, you might as well ignore this for now.

[diemol]

Is this still an issue, several things have been improved and it would be nice to know if people are still running into this problem with the latest release.

[diemol]

Closing as this went stale.

[chrisv-dev]

I noticed the same behaviour that has already been mentioned about these profile (?) directories in /tmp today. I used the following image: selenium/standalone-chrome (digest: sha256:778b6f15c2bd40100ea4e25ae38baa79402061a79d234c0a233295d5816c91a5).

My container was running for about 11 days and du -hs /tmp inside the container showed 25GB. /tmp contained a lot of folders that start with .com.google.Chrome.. The 25GB was the limit which made me notice the problem in my case, as the docker host had no more space left to use.