Docker-selenium: x11vnc High Cpu Usage after running a day

Created on 29 Mar 2020  ·  13Comments  ·  Source: SeleniumHQ/docker-selenium

🐛 Bug Report

To Reproduce

I have build a docker image locally from official image to change default vnc password , it works fine at the beginning , but x11vnc process get high cpu usage （100% ）after running a day

here is the command to start x11vnc :
x11vnc -usepw -forever -shared -rfbport 5900 -display :99.0 -noxrecord

Environment

OS: centos 7
Docker-Selenium image version: standalone-chrome-debug:3.141.59-zirconium
Docker version: Docker version 19.03.7
Exact Docker command to start the containers (if using docker-compose, provide
the docker-compose file as well):
docker run -d --net=host -v /dev/shm:/dev/shm selenium/standalone-chrome-debug:3.141.59-zirconium

enhancement help wanted
Source
picture mokitoo

All 13 comments

There is nothing strange in /var/log/supervisor/vnc-stderr.log

28/03/2020 12:52:46  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:52:46 destroyed xdamage object: 0x4000f9
28/03/2020 12:52:48 Got connection from client 78.128.112.26
28/03/2020 12:52:48   other clients:
28/03/2020 12:52:48 Normal socket connection
28/03/2020 12:52:48 Disabled X server key autorepeat.
28/03/2020 12:52:48   to force back on run: 'xset r on' (3 times)
28/03/2020 12:52:48 incr accepted_client=294 for 78.128.112.26:33113  sock=8
28/03/2020 12:52:48 created   xdamage object: 0x4000fa
28/03/2020 12:52:50 Client Protocol Version 3.8
28/03/2020 12:52:50 Protocol version sent 3.8, using 3.8
28/03/2020 12:52:59 rfbProcessClientSecurityType: executing handler for type 2
28/03/2020 12:53:00 authProcessClientMessage: authentication failed from 78.128.112.26
28/03/2020 12:53:00 rfbAuthProcessClientMessage: password check failed
28/03/2020 12:53:00 rfbClientSendString("password check failed!")
28/03/2020 12:53:00 client_count: 0
28/03/2020 12:53:00 Restored X server key autorepeat to: 1
28/03/2020 12:53:00 Client 78.128.112.26 gone
28/03/2020 12:53:00 Statistics             events    Transmit/ RawEquiv ( saved)
28/03/2020 12:53:00  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:00 Statistics             events    Received/ RawEquiv ( saved)
28/03/2020 12:53:00  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:01 destroyed xdamage object: 0x4000fa
28/03/2020 12:53:03 Got connection from client 78.128.112.26
28/03/2020 12:53:03   other clients:
28/03/2020 12:53:03 Normal socket connection
28/03/2020 12:53:03 Disabled X server key autorepeat.
28/03/2020 12:53:03   to force back on run: 'xset r on' (3 times)
28/03/2020 12:53:03 incr accepted_client=295 for 78.128.112.26:2173  sock=8
28/03/2020 12:53:03 created   xdamage object: 0x4000fb
28/03/2020 12:53:05 Client Protocol Version 3.8
28/03/2020 12:53:05 Protocol version sent 3.8, using 3.8
28/03/2020 12:53:14 rfbProcessClientSecurityType: executing handler for type 2
28/03/2020 12:53:16 authProcessClientMessage: authentication failed from 78.128.112.26
28/03/2020 12:53:16 rfbAuthProcessClientMessage: password check failed
28/03/2020 12:53:16 rfbClientSendString("password check failed!")
28/03/2020 12:53:16 client_count: 0
28/03/2020 12:53:16 Restored X server key autorepeat to: 1
28/03/2020 12:53:16 Client 78.128.112.26 gone
28/03/2020 12:53:16 Statistics             events    Transmit/ RawEquiv ( saved)
28/03/2020 12:53:16  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:16 Statistics             events    Received/ RawEquiv ( saved)
28/03/2020 12:53:16  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:18 Got connection from client 78.128.112.26
28/03/2020 12:53:18   other clients:
28/03/2020 12:53:18 Normal socket connection
28/03/2020 12:53:18 Disabled X server key autorepeat.
28/03/2020 12:53:18   to force back on run: 'xset r on' (3 times)
28/03/2020 12:53:18 incr accepted_client=296 for 78.128.112.26:49637  sock=8
28/03/2020 12:53:21 Client Protocol Version 3.8
28/03/2020 12:53:21 Protocol version sent 3.8, using 3.8
28/03/2020 12:53:30 rfbProcessClientSecurityType: executing handler for type 2
28/03/2020 12:53:31 authProcessClientMessage: authentication failed from 78.128.112.26
28/03/2020 12:53:31 rfbAuthProcessClientMessage: password check failed
28/03/2020 12:53:31 rfbClientSendString("password check failed!")
28/03/2020 12:53:31 client_count: 0
28/03/2020 12:53:31 Restored X server key autorepeat to: 1
28/03/2020 12:53:31 Client 78.128.112.26 gone
28/03/2020 12:53:31 Statistics             events    Transmit/ RawEquiv ( saved)
28/03/2020 12:53:31  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:31 Statistics             events    Received/ RawEquiv ( saved)
28/03/2020 12:53:31  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:32 destroyed xdamage object: 0x4000fb
28/03/2020 12:53:33 Got connection from client 78.128.112.26
28/03/2020 12:53:33   other clients:
28/03/2020 12:53:33 Normal socket connection
28/03/2020 12:53:33 Disabled X server key autorepeat.
28/03/2020 12:53:33   to force back on run: 'xset r on' (3 times)
28/03/2020 12:53:33 incr accepted_client=297 for 78.128.112.26:46695  sock=8
28/03/2020 12:53:34 created   xdamage object: 0x4000fc
28/03/2020 12:53:35 Client Protocol Version 3.8
28/03/2020 12:53:35 Protocol version sent 3.8, using 3.8
28/03/2020 12:53:46 rfbProcessClientSecurityType: executing handler for type 2
28/03/2020 12:53:47 authProcessClientMessage: authentication failed from 78.128.112.26
28/03/2020 12:53:47 rfbAuthProcessClientMessage: password check failed
28/03/2020 12:53:47 rfbClientSendString("password check failed!")
28/03/2020 12:53:47 client_count: 0
28/03/2020 12:53:47 Restored X server key autorepeat to: 1
28/03/2020 12:53:47 Client 78.128.112.26 gone
28/03/2020 12:53:47 Statistics             events    Transmit/ RawEquiv ( saved)
28/03/2020 12:53:47  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:47 Statistics             events    Received/ RawEquiv ( saved)
28/03/2020 12:53:47  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:53:50 Got connection from client 78.128.112.26
28/03/2020 12:53:50   other clients:
28/03/2020 12:53:50 Normal socket connection
28/03/2020 12:53:50 Disabled X server key autorepeat.
28/03/2020 12:53:50   to force back on run: 'xset r on' (3 times)
28/03/2020 12:53:50 incr accepted_client=298 for 78.128.112.26:9638  sock=8
28/03/2020 12:53:57 Client Protocol Version 3.8
28/03/2020 12:53:57 Protocol version sent 3.8, using 3.8
28/03/2020 12:54:00 rfbProcessClientSecurityType: executing handler for type 2
28/03/2020 12:54:01 authProcessClientMessage: authentication failed from 78.128.112.26
28/03/2020 12:54:01 rfbAuthProcessClientMessage: password check failed
28/03/2020 12:54:01 rfbClientSendString("password check failed!")
28/03/2020 12:54:01 client_count: 0
28/03/2020 12:54:01 Restored X server key autorepeat to: 1
28/03/2020 12:54:01 Client 78.128.112.26 gone
28/03/2020 12:54:01 Statistics             events    Transmit/ RawEquiv ( saved)
28/03/2020 12:54:01  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:54:01 Statistics             events    Received/ RawEquiv ( saved)
28/03/2020 12:54:01  TOTALS              :      0 |         0/        0 (  0.0%)
28/03/2020 12:54:03 destroyed xdamage object: 0x4000fc
28/03/2020 12:54:04 Got connection from client 78.128.112.26
28/03/2020 12:54:04   other clients:
28/03/2020 12:54:04 Normal socket connection
28/03/2020 12:54:04 Disabled X server key autorepeat.
28/03/2020 12:54:04   to force back on run: 'xset r on' (3 times)
28/03/2020 12:54:04 incr accepted_client=299 for 78.128.112.26:34659  sock=8
28/03/2020 12:54:04 created   xdamage object: 0x4000fd
28/03/2020 12:54:11 Client Protocol Version 3.8
28/03/2020 12:54:11 Protocol version sent 3.8, using 3.8
28/03/2020 12:54:12 Got connection from client 198.108.67.48
28/03/2020 12:54:12   other clients:
28/03/2020 12:54:12      78.128.112.26
mokitoo picture mokitoo on 29 Mar 2020

@Buffer2Disk does this happen when you use the images we release from this project?
What was the analysis done to conclude that the high CPU was caused by x11vnc?

diemol picture diemol on 8 Apr 2020

@Buffer2Disk does this happen when you use the images we release from this project?
What was the analysis done to conclude that the high CPU was caused by x11vnc?

Thanks for your reply.
1.Yes，I have build a docker image locally from official image just to change default vnc password.
I don't know exactly when will this bug occured，because it has nothing strange in x11vnc log. But it will appear in a day every time. I think this issue may due to x11vnc.

2.I use htop to monitor process resource usage，such as cpu，and x11vnc will show 100% cpu in htop

So, I just thinking , is there any better way to upgrade x11vnc in docker or any other vnc server to replace it ?

I have checked x11vnc version and it's really old one in docker image , i tried to build new version of x11vnc (https://github.com/LibVNC/x11vnc/) ,but it has some errors in docker environment

mokitoo picture mokitoo on 9 Apr 2020

What I meant from my first question is: does the same behavior happen when you use the images we generate.
So far, this is the first time we hear about this. I wonder if you can:

  • Try the images we release and see if it happens.
  • Try in a different machine, and see if it happens.
diemol picture diemol on 15 Apr 2020

What I meant from my first question is: does the same behavior happen when you use the images we generate.
So far, this is the first time we hear about this. I wonder if you can:

  • Try the images we release and see if it happens.
  • Try in a different machine, and see if it happens.

Thanks for your reply.
I followed official document and build a docker image locally from official image，just to change default vnc password. So i think there is not so much difference between official image and my local image. Here is my local docker file 

FROM selenium/node-chrome-debug:3.141.59-20200409
RUN x11vnc -storepasswd <your-password-here> /home/seluser/.vnc/passwd

By the way , i have found someone else already submit a issue about this ,
x11vnc server on ubuntu 18.04 hangs after ~ 24 hours

The main reason may due to x11vnc memory leaks bug in that old version (version is 0.9.13 ,it was 8 yeas old) , and he has builded x11vnc to upgrade it .After overnight testing has shown that it is more stable and doesn't have memory leaks observed in the older version.
Why does the newest version of Ubuntu (18.04) have 8 years old x11vnc

I have folloed his instruction to build new version of x11vnc in my selenium docker container, and it works fine now ,so i highly recommend to upgrade x11vnc version in official selenium docker image.

mokitoo picture mokitoo on 18 Apr 2020
👍1

Here is my own docker file to upgrade x11vnc , it works fine now

#pull a basic  image
FROM selenium/standalone-chrome-debug:3.141.59-zirconium

USER root

#MAINTAINER
MAINTAINER ***

WORKDIR /root

# copy apt-get sources list to image
COPY sources.list /etc/apt/


# install tools for compile
RUN apt-get update && apt-get install -y \
    autoconf \
    automake \
    libtool \
    libffi-dev \
    git \
    libssl-dev \
    xorg-dev \
    libvncserver-dev


# download  source code
RUN cd /root && mkdir src && cd src && git clone https://github.com/LibVNC/x11vnc

# compile and install , default install path /usr/local/bin/x11vnc
RUN apt-get remove -y x11vnc \
    && cd /root/src/x11vnc \
    && autoreconf -fiv \
    && ./configure \
    && make \
    && make install

# clean source code and some tools
RUN rm -rf /root/src/ \
    && apt-get remove --purge -y autoconf automake libtool libffi-dev git libssl-dev xorg-dev libvncserver-dev


#change vnc password
RUN x11vnc -storepasswd ******** /home/seluser/.vnc/passwd

source list

deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse

deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
mokitoo picture mokitoo on 19 Apr 2020
🎉1

Thank you for all this information @Buffer2Disk!
We'll definitely have a look and find a way to include this improvement to the containers.

diemol picture diemol on 13 May 2020

Any news about that ? I have the same issue after running a day (cpu usage)
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 92a5cf5e324a chrome-1_1 360.47% 7.722GiB / 62.83GiB 12.29% 37.7GB / 2.2GB 157MB / 10.5GB 2361 ddb309e5efdf chrome-3_1 267.94% 7.263GiB / 62.83GiB 11.56% 33.9GB / 2.03GB 160MB / 9.61GB 2240 a66d87e336e7 chrome-2_1 297.31% 7.579GiB / 62.83GiB 12.06% 34GB / 1.95GB 176MB / 9.43GB 2367 04344bd3953f chrome-6_1 311.54% 6.772GiB / 62.83GiB 10.78% 28.5GB / 1.68GB 151MB / 8.57GB 2108 1650acda4ff5 chrome-5_1 264.54% 7.647GiB / 62.83GiB 12.17% 37.1GB / 2.18GB 173MB / 10.2GB 2369 91df234e8897 chrome-4_1 293.95% 7.334GiB / 62.83GiB 11.67% 36.4GB / 2.03GB 170MB / 9.58GB 2240 188e19fe8995 selenium-hub 7.33% 1.527GiB / 62.83GiB 2.43% 98.6GB / 102GB 81.1MB / 0B 154

hbouhadji picture hbouhadji on 22 Dec 2020

@hbouhadji, we have not had the time to do that. The whole build and packaging pipeline needs to be built because X11vnc has no recent release. It is not a complex task, but it requires a lot of detail and setup to plug it into the build process.

If you have the time, we'd be happy to chat here about approaches to work on this and naturally happy to receive PRs. @hbouhadji, would you like to contribute?

diemol picture diemol on 22 Dec 2020

https://packages.ubuntu.com/bionic/x11vnc
https://packages.ubuntu.com/focal/x11vnc

I can see that on focal (20.04) the version of x11vnc is 0.9.16-3 and for bionic (18.04) it's 0.9.13-3.
Isn't it easier to upgrade to ubuntu 20.04?
If not, I will be happy to contribute to the project.

hbouhadji picture hbouhadji on 23 Dec 2020

Upgrading to 20.04 is a good idea, but does that solve the issue?

diemol picture diemol on 23 Dec 2020

Just created the PR #1169 @diemol

hbouhadji picture hbouhadji on 26 Dec 2020

This should be fixed through https://github.com/SeleniumHQ/docker-selenium/pull/1169, thanks to @hbouhadji!

We'll do a release in the next couple of days.

diemol picture diemol on 5 Jan 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

testphreak picture testphreak  ·  3Comments
rosskevin picture rosskevin  ·  5Comments
timmipetit picture timmipetit  ·  6Comments
Dhananjayulu picture Dhananjayulu  ·  5Comments
vikramvi picture vikramvi  ·  6Comments