Kibana: [Reporting] Headless Chromium can not start in some Linuxes

[tsullivan]

Kibana version: 6.5.4

Elasticsearch version:

Server OS version: seen in Ubuntu and CentOS 7

Original install method (e.g. download page, yum, from source, etc.): .tar.gz

Describe the bug:
Reporting fails and logs Caught error spawning Chromium.

Steps to reproduce:

1. Start a GCP instance of CentOS 7
2. Download Kibana .tar.gz and untar
3. Try running the binary for the headless browser directly, using the command-line parameters that Kibana Reporting uses internally:
   /kibana/data/headless_shell-linux/$ ./headless_shell \ --disable-translate \ --disable-extensions \ --disable-background-networking \ --safebrowsing-disable-auto-update \ --disable-sync \ --metrics-recording-only \ --disable-default-apps \ --mute-audio \ --no-first-run \ --user-data-dir=/tmp/test \ --disable-gpu \ --headless \ --hide-scrollbars \ --window-size=1024,768 \ --enable-logging \ --v=1 \ --disable-setuid-sandbox

Expected behavior:
Should create a chrome_debug.log file with non-fatal logs, and exit.

Error Logs

$ cat chrome_debug.log
[0104/215750.718914:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale
[0104/215750.725622:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale
[0104/215750.726741:INFO:cpu_info.cc(46)] Available number of cores: 1
[0104/215750.726793:VERBOSE1:zygote_main_linux.cc(217)] ZygoteMain: initializing 0 fork delegates
[0104/215751.427159:ERROR:gpu_process_transport_factory.cc(968)] Lost UI shared context.
[0104/215751.427389:VERBOSE1:webrtc_internals.cc(123)] Could not get the download directory.
[0104/215751.466982:VERBOSE1:proxy_resolution_service.cc(1049)] PAC support disabled because there is no system implementation
[0104/215751.484741:VERBOSE1:proxy_resolution_service.cc(1049)] PAC support disabled because there is no system implementation
[0104/215751.502055:WARNING:resource_bundle.cc(358)] locale_file_path.empty() for locale
[0104/215751.502718:ERROR:egl_util.cc(59)] Failed to load GLES library: /home/test/kibana/data/headless_shell-linux/swiftshader/libGLESv2.so: cannot open shared object file: No such file or directory

A missing libGLESv2.so libs dependency causes Chromium to fail.

Additionally, fulfilling the reference to that library may yield to other dependency errors due to not having a running XServer. From an earlier analysis by @lucabelluccini

using the swiftshader libraries shipped with Chrome Headless allow the rendering without any XServer running

[lucabelluccini]

Hello @tsullivan, thank you for opening the issue.
Regarding the XServer dependency, it only occurs in the case we try to use the libGL* libraries shipped with the OS (e.g. from the mesa packages).
If we use the swiftshader libraries, the X server API is no more required.
I think they've been crafted exactly for this purpose.

Shipping the chrome_headless with the swiftype libraries would solve this (or maybe bundle the libraries in the binary itself).

[joelgriffith]

Interestingly enough swiftshader has been shipped with Chromium for a while now (according to their issue tracker: https://bugs.chromium.org/p/chromium/issues/detail?id=630728). I'm going to do some digging in Ubuntu on this and see if it's an issue there as well.

[lucabelluccini]

Hello @joelgriffith! I've seen the chromium issue tracker but downloading the latest version of Chrome headless (should be https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/619659/chrome-linux.zip), the swiftshader libraries are delivered as separate files.

[tsullivan]

Hi @lucabelluccini, while @joelgriffith and I are digging into the issue, we noticed that in an environment running Kibana + Reporting from source, on a Mac OS workstation, log lines in chrome_debug.log will look something like this (I had them numbered so I could tell apart trial runs):
https://gist.github.com/tsullivan/213356dd9e36c8936e53fa012776a2ad

This is after enabling verbose debugging for the headless browser in Kibana, which causes the chrome_debug.log file to be generated.

It includes lines very similar to your report details, lines that are labelled as errors:

    12  [0107/165956.517732:ERROR:gl_implementation.cc(281)] Failed to load /Users/tsullivan/elastic/kibana/data/headless_shell-darwin/libswiftshader_libGLESv2.dylib: dlopen(/Users/tsullivan/elastic/kibana/data/headless_shell-darwin/libswiftshader_libGLESv2.dylib, 1): image not found
    13  [0107/165956.742811:ERROR:gl_implementation.cc(281)] Failed to load /Users/tsullivan/elastic/kibana/data/headless_shell-darwin/libswiftshader_libGLESv2.dylib: dlopen(/Users/tsullivan/elastic/kibana/data/headless_shell-darwin/libswiftshader_libGLESv2.dylib, 1): image not found
    14  [0107/165956.742885:VERBOSE1:gpu_init.cc(238)] gl::init::InitializeGLNoExtensionsOneOff failed
    15  [0107/165956.854435:ERROR:viz_main_impl.cc(184)] Exiting GPU process due to errors during initialization
    16  [0107/165956.855603:VERBOSE1:statistics_recorder.cc(410)] Collections of all histograms
    17  Histogram: Mojo.MachPortRelay.ChildError recorded 1 samples, mean = 0.0 (flags = 0x1)

It will take further investigation to find if the swiftshader dependency will actually fix the error with Chromium not starting, or is a red herring

[joelgriffith]

Since this does little with Chromium when ran, I've added a few switches to see if it _truly_ can do something (in this case generate a PDF):

./headless_shell \
  --disable-translate \
  --disable-extensions \
  --disable-background-networking \
  --safebrowsing-disable-auto-update \
  --disable-sync \
  --metrics-recording-only \
  --disable-default-apps \
  --mute-audio \
  --no-first-run  \
  --user-data-dir=/tmp/test \
  --disable-gpu \
  --headless \
  --hide-scrollbars \
  --window-size=1024,768 \
  --enable-logging \
  --v=1 \
  --disable-setuid-sandbox \
  --no-sandbox \
  --print-to-pdf="/usr/share/kibana/data/headless_shell-linux/temp.pdf" \
  https://example.com

I've ran this inside the docker container that we maintain, and when _not_ giving Chrome a site it shows a debug log similar to what we've found in the previous SDH issues (as Tim mentioned). So I think we've been bamboozled by a red herring :(

@lucabelluccini could you try and run this new command inside your host to see if it works or not? It should generate a PDF at the path you give it.

[joelgriffith]

FWIW our docker image doesn't load much (seen here). The most notable line is:

# Add Reporting dependencies.
RUN yum update -y && yum install -y fontconfig freetype && yum clean all

[tsullivan]

It's expected for Chromium to not start of the OS is missing dependencies. When that happens in normal operation, Chromium will log to stderr about the missing dependencies. It appears there is a regression in Kibana making it not able to read the stderr: https://github.com/elastic/kibana/issues/28408

Closing this because the bug isn't that Chrome can't start, the bug is we aren't able to figure out why it can't start. #28408 will track that.

[lucabelluccini]

Opened https://github.com/elastic/kibana/pull/29940