Elasticsearch: SIGBUS with ~StubRoutines::jlong_disjoint_arraycopy

[dixingxing0]

Elasticsearch 5.2.2

Plugins installed: [elasticsearch-analysis-ik, sql]

JVM version (java -version): 1.8.0_74

OS version (uname -a if on a Unix-like system): 2.6.32-573.22.1.el6.centos.plus.x86_64

Description of the problem including expected versus actual behavior:

Steps to reproduce:

I dont know how to reproduce, now jvm will crash after elasticsearch started a few minutes.
I run two elasticsearch instances per physical machine.

Provide logs (if relevant):

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f4d25052ab0, pid=29234, tid=139965471303424
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v  ~StubRoutines::jlong_disjoint_arraycopy
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f4c1800f000):  JavaThread "elasticsearch[es-test-47-2][generic][T#4]" daemon [_thread_in_Java, id=30540, stack(0x00007f4c402ea000,0x00007f4c4032b000)]

siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007ea221ea0000

Full log : hs_err_pid29234.log

[tlrx]

Looks like you hit https://bugs.java.com/view_bug.do?bug_id=JDK-8168628, there's not much we can do.

You should try to upgrade your JRE.

[dixingxing0]

@tlrx Thank you !

[nalovelyna]

Does that works? I use 8u172, getting the same crash

[robert-blankenship]

Getting the same issue, on Java 8 u181. We have containers running on multiple appliances, each with identical hardware, but the issue only occurs on a certain container.

Can we please reopen this?

[tinawenqiao]

@dijingran ding，Have you solved the problem after you update jdk? Which jdk version do you choose?

[JosuGZ]

UPDATE: Apparently I have bad blocks on my drive, so maybe that is the cause.
UPDATE 2: I can confirm that the problem was due to the inability of Elastic to read a particular file (nodes/0/indices/j8uXlvkpRSOwZyS_c5TDgg/1/index/_62yk.cfs). After ddrescuing the hard drive I'm not experiencing the issue anymore.

---

I'm experiencing this issue with the docker.elastic.co/elasticsearch/elasticsearch:6.8.4 image, I'm not sure if this bug should be solved with the included OpenJDK.

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fbf083cc2d0, pid=1, tid=151
#
# JRE version: OpenJDK Runtime Environment (13.0+33) (build 13+33)
# Java VM: OpenJDK 64-Bit Server VM (13+33, mixed mode, sharing, tiered, compressed oops, concurrent mark sweep gc, linux-amd64)
# Problematic frame:
# v  ~StubRoutines::jlong_disjoint_arraycopy

Aditionally, I'm getting this on startup:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=1

It happens right after recovering a huge translog, at the moment one of the shards reach 100% in top, it crashes. Here is about to crash (seccond shard):

rep snap shost snode thost      tnode   t     ty             st       f fr fp     b br bp     to      tor     top
n/a n/a  n/a   n/a   172.18.0.4 KiI4941 45m   existing_store translog 0 0  100.0% 0 0  100.0% 5372000 4130521 76.9%
n/a n/a  n/a   n/a   172.18.0.4 KiI4941 43.6m existing_store done     0 0  100.0% 0 0  100.0% 5369666 5369666 100.0%
n/a n/a  n/a   n/a   172.18.0.4 KiI4941 45m   existing_store translog 0 0  100.0% 0 0  100.0% 5372000 524881  9.8%