Ubuntu 18.04.2 LTS in an LXC container. The physical host is also under 18.04.2. Total physical RAM: 32GB.
Binary packages from Scilla repository:
root@scdb:~# aptitude show scylla
Package: scylla
Version: 3.0.4-0.20190313.5e3a52024-0ubuntu1~bionic
State: installed
The RAM footprint is limited to 10GB, and also 4 cores out out of 8 are allowed to use for Scylla:
SCYLLA_ARGS="--log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --memory 10G --smp 4"
nodetool is unusable, always giving NullPointerException:
root@scdb:~# nodetool cfstats
error: null
-- StackTrace --
java.lang.NullPointerException
at org.apache.cassandra.config.DatabaseDescriptor.getDiskFailurePolicy(DatabaseDescriptor.java:1881)
at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:82)
at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:79)
at org.apache.cassandra.utils.FBUtilities.getToolsOutputDirectory(FBUtilities.java:824)
at org.apache.cassandra.tools.NodeTool.printHistory(NodeTool.java:204)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:170)
root@scdb:~# nodetool --help
nodetool: Found unexpected parameters: [--help]
See 'nodetool help' or 'nodetool help <command>'.
root@scdb:~# nodetool help
error: null
-- StackTrace --
java.lang.NullPointerException
at org.apache.cassandra.config.DatabaseDescriptor.getDiskFailurePolicy(DatabaseDescriptor.java:1881)
at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:82)
at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:79)
at org.apache.cassandra.utils.FBUtilities.getToolsOutputDirectory(FBUtilities.java:824)
at org.apache.cassandra.tools.NodeTool.printHistory(NodeTool.java:204)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:170)
root@scdb:~# nodetool help cfstats
error: null
-- StackTrace --
java.lang.NullPointerException
at org.apache.cassandra.config.DatabaseDescriptor.getDiskFailurePolicy(DatabaseDescriptor.java:1881)
at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:82)
at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:79)
at org.apache.cassandra.utils.FBUtilities.getToolsOutputDirectory(FBUtilities.java:824)
at org.apache.cassandra.tools.NodeTool.printHistory(NodeTool.java:204)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:170)
cc32d9
All 19 comments
Scylla server is working (although with problems described in #4417), and cqlsh is working too.
cc32d9
on 10 Apr 2019
How much ram is allocated for the container ?
Please note that you can instruct scylla to reserve more memory to the OS and everything else that needs to run using --reserve-memory https://docs.scylladb.com/kb/memory-usage/#scylla-memory-usage
slivne
on 11 Apr 2019
the container is not limited, and it has full access to 32GB of physical RAM. About 10GB RAM is free, so it should not be a problem.
cc32d9
on 11 Apr 2019
I get the same error on a freshly installed baremetal Ubuntu 18.04 server, and it's regardless if Scylla is running:
root@x21:~# nodetool status
error: null
-- StackTrace --
java.lang.NullPointerException
at org.apache.cassandra.config.DatabaseDescriptor.getDiskFailurePolicy(DatabaseDescriptor.java:1881)
at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:82)
at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:79)
at org.apache.cassandra.utils.FBUtilities.getToolsOutputDirectory(FBUtilities.java:824)
at org.apache.cassandra.tools.NodeTool.printHistory(NodeTool.java:204)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:170)
(failed reverse-i-search)`stop': nodetool ^Catus
root@x21:~# systemctl stop scylla-server
root@x21:~# nodetool status
error: null
-- StackTrace --
java.lang.NullPointerException
at org.apache.cassandra.config.DatabaseDescriptor.getDiskFailurePolicy(DatabaseDescriptor.java:1881)
at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:82)
at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:79)
at org.apache.cassandra.utils.FBUtilities.getToolsOutputDirectory(FBUtilities.java:824)
at org.apache.cassandra.tools.NodeTool.printHistory(NodeTool.java:204)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:170)
maybe it misses some package that was not installed by "apt install scylla" ?
cc32d9
on 11 Apr 2019
It requires scylla-conf, but it should be installed (and scylla won't run without it either). The error points at missing configuration. Please strace -f -e trace=file to see if it failed to open something.
avikivity
on 11 Apr 2019
See attachment. It tries to access a bunch of Cassandra libraries which don't seem to be present.
nodetool_trace.log
cc32d9
on 11 Apr 2019
I can provide ssh access to the server. It's a machine dedicated to testing, so nothing valuable on it.
cc32d9
on 11 Apr 2019
This problem happened to me, the problem is the java version
the nodetool needs version 8 of java, if the version is higher than version 8 you have this error.
torresglauco
on 16 Apr 2019
@torresglauco thanks a lot. So, it's a package dependency issue. The one that got installed automatically on Ubuntu 18.04 is:
root@x21:~# java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment (build 11.0.2+9-Ubuntu-3ubuntu118.04.3)
OpenJDK 64-Bit Server VM (build 11.0.2+9-Ubuntu-3ubuntu118.04.3, mixed mode, sharing)
root@x21:~# aptitude search jdk
p default-jdk - Standard Java or Java compatible Development Kit
p default-jdk-doc - Standard Java or Java compatible Development Kit (documentation)
p default-jdk-headless - Standard Java or Java compatible Development Kit (headless)
p golang-github-jdkato-prose-dev - Golang library for text processing
p golang-github-jdkato-syllables-dev - Go syllable counter
p libhtsjdk-java - Java API for high-throughput sequencing data (HTS) formats
p libhtsjdk-java-doc - Documentation for the java HTSJDK library
p openjdk-11-dbg - Java runtime based on OpenJDK (debugging symbols)
p openjdk-11-demo - Java runtime based on OpenJDK (demos and examples)
p openjdk-11-doc - OpenJDK Development Kit (JDK) documentation
p openjdk-11-jdk - OpenJDK Development Kit (JDK)
p openjdk-11-jdk-headless - OpenJDK Development Kit (JDK) (headless)
p openjdk-11-jre - OpenJDK Java runtime, using Hotspot JIT
i A openjdk-11-jre-headless - OpenJDK Java runtime, using Hotspot JIT (headless)
p openjdk-11-jre-zero - Alternative JVM for OpenJDK, using Zero
p openjdk-11-source - OpenJDK Development Kit (JDK) source files
p openjdk-8-dbg - Java runtime based on OpenJDK (debugging symbols)
p openjdk-8-demo - Java runtime based on OpenJDK (demos and examples)
p openjdk-8-doc - OpenJDK Development Kit (JDK) documentation
p openjdk-8-jdk - OpenJDK Development Kit (JDK)
p openjdk-8-jdk-headless - OpenJDK Development Kit (JDK) (headless)
p openjdk-8-jre - OpenJDK Java runtime, using Hotspot JIT
p openjdk-8-jre-dcevm - Alternative VM for OpenJDK 8 with enhanced class redefinition
i A openjdk-8-jre-headless - OpenJDK Java runtime, using Hotspot JIT (headless)
p openjdk-8-jre-zero - Alternative JVM for OpenJDK, using Zero/Shark
p openjdk-8-source - OpenJDK Development Kit (JDK) source files
p uwsgi-plugin-jvm-openjdk-8 - Java plugin for uWSGI (OpenJDK 8)
p uwsgi-plugin-jwsgi-openjdk-8 - JWSGI plugin for uWSGI (OpenJDK 8)
p uwsgi-plugin-ring-openjdk-8 - Closure/Ring plugin for uWSGI (OpenJDK 8)
p uwsgi-plugin-servlet-openjdk-8 - JWSGI plugin for uWSGI (OpenJDK 8)
Now it worked:
root@x21:~# update-java-alternatives --list
java-1.11.0-openjdk-amd64 1111 /usr/lib/jvm/java-1.11.0-openjdk-amd64
java-1.8.0-openjdk-amd64 1081 /usr/lib/jvm/java-1.8.0-openjdk-amd64
root@x21:~# update-java-alternatives --set java-1.8.0-openjdk-amd64
<output skipped>
root@x21:~# java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
root@x21:~# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 1.2.3.4 502.47 GB 256 ? eca56fcf-5d12-4c7a-822a-633b88baaecc rack1
Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
root@x21:~#
Same version of Ubuntu, same error, same fix.
urdnot-ios
on 17 Apr 2019
Same error happened to me too and I was able to fix it by installing java version 8.
Something to note is that a lot of the tutorials on how to install java version 8 use webupd8 but as of april 16th 2019 the webupd8 ppa is discontinued because oracle JDK's license has changed
so you will have to find another route to install java that doesn't include the following
sudo add-apt-repository ppa:webupd8team/java
forbesjon2
on 21 Apr 2019
for some reason scylla installs both 8 and 11 (openjdk-11-jre-headless openjdk-8-jre-headless):
~$ sudo apt-get install scylla
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
ca-certificates-java default-jre-headless fontconfig-config fonts-dejavu-core hugepages hwloc-nox java-common libasound2 libasound2-data libavahi-client3 libavahi-common-data
libavahi-common3 libcrypto++6 libcups2 libfontconfig1 libgnutlsxx28 libhugetlbfs0 libhwloc-plugins libhwloc5 libjpeg-turbo8 libjpeg8 libjsoncpp1 liblcms2-2 libltdl7 libnspr4
libnss3 libopts25 libpciaccess0 libpcsclite1 libprotobuf10 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libsnappy1v5 libxi6 libxrender1 libxtst6 ntp
ocl-icd-libopencl1 openjdk-11-jre-headless openjdk-8-jre-headless python python-asn1crypto python-certifi python-cffi-backend python-chardet python-cryptography python-enum34
python-idna python-ipaddress python-minimal python-openssl python-pkg-resources python-requests python-six python-urllib3 python-urwid python-yaml python2.7 python2.7-minimal
python3-pyudev scylla-conf scylla-env scylla-gcc73-gcc-7-base scylla-gcc73-libatomic1 scylla-gcc73-libgcc1 scylla-gcc73-libstdc++6 scylla-jmx scylla-kernel-conf scylla-server
scylla-tools scylla-tools-core sntp x11-common
sebastianreloaded
on 27 Apr 2019
Quick workaround, if you want to keep both versions around and prefer Java 11:
alias nt='JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/ nodetool'
lydickaw
on 26 May 2019
related to https://github.com/scylladb/scylla-tools-java/pull/76 and scylladb/scylla-jmx@9eec9eabf6d291a2f7ce98dfce9df510dead42d5
slivne
on 11 Jul 2019
I think a viable workaround is to rebuild scylla-tools-java with the new java version (1.11).
kostja
on 20 Dec 2019
this is work in progress (still trying to get scylla-tools-java to compile with java 11) - should be fixed by 4.1
slivne
on 24 Mar 2020
the fix to support java 11 is part of master and will be part of 4.1
slivne
on 6 May 2020
Related issues
hellowaywewe
路
3Comments
avikivity
路
4Comments
mattcobley
路
6Comments
amoskong
路
6Comments
vladzcloudius
路
5Comments
Most helpful comment
Now it worked: