Elasticsearch: [CI] Failure in BWC on debian 8 tests with `_cat/nodes`. Error: Values less than -1 bytes are not supported

Created on 30 Mar 2020  路  24Comments  路  Source: elastic/elasticsearch

Multiple BWC tests fail in similar ways.

Values less than -1 bytes are not supported: -44383772672b

This occurs in BWC tests with 7.7.

Some failures:

https://gradle-enterprise.elastic.co/s/rpa3bpzgm7bkg

https://gradle-enterprise.elastic.co/s/or7qqkbuuazto

Here is one of the many stack traces:

Full stack trace

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:38624], URI [/_cat/nodes?h=master%2Cversion&error_trace=true], status line [HTTP/1.1 400 Bad Request]
    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -46886875136b","stack_trace":"[Values less than -1 bytes are not supported: -46886875136b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -46886875136b];
 org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
 org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
 org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
 org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
 org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)
 org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:49)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
 org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
 org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
 org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
 org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
 org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
 org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
 org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
 org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
 org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
 io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
 io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
 io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
 io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
 io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
 io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
 org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
 org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
 org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
\t... 50 more
"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -46886875136b","stack_trace":"java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
 org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
 org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
 org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
 org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
 org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
 org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
 org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
 org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
 org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
 org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
 org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
 org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
 io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
 io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
 io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
 io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
 io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
 io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 java.base/java.lang.Thread.run(Thread.java:832)
"},"status":400}
        at __randomizedtesting.SeedInfo.seed([30DEEF650CBC9654:34BFE11467792139]:0)
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
        at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.lambda$waitForTemplates$0(XPackRestTestHelper.java:48)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:905)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:890)
        at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.waitForTemplates(XPackRestTestHelper.java:44)
:CorFeatureStats >test-failure CorFeatures
Source
picture benwtrent

Most helpful comment

Oh nevermind! sorry @gwbrown, I read this as relating to the TimeValue change but this is bytes, so ignore this!

picture dakrone on 30 Mar 2020
❤1 👍1

All 24 comments

Pinging @elastic/es-core-features (:Core/Features/CAT APIs)

elasticmachine picture elasticmachine on 30 Mar 2020

Here are some nifty server logs:
659.tar.gz

benwtrent picture benwtrent on 30 Mar 2020

Related to PR: https://github.com/elastic/elasticsearch/pull/54259

benwtrent picture benwtrent on 30 Mar 2020

@gwbrown I think we may need a way to be more lenient for BWC when receiving values from other nodes, as otherwise users may run into this problem with mixed version clusters, what do you think?

dakrone picture dakrone on 30 Mar 2020

Oh nevermind! sorry @gwbrown, I read this as relating to the TimeValue change but this is bytes, so ignore this!

dakrone picture dakrone on 30 Mar 2020
❤1 👍1

Ha, I made the same mistake at first, then started wondering if I'd done some work on preventing negative byte values and just forgotten. Happy to help if you want assistance, of course!

gwbrown picture gwbrown on 30 Mar 2020

Another failure today:
https://gradle-enterprise.elastic.co/s/wthzhjyetha7c

Seems to be specific 7.x and debian08

https://gradle-enterprise.elastic.co/scans/tests?list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.startTimeMax=1585699386158&search.startTimeMin=1585094586151&search.tags=CI&search.tzOffset=660&tests.container=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT&tests.sortField=FAILED&tests.test=test%20%7Bp0%3Dold_cluster/10_basic/Create%20a%20task%20result%20record%20in%20the%20old%20cluster%7D&tests.unstableOnly=true

ywangd picture ywangd on 1 Apr 2020
👍1

It also appears that the affected tests are UpgradeClusterClientYamlTestSuiteIT and MlConfigIndexMappingsFullClusterRestartIT#testMlConfigIndexMappingsAfterMigration.

Now that we have an error trace on the response, there's a stack trace in the failing response for the ML test that we didn't have when this ticket was filed:

java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
    at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
    at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
    at org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
    at org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
    at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
    at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
    at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
    at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
    at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
    at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
    at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
    at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
    at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
    at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
    at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
    at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.base/java.lang.Thread.run(Thread.java:832)

It appears that this error happens when OsStats.Mem has a larger value for free memory than for total memory:

        public ByteSizeValue getUsed() {
            return new ByteSizeValue(total - free);
        }

The first failure was on March 25th, not long after we cut the 7.7 branch. There have been consistent failures in the twice-daily bwc run for debian 8. Chart: gradle enterprise

williamrandolph picture williamrandolph on 2 Apr 2020
👍1

Recent failures for today on debian-8 and 7.x:

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=debian-8&&immutable/667/console
Build scan: https://gradle-enterprise.elastic.co/s/zgglcpzvad2m6

Failing tests:
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Create a task result record in the old cluster}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Create things in the cluster state that we'll validate are there after the upgrade}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Index percolator queries and use the percolate query in old cluster}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with java style index in 6}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time (>6.1)}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time. (6.0)}
org.elasticsearch.xpack.restart.MlConfigIndexMappingsFullClusterRestartIT testMlConfigIndexMappingsAfterMigration

The same error message:
method [GET], host [http://127.0.0.1:44582], URI [/_cat/nodes?h=version%2Cmaster], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -48852353024b"}]

mayya-sharipova picture mayya-sharipova on 3 Apr 2020

Another failure: Jenkins, Build scan

dimitris-athanasiou picture dimitris-athanasiou on 6 Apr 2020

And another: https://gradle-enterprise.elastic.co/s/oauuxgrtd6i5e/console-log?task=:x-pack:qa:full-cluster-restart:v7.7.0%23oldClusterTest

romseygeek picture romseygeek on 14 Apr 2020

Another instance of the the MlConfigIndexMappingsFullClusterRestartIT.testMlConfigIndexMappingsAfterMigration failure on 7.x:
https://gradle-enterprise.elastic.co/s/i73lmau2ago5w

rg.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:59558], URI [/_cat/nodes?h=master%2Cversion&error_trace=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -43149541376b","stack_trace":"[Values less than -1 bytes are not supported: -43149541376b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -43149541376b];
at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat
cbuescher picture cbuescher on 16 Apr 2020

It happened again today on 7.x:

https://gradle-enterprise.elastic.co/s/negka76witrbo

This test is failing twice a day for some time now so I'm going to mute it (#55389).

I'm also assigning this issue to @jakelandis so that he can assign it to someone from the core/feature team.

tlrx picture tlrx on 17 Apr 2020

Failed again: https://gradle-enterprise.elastic.co/s/hsoc4hi5sf5p6 (7.x, debian)
Same issue as the one from my last shift. 

test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time (>6.1)} 
test {p0=old_cluster/10_basic/Index percolator queries and use the percolate query in old cluster}
test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time. (6.0)}
test {p0=old_cluster/10_basic/Create a task result record in the old cluster}
test {p0=old_cluster/10_basic/Create things in the cluster state that we'll validate are there after the upgrade
test {p0=old_cluster/20_date_range/Create index with java style index in 6}

Error message is {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -40171786240b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -40171786240b"},"status":400}

ywangd picture ywangd on 24 Apr 2020

Still happening: https://gradle-enterprise.elastic.co/s/pzl2mgclmc2zq

jkakavas picture jkakavas on 6 May 2020

Still happening: https://gradle-enterprise.elastic.co/s/wyh6iqz76h54u

andreidan picture andreidan on 7 May 2020

Another one: https://gradle-enterprise.elastic.co/s/j6vigxs56aes2/

I'm going to try and mute more tests for this.

tvernum picture tvernum on 8 May 2020

I couldn't see a way to mute this easily, and I also tried to work out how this could happen, but didn't get anywhere.

Given that it's a problem on 7.x debian-8 only, I suspect this is actually an issue with the OS Management Bean on Java 8, and we should just work around it by detecting that free > total and return 0 or -1.

We could try and work out _why_, but I suspect that will just end up finding that it's a JDK bug we can't do anything about

tvernum picture tvernum on 8 May 2020
👍1

I spent some time on this yesterday and came to a similar conclusion regarding detecting free > total situations. The one possibility that I could come up with was that while @dakrone demonstrated in https://github.com/elastic/elasticsearch/pull/42725 that negative values can be returned for either total or free memory, those occurrences may not coincide such that total is reported as negative but free is reported as positive thus resulting in a free > total situation. I've opened https://github.com/elastic/elasticsearch/pull/56412 as a possible way to address it.

danhermann picture danhermann on 8 May 2020

https://gradle-enterprise.elastic.co/s/cno7wedfdoqfg

https://gradle-enterprise.elastic.co/s/viy6zvnenjkn4

Same failure, but on 7.7.

Would be good to have the fix backported into 7.7 when possible.

benwtrent picture benwtrent on 11 May 2020

More today:

https://gradle-enterprise.elastic.co/s/fnqczt5dpd7r6 (7.x, testing vs 7.7.1)
https://gradle-enterprise.elastic.co/s/qcsgtthijgysc (7.8, testing vs 7.7.1)
https://gradle-enterprise.elastic.co/s/efruirda6froi (7.x, testing vs 7.7.1)

DaveCTurner picture DaveCTurner on 14 May 2020

I will backport the fix for these into 7.7 and 7.8 shortly.

danhermann picture danhermann on 14 May 2020

Closing this as the fix has been backported to 7.7 and 7.8 in addition to 7.x and master.

danhermann picture danhermann on 18 May 2020

In our case it seemed to be the swap, so switching it off with sudo swapoff -a immediately stoped the warnings.
However, we are ES 6.8.10 on Ubuntu 18.04 (didn't see this on ES 6.8.7)

IvanBoyko picture IvanBoyko on 9 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ttaranov picture ttaranov  路  3Comments
Praveen82 picture Praveen82  路  3Comments
rjernst picture rjernst  路  3Comments
clintongormley picture clintongormley  路  3Comments
dawi picture dawi  路  3Comments