Elasticsearch: [test failure] :x-pack:plugin:watcher:test can fail with java.lang.IllegalArgumentException: Both text and attachments cannot be null.

[jakelandis]

I ran into this on an unrelated PR on the master branch, and can reliably reproduce.

ERROR   0.22s | SlackMessageTests.testTemplateRender <<< FAILURES!
   > Throwable #1: java.lang.IllegalArgumentException: Both text and attachments cannot be null.
   >    at __randomizedtesting.SeedInfo.seed([7E16EC5523F952E7:5C93EBB62B66EC8]:0)

Reproduce:

./gradlew :x-pack:plugin:watcher:test -Dtests.seed=7E16EC5523F952E7 -Dtests.class=org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests -Dtests.method="testTemplateRender" -Dtests.security.manager=true -Dtests.locale=hr -Dtests.timezone=Asia/Omsk

Recent relevant change: https://github.com/elastic/elasticsearch/commit/8ec33b742dc09eebdfba7233d28c61bcb30d5ec5#diff-b5bf09a9e8e2ab5019c940fe22bd8b25R34

Full text:

Suite: org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests
  1> [2018-07-10T20:58:22,638][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=78, reason=Function not implemented
  1> [2018-07-10T20:58:22,640][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
  1> [2018-07-11T07:58:23,314][INFO ][o.e.x.w.n.s.m.SlackMessageTests] [testTemplateRender]: before test
  1> [2018-07-11T07:58:23,507][INFO ][o.e.x.w.n.s.m.SlackMessageTests] [testTemplateRender]: after test
  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:watcher:test -Dtests.seed=7E16EC5523F952E7 -Dtests.class=org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests -Dtests.method="testTemplateRender" -Dtests.security.manager=true -Dtests.locale=hr -Dtests.timezone=Asia/Omsk
ERROR   0.22s | SlackMessageTests.testTemplateRender <<< FAILURES!
   > Throwable #1: java.lang.IllegalArgumentException: Both text and attachments cannot be null.
   >    at __randomizedtesting.SeedInfo.seed([7E16EC5523F952E7:5C93EBB62B66EC8]:0)
   >    at org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessage.<init>(SlackMessage.java:35)
   >    at org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessage$Template.render(SlackMessage.java:210)
   >    at org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests.testTemplateRender(SlackMessageTests.java:497)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)
  2> NOTE: leaving temporary files on disk at: /Users/jake/workspace/elasticsearch/x-pack/plugin/watcher/build/testrun/test/J0/temp/org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests_7E16EC5523F952E7-001
  2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=hr, timezone=Asia/Omsk
  2> NOTE: Mac OS X 10.12.6 x86_64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=1,free=493009280,total=536870912
  2> NOTE: All tests run in this JVM: [SlackMessageTests]
Completed [1/1] in 1.50s, 1 test, 1 error <<< FAILURES!

cc: @hub-cap

[elasticmachine]

Pinging @elastic/es-core-infra

[spinscale]

@hub-cap can you take a look? Might be caused by https://github.com/elastic/elasticsearch/pull/31596

[ywelsch]

I've muted the test in dadf96a840b

[hub-cap]

on it. I already backported the other test this morning tho before seeing this. Should i push a mute to that test too?

[hub-cap]

I have a fix for this in https://github.com/elastic/elasticsearch/pull/31968

[martijnvg]

@hub-cap This error is still occuring: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=sles/1176/

First I thought the last good commit was not up to date, but it seems to use a commit after you pushed the fix.

  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.x+multijob-unix-compatibility/os/sles/x-pack/plugin/watcher/build/testrun/test/J3/temp/org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests_C35892AD1F91E7C7-001
08:21:48    > Throwable #1: java.lang.IllegalArgumentException: Both text and attachments cannot be null.
08:21:48    >   at __randomizedtesting.SeedInfo.seed([C35892AD1F91E7C7:B88740435EDEDBE8]:0)
08:21:48   2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=428, maxMBSortInHeap=6.408206805655725, sim=RandomSimilarity(queryNorm=true): {}, locale=ar, timezone=America/Coral_Harbour
08:21:48   2> NOTE: Linux 4.4.103-6.38-default amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=16,threads=1,free=341246616,total=505937920
08:21:48    >   at org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessage.<init>(SlackMessage.java:35)
08:21:48   2> NOTE: All tests run in this JVM: [CronEvalToolTests, IncidentEventDefaultsTests, ArrayCompareConditionSearchTests, HipChatAccountsTests, WatcherTransportActionTests, HistoryTemplateEmailMappingsTests, TransportWatcherStatsActionTests, PagerDutyActionTests, ChainTransformTests, MapPathTests, SlackMessageTests]
08:21:48    >   at org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessage$Template.render(SlackMessage.java:210)
08:21:48    >   at org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests.testTemplateRender(SlackMessageTests.java:497)
08:21:48    >   at java.lang.Thread.run(Thread.java:748)

[original-brownbear]

+1 this is still failing for me locally with master as well

> Task :x-pack:plugin:watcher:test
==> Test Info: seed=53EDEE31F2D2E609; jvms=4; suites=139

Suite: org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests
  1> [2018-07-12T09:11:18,855][INFO ][o.e.x.w.n.s.m.SlackMessageTests] [testTemplateRender]: before test
  1> [2018-07-12T09:11:18,857][INFO ][o.e.x.w.n.s.m.SlackMessageTests] [testTemplateRender]: after test
  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:watcher:test -Dtests.seed=53EDEE31F2D2E609 -Dtests.class=org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests -Dtests.method="testTemplateRender" -Dtests.security.manager=true -Dtests.locale=pt -Dtests.timezone=Africa/Bissau
ERROR   0.01s J3 | SlackMessageTests.testTemplateRender <<< FAILURES!

[cbuescher]

This still fails reproducibly on master:

REPRODUCE WITH:

 ./gradlew :x-pack:plugin:watcher:test -Dtests.seed=D037A2BE0DBB13AD -Dtests.class=org.elasticsearch.xpack.watcher.notification.slack.message.SlackMessageTests -Dtests.method="testTemplateRender" -Dtests.security.manager=true -Dtests.locale=en -Dtests.timezone=Atlantic/Azores

[cbuescher]

@hub-cap I muted this again in 332c134517bdcf11acd74c205482245c692186fb, maybe its a different reason now but the error looks different. Could you check?

[cbuescher]

Also muted 6.x with where the same reproduction worked with fb6cce0fb4a64b183393ad62cf4020fe6e808e6e

[hub-cap]

Thanks, ill verify. I thought there could be some funkyness with this change and the random behavior. Ill get on it, sorry for the noise.

[hub-cap]

fixed and backported and tested with -Dtests.iters=2000 :)

[cbuescher]