Elasticsearch: Running integration test with ES 2.2 throw ControlAccessException

[andrejserafim]

It seems the security controls kick in on integration tests using ESIntegTestCase.

Relevant stack overflow thread: http://stackoverflow.com/questions/35205150/running-unit-tests-for-elasticsearch-2-2-0-in-intellij-fails-with-accesscontrole/35208827#35208827

Repro:

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST, numDataNodes = 0)
public class AndrejTest extends ESIntegTestCase {
    @Test
    public void shouldPrintHello() throws Exception {
        System.out.println("hello");
    }
}

18:34:04 WARN  bootstrap:48 - JNA not found. native methods will be disabled.
java.lang.ClassNotFoundException: com.sun.jna.Native
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45)
    at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:89)
    at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:85)
    at org.elasticsearch.test.ESTestCase.<clinit>(ESTestCase.java:99)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:573)
18:34:04 WARN  bootstrap:65 - cannot check if running as root because JNA is not available
18:34:04 WARN  bootstrap:96 - cannot install syscall filters because JNA is not available
18:34:04 WARN  bootstrap:57 - cannot mlockall because JNA is not available
18:34:04 WARN  bootstrap:81 - cannot register console handler because JNA is not available
java.security.AccessControlException: access denied ("org.elasticsearch.ThreadPermission" "modifyArbitraryThreadGroup")

    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
    at java.security.AccessController.checkPermission(AccessController.java:884)
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
    at org.elasticsearch.SecureSM.checkThreadGroupAccess(SecureSM.java:166)
    at org.elasticsearch.SecureSM.checkAccess(SecureSM.java:113)
    at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
    at java.lang.ThreadGroup.getParent(ThreadGroup.java:167)
    at com.carrotsearch.randomizedtesting.Threads.getTopThreadGroup(Threads.java:113)
    at com.carrotsearch.randomizedtesting.Threads.getAllThreads(Threads.java:97)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.<init>(ThreadLeakControl.java:346)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:654)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:579)

REPRODUCE WITH: mvn test -Pdev -Dtests.seed=15E6042541843D23 -Dtests.class=io.searchbox.client.AndrejTest -Dtests.locale=en_GB -Dtests.timezone=Europe/London
NOTE: test params are: codec=null, sim=null, locale=null, timezone=(null)
NOTE: Linux 4.2.0-27-generic amd64/Oracle Corporation 1.8.0_72 (64-bit)/cpus=4,threads=1,free=92285192,total=122683392
NOTE: All tests run in this JVM: [AndrejTest]
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "setDefaultUncaughtExceptionHandler")
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
    at java.security.AccessController.checkPermission(AccessController.java:884)
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
    at java.lang.Thread.setDefaultUncaughtExceptionHandler(Thread.java:1893)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:602)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:444)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:234)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

Process finished with exit code 254

[centic9]

Workaround from the so-discussion: Set VM options "-Dtests.security.manager=false"

[wallecnik]

I am not sure if I'm doing anything wrong, but running this empty test fails with the same exception:

public class MyTestES extends ESIntegTestCase { 
    @Test 
    public void test() {} 
}

I would appreciate the testing framework to be simple as extending the ESIntegTestCase class.

[s1monw]

hmm this seems like a bug but I wonder how you integrated it into you project? Which version of randomized runner is referenced by your projects since we are granting this permission to grant codeBase "${codebase.randomizedtesting-runner-2.2.0.jar}" {

[s1monw]

also does it work if you run a build outside of your IDE?

[centic9]

For me the only relevant dependencies are

dependencies {
testCompile 'org.elasticsearch:elasticsearch:2.2.0'
testCompile 'org.elasticsearch:elasticsearch:2.2.0:tests'
testCompile 'org.apache.lucene:lucene-test-framework:5.4.1'
}

randomizedtesting-runner is not specified explicitely, the transitive dependencies seem to pull in a difference version via the lucene-test-framework:

--- org.apache.lucene:lucene-test-framework:5.4.1
+--- org.apache.lucene:lucene-codecs:5.4.1
+--- org.apache.lucene:lucene-core:5.4.1
+--- com.carrotsearch.randomizedtesting:junit4-ant:2.3.1
+--- com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.3.1

Version 5.4.1 of lucene-test-framework is listed at http://mvnrepository.com/artifact/org.elasticsearch/elasticsearch/2.2.0

[centic9]

And for me it also happens outside the IDE when using Gradle.

[wallecnik]

Same here, using Maven and Intellij. Dependencies are:

<dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-core</artifactId>
    <version>5.4.1</version>
</dependency>
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>2.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-test-framework</artifactId>
    <version>5.4.1</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>2.2.0</version>
    <scope>test</scope>
    <type>test-jar</type>
</dependency>

EDIT: Running it from IDE fails as well as running mvn clean test
I've created blank project with only these dependencies and it fails too with the same test class:

public class MyTestES extends ESIntegTestCase {
    @Test
    public void test() {}
}

[rendel]

We are facing a similar issue. We managed to solve the modifyArbitraryThreadGroup and setDefaultUncaughtExceptionHandler by setting the com.carrotsearch.randomizedtesting dependency to version 2.2.0. However, we stumbled upon another issue about java.io.FilePermission.
We are using ESIntegTestCase for our unit tests, and it looks like it does not have the proper file permissions for our test directories. The only way we found so far to be able to run unit tests is to specify our own policy at runtime using -Djava.security.policy=my_policy.

You can find more information in my post on discuss: https://discuss.elastic.co/t/permission-denied-error-in-plugin-for-es-2-2-0/40915/4?u=renaud1

[s1monw]

ok so there are several problems which are unrelated. Adding the right randoimized runner version should fix the problem for @wallecnik - can you check if it does? It's a bug we will have to fix in 2.2.1

[s1monw]

@rendel @wallecnik are you using our parent pom file?

[rendel]

@s1monw, after adding the elasticsearch parent pom:

  <parent>
    <groupId>org.elasticsearch</groupId>
    <artifactId>parent</artifactId>
    <version>2.2.0</version>
  </parent>

to our pom, we still have the same file permission error.

[rmuir]

something isn't right with the maven build here i think. @dweiss gave me a simple repro.

[dweiss]

Unfortunately I don't think this was related -- my problem was that the plugin used a custom assembly descriptor (Maven) which after upgrading to ES 2.2.0 didn't include the plugin's security policy, causing errors. A fix to include the security policy worked.

[rendel]

I tried to reproduce the issue with a simple maven project, and I found that the cause was due to our custom path.data node setting (see below). Removing this custom setting fixed the FilePermission error.

  @Override
  protected Settings nodeSettings(int nodeOrdinal) {
    return settingsBuilder()
      .put("path.data", "./target/elasticsearch-test/data/")
      .put(super.nodeSettings(nodeOrdinal)).build();
  }

[s1monw]

@rendel thanks for clarifying!!

[wallecnik]

Adding the randomizedtesting-runner:2.2.0 fixes the empty test as @s1monw asked. Thanks for that!

[andrejserafim]

Adding the randomizedtesting-runner:2.2.0 does indeed fix the basic test. Thank you for that.

However I'm still getting some proxy related security exceptions. Could you give me a hand with these?

A repro:

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.TEST, numDataNodes = 0)
public class AndrejTest extends ESIntegTestCase {
    @Test
    public void shouldPrintHello() throws Exception {
        System.out.println("default proxy settings" + ProxySelector.getDefault());
    }
}

Exception:

java.security.AccessControlException: access denied ("java.net.NetPermission" "getProxySelector")

    at __randomizedtesting.SeedInfo.seed([FB4E16B7BA293FA:4AD43B0D357967E9]:0)
    at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
    at java.security.AccessController.checkPermission(AccessController.java:884)
    at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
    at java.net.ProxySelector.getDefault(ProxySelector.java:94)
    at AndrejTest.shouldPrintHello(AndrejTest.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
    at java.lang.Thread.run(Thread.java:745)

[s1monw]

fixed by https://github.com/elastic/elasticsearch/commit/6fe88cee78c72fde021898eb79883c12edf07c3d

[s1monw]

@andrejserafim you can't just use arbitrary things in the ES test framework. If you are testing a plugin you have to give it the right permissions first.