Janusgraph: Failure seen while porting JanusGraph on ppc64le
Hi All,
I am trying run the JanusGraph build and automated tests on ubuntu 16.10 and RHEL 7.3 VM ppc64le and for the module janusgraph-hadoop-parent, test case verify-janusgraph-cassandra-test fails on both the distros.
The testcase summary file (/janusgraph/janusgraph-hadoop-parent/janusgraph-hadoop-1/target/failsafe-reports/failsafe-janusgraph-cassandra.xml) shows error in 21 tests.
Further analysis showed 7 testcases failed with IncompatibleClassChangeError and 3 testcases failed with IllegalStateException
Analysis of IncompatibleClassChangeError:
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.cassandra.hadoop.HadoopCompat.
at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:120)
at org.janusgraph.hadoop.formats.cassandra.CassandraBinaryInputFormat.getSplits(CassandraBinaryInputFormat.java:62)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at org.janusgraph.hadoop.scan.HadoopScanRunner.runJob(HadoopScanRunner.java:136)
at org.janusgraph.hadoop.MapReduceIndexManagement.updateIndex(MapReduceIndexManagement.java:186)
at org.janusgraph.hadoop.AbstractIndexManagementIT.testRemoveGraphIndex(AbstractIndexManagementIT.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Analysis on IllegalStateException
java.lang.IllegalStateException: java.lang.ExceptionInInitializerError
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:80)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:126)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:37)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:157)
at org.janusgraph.hadoop.CassandraInputFormatIT.testReadGraphOfTheGods(CassandraInputFormatIT.java:55) Caused by: java.lang.ExceptionInInitializerError: null
at org.apache.spark.storage.DiskBlockManager.addShutdownHook(DiskBlockManager.scala:147)
at org.apache.spark.storage.DiskBlockManager.
at org.apache.spark.storage.BlockManager.
at org.apache.spark.storage.BlockManager.
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:347)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
at org.apache.spark.SparkContext.
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2258)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at org.apache.tinkerpop.gremlin.spark.structure.Spark.create(Spark.java:53)
at org.apache.tinkerpop.gremlin.spark.structure.io.SparkContextStorage.open(SparkContextStorage.java:60)
at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$28(SparkGraphComputer.java:138)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NoSuchFieldException: SHUTDOWN_HOOK_PRIORITY
at java.lang.Class.getField(Class.java:1703)
at org.apache.spark.util.SparkShutdownHookManager.install(ShutdownHookManager.scala:220)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks$lzycompute(ShutdownHookManager.scala:50)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks(ShutdownHookManager.scala:48)
at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:189)
at org.apache.spark.util.ShutdownHookManager$.
at org.apache.spark.util.ShutdownHookManager$.
at org.apache.spark.storage.DiskBlockManager.addShutdownHook(DiskBlockManager.scala:147)
at org.apache.spark.storage.DiskBlockManager.
at org.apache.spark.storage.BlockManager.
at org.apache.spark.storage.BlockManager.
Steps I followed were:
git clone https://github.com/JanusGraph/janusgraph && cd janusgraph
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
mvn install
I am doing this build on Ubuntu 16.04 and RHEL 7.3 ppc64le. Please help me in resolving this issue.
Thanks,
Archa
ArchaB
All 8 comments
JanusGraph is currently using TinkerPop 3.1.0 which doesn't look like it supports Hadoop1 anymore. See Hadoop-Gremlin section on the page below. The janusgraph-hadoop-1 module probably just needs to be removed.
http://tinkerpop.apache.org/docs/3.1.1-incubating/upgrade/#_tinkerpop_3_1_0_2
sjudeng
on 20 Jan 2017
Thanks @sjudeng for the quick response.
Will try building JanusGraph by removing janusgraph-hadoop-1 module and update the behavior
ArchaB
on 23 Jan 2017
Enabled Hadoop1 which resulted in 1 testcase (testReadWideVertexWithManyProperties) failure in janusgraph-hadoop-2 package
Failure seen:
<error type="java.lang.IllegalStateException" message="org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory output/~reducing already exists">java.lang.IllegalStateException: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory output/~reducing already exists at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:80) at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:126) at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:37) at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:157) at org.janusgraph.hadoop.CassandraInputFormatIT.testReadWideVertexWithManyProperties(CassandraInputFormatIT.java:81) Caused by: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory output/~reducing already exists at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1011) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:938) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:930) at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:809) at org.apache.tinkerpop.gremlin.spark.structure.io.OutputFormatRDD.writeMemoryRDD(OutputFormatRDD.java:65) at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$28(SparkGraphComputer.java:271) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) </error>
ArchaB
on 24 Jan 2017
That's a known bug. I think the correct fix is to read gremlin.hadoop.outputLocation and create a new temporary directory based on that during test setup. I have the fix in the below branch, though it's part of a wider refactoring to use an abstract base class for these tests to support HBase tests.
If you're okay with a quicker temporary solution/hack you could just add a setup method and remove the output directory.
sjudeng
on 24 Jan 2017
Thanks @sjudeng .
Will try on removing the output directory.
ArchaB
on 31 Jan 2017
FYI the fix described above has now been merged into master so hopefully this issue is resolved.
sjudeng
on 16 Feb 2017
Thanks @sjudeng for the intimation.
I tried to build the latest Janusgraph code, but it build failed for the 'JanusGraph-Core: Core Library for JanusGraph' package.
Failure seen:
[ERROR] Failed to execute goal on project janusgraph-core: Could not resolve dependencies for project org.janusgraph:janusgraph-core:jar:0.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project janusgraph-core: Could not resolve dependencies for project org.janusgraph:janusgraph-core:jar:0.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:221)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:862)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:286)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.janusgraph:janusgraph-core:jar:0.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:211)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
... 23 more
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:205)
... 24 more
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: org.apache.tinkerpop:gremlin-core:jar:3.2.3, org.apache.tinkerpop:gremlin-groovy:jar:3.2.3, org.apache.tinkerpop:tinkergraph-gremlin:jar:3.2.3, com.codahale.metrics:metrics-ganglia:jar:3.0.1: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
... 25 more
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.tinkerpop:gremlin-core:jar:3.2.3 in pentaho.resolve.repo (http://nexus.pentaho.org/content/groups/omni)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:39)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[ERROR]
Am I missing something? I followed all the steps I had performed last time.
ArchaB
on 21 Feb 2017
@ArchaB Is your Maven configured to use a proxy? That pentho mirror doesn't seem to have the tinkerpop libraries. One option would be for you to build/install TinkerPop 3.2.3 on the same machine, then it would find it in your local repo.
pluradj
on 21 Feb 2017
Related issues
bionicles
路
4Comments
pratikjain1991
路
3Comments
zifnab87
路
3Comments
robertdale
路
4Comments
lazharichir
路
5Comments
Most helpful comment
JanusGraph is currently using TinkerPop 3.1.0 which doesn't look like it supports Hadoop1 anymore. See Hadoop-Gremlin section on the page below. The janusgraph-hadoop-1 module probably just needs to be removed.
http://tinkerpop.apache.org/docs/3.1.1-incubating/upgrade/#_tinkerpop_3_1_0_2