Ray: Unable to create ActorHandle for already created inherited classes object list [java][ray]

Created on 30 Oct 2020  路  26Comments  路  Source: ray-project/ray

What is the problem?

I have a list having different child class objects inheriting from same parent class. But when I try to pass this object list to ActorHandle, it gives below error:

java.lang.ClassCastException: actors.misc.Calendar cannot be cast to io.ray.api.ActorHandle

I have written below code :
for(TimedActor actor: actors_mod) {
counters.add(Ray.actor((RayFunc0) actor).remote());
}
TimedActor is the Parent class.

Using latest version of ray.
Ray version and other system information (Python version, TensorFlow version, OS):

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

If we cannot run your script, we cannot fix your issue.

  • [ ] I have verified my script runs in a clean environment and reproduces the issue.
  • [ ] I have verified the issue also occurs with the latest wheels.
    [java][ray]
P2 bug java
Source
picture SanaIRnI

Most helpful comment

Hi, @SanaIRnI. Sorry for the late reply.

Your way of creating actors seems a bit strange to me. We usually pass a method to Ray.actor(...), not an instance of a class. There are typically two ways for creating actors:

1) Creating from a constructor.

Suppose we have an actor class named Counter with a constructor Counter(int initValue), we can create an actor instance via ActorHandle<Counter> actor = Ray.actor(Counter::new, 1).remote();.

2) Creating from a factory method.

Suppose we have an actor class named Counter and a factory method Counter create(int initValue) in class CounterFactory, we can create an actor instance via ActorHandle<Counter> actor = Ray.actor(CounterFactory::create, 1).remote();.

Which way are you trying to implement?

Could you please also paste the stack trace of the exception? You may show me the lines start with io.ray.

picture kfstorm on 3 Nov 2020
👍3

All 26 comments

cc @kfstorm

rkooo567 picture rkooo567 on 2 Nov 2020

Hi, @SanaIRnI. Sorry for the late reply.

Your way of creating actors seems a bit strange to me. We usually pass a method to Ray.actor(...), not an instance of a class. There are typically two ways for creating actors:

1) Creating from a constructor.

Suppose we have an actor class named Counter with a constructor Counter(int initValue), we can create an actor instance via ActorHandle<Counter> actor = Ray.actor(Counter::new, 1).remote();.

2) Creating from a factory method.

Suppose we have an actor class named Counter and a factory method Counter create(int initValue) in class CounterFactory, we can create an actor instance via ActorHandle<Counter> actor = Ray.actor(CounterFactory::create, 1).remote();.

Which way are you trying to implement?

Could you please also paste the stack trace of the exception? You may show me the lines start with io.ray.

kfstorm picture kfstorm on 3 Nov 2020
👍3

cc @raulchen @chaokunyang

kfstorm picture kfstorm on 3 Nov 2020

Hi @kfstorm
You have mentioned two ways of creating Actors . But my requirement is bit different.
The methods you mentioned create new object of class Counter from its constructer or factory method and then that object is passed to Ray.actor().remote.
In my case I already have list of objects created, now I need to convert these lists of objects as Ray Actors. I am copy pasting my complete method :

public void preExecution(Queue<ParentClass> actors) {

    List<ActorHandle<ParentClass>> counters = new ArrayList<>();

        for(ParentCalss actor: actors) {
            counters.add(Ray.actor((RayFunc0<ParentClass>) actor).remote());
        }
         for (ActorHandle<ParentClass> counter : counters) {
         counter.task(ParentClass::preExecution).remote();
     }
}

Stack Trace of Exception :
I1030 08:40:39.010767 7415 7419 global_state_accessor.cc:25] Redis server address = 192.168.140.63:52901, is test flag = 0
I1030 08:40:39.012615 7415 7419 redis_client.cc:146] RedisClient connected.
I1030 08:40:39.021255 7415 7419 redis_gcs_client.cc:89] RedisGcsClient Connected.
I1030 08:40:39.023068 7415 7419 service_based_gcs_client.cc:193] Reconnected to GCS server: 192.168.140.63:65535
I1030 08:40:39.023561 7415 7419 service_based_accessor.cc:92] Reestablishing subscription for job info.
I1030 08:40:39.023592 7415 7419 service_based_accessor.cc:422] Reestablishing subscription for actor info.
I1030 08:40:39.023605 7415 7419 service_based_accessor.cc:797] Reestablishing subscription for node info.
I1030 08:40:39.023617 7415 7419 service_based_accessor.cc:1073] Reestablishing subscription for task info.
I1030 08:40:39.023630 7415 7419 service_based_accessor.cc:1248] Reestablishing subscription for object locations.
I1030 08:40:39.023650 7415 7419 service_based_accessor.cc:1368] Reestablishing subscription for worker failures.
I1030 08:40:39.023671 7415 7419 service_based_gcs_client.cc:86] ServiceBasedGcsClient Connected.
java.lang.ClassCastException: actors.misc.ChildClass cannot be cast to io.ray.api.function.RayFunc0
at simulator.RayBasedSimulator.preExecution(RayBasedSimulator.java:55)
at simulator.Simulator.simulate(Simulator.java:35)
at simulator.Configuration.simulate(Configuration.java:140)
at driverLocal.Warehouse.runDriver(Warehouse.java:30)
at driverLocal.Warehouse.run(Warehouse.java:85)
at app.RetailDCApplication.run(RetailDCApplication.java:27)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
at app.RetailDCApplication.main(RetailDCApplication.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

_My requirement is exactly that I have a Concurrent Queue having objects of various child class derived from one Parent CLass. I want to run the loop distributed in RAY for each object in the given Queue._

SanaIRnI picture SanaIRnI on 3 Nov 2020

@SanaIRnI If I understand correctly, you want to make the execution of concurrent queues parallel, but not distributed. You still want to execute ParentClass::preExecution of different queues in the same process.

Ray is a framework for building distributed applications. You need to create actors and run tasks on remote nodes/workers to make your application distributed. I'm not sure if this is what you want. If it's not, maybe Ray is not suitable for you and you need to pick other Java frameworks. If it is, we can talk about refactoring your code and making it distributed.

kfstorm picture kfstorm on 4 Nov 2020

Hi @kfstorm

I understand well Ray is used for building distributed applications. We want our application to be distributed, and as Ray is now providing Java support we are trying to port our application Ray Java.
I want to make Parent class as Ray actor and want all the classes that is extending parent class to execute its tasks on remote/worker node.

SanaIRnI picture SanaIRnI on 9 Nov 2020

@SanaIRnI I'm not sure why you talk about parent class and child class. Does the parent/child pattern make a difference here?

Can you try creating instances of child class with ActorHandle<ChildClass> actor = Ray.actor(ChildClass::new, arg1, arg2, ...).remote()? Ray actors need to be created remotely. If you already have instances of ChildClass locally, they cannot be distributed later.

If your child class only holds some data and you can't control its creation, maybe you can try adding a proxy class and creating instances of the proxy class via Ray.actor(...).remote(). Ray will handle serialization for you. e.g.

class ProxyClass {
  private ParentClass actual;

  public ProxyClass(ParentClass actual) {
    this.actual = actual;
  }

  public void preExecution() {
    actual.preExecution();
  }
}

public void preExecution(Queue<ParentClass> dataList) {

    List<ActorHandle<ProxyClass>> counters = new ArrayList<>();

        for(ParentCalss data: dataList) {
            counters.add(Ray.actor(ProxyClass::new, data).remote());
        }
         for (ActorHandle<ProxyClass> counter : counters) {
         counter.task(ProxyClass::preExecution).remote();
     }
}
kfstorm picture kfstorm on 9 Nov 2020

Hi @kfstorm

I implemented the solution you provided, but it gave me null pointer error while creating objects of ProxyParentClass. PFB error stactrace :

java.lang.NullPointerException
at io.ray.api.call.ActorCreator.remote(ActorCreator.java:41)
at simulator.RayBasedSimulator.preExecution(RayBasedSimulator.java:30)
at simulator.Simulator.simulate(Simulator.java:34)
at simulator.Configuration.simulate(Configuration.java:140)
at driverLocal.Warehouse.runDriver(Warehouse.java:30)
at driverLocal.Warehouse.run(Warehouse.java:86)
at app.RetailDCApplication.run(RetailDCApplication.java:26)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
at app.RetailDCApplication.main(RetailDCApplication.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

Also the code implemented :

public class ProxyParentActor {
private ParentActor actor;

  public ProxyParentActor(ParentActor actor) {
    this.actor = actor;
  }

  public void preExecution() {
    actor.preExecution();
  }

}

public void preExecution(Queue<ParentActor> actors) {

    List<ActorHandle<ProxyParentActor>> counters = new ArrayList<>();

    for(ParentActor actor: actors) {
        counters.add(Ray.actor(ProxyParentActor::new, actor).remote());
        System.out.println("Creating Proxy Actors");
    }
     for (ActorHandle<ProxyParentActor> counter : counters) {
            System.out.println("Executing Proxy Actors");

     counter.task(ProxyParentActor::preExecution).remote();
 }
        System.out.println("Done");

}
SanaIRnI picture SanaIRnI on 9 Nov 2020

Did you forget to call Ray.init()? BTW, which ray version are you using?

kfstorm picture kfstorm on 9 Nov 2020

Oh sorry my bad, I forgot to call Ray.init() at start in void main(). My ray version is v1.1.0.dev0
After initilizing Ray and running the application got below error :

I1109 16:06:09.952931 22207 22208 global_state_accessor.cc:25] Redis server address = 192.168.140.63:53029, is test flag = 0
I1109 16:06:09.954768 22207 22208 redis_client.cc:146] RedisClient connected.
I1109 16:06:09.963248 22207 22208 redis_gcs_client.cc:89] RedisGcsClient Connected.
I1109 16:06:09.964540 22207 22208 service_based_gcs_client.cc:193] Reconnected to GCS server: 192.168.140.63:65535
I1109 16:06:09.964761 22207 22208 service_based_accessor.cc:92] Reestablishing subscription for job info.
I1109 16:06:09.964776 22207 22208 service_based_accessor.cc:422] Reestablishing subscription for actor info.
I1109 16:06:09.964787 22207 22208 service_based_accessor.cc:797] Reestablishing subscription for node info.
I1109 16:06:09.964794 22207 22208 service_based_accessor.cc:1073] Reestablishing subscription for task info.
I1109 16:06:09.964802 22207 22208 service_based_accessor.cc:1248] Reestablishing subscription for object locations.
I1109 16:06:09.964814 22207 22208 service_based_accessor.cc:1368] Reestablishing subscription for worker failures.
I1109 16:06:09.964826 22207 22208 service_based_gcs_client.cc:86] ServiceBasedGcsClient Connected.
java.lang.RuntimeException: java.lang.RuntimeException: Class actors.misc.SubClass does not implement Serializable or externalizable
at io.ray.runtime.serializer.MessagePackSerializer.encode(MessagePackSerializer.java:243)
at io.ray.runtime.serializer.Serializer.encode(Serializer.java:8)
at io.ray.runtime.object.ObjectSerializer.serialize(ObjectSerializer.java:103)
at io.ray.runtime.task.ArgumentsBuilder.wrap(ArgumentsBuilder.java:44)
at io.ray.runtime.AbstractRayRuntime.createActorImpl(AbstractRayRuntime.java:261)
at io.ray.runtime.AbstractRayRuntime.createActor(AbstractRayRuntime.java:148)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.ray.runtime.RayRuntimeProxy.invoke(RayRuntimeProxy.java:45)
at com.sun.proxy.$Proxy44.createActor(Unknown Source)
at io.ray.api.call.ActorCreator.remote(ActorCreator.java:41)
at simulator.RayBasedSimulator.preExecution(RayBasedSimulator.java:29)
at simulator.Simulator.simulate(Simulator.java:34)
at simulator.Configuration.simulate(Configuration.java:140)
at driverLocal.Warehouse.runDriver(Warehouse.java:30)
at driverLocal.Warehouse.run(Warehouse.java:86)
at app.RetailDCApplication.run(RetailDCApplication.java:27)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
at app.RetailDCApplication.main(RetailDCApplication.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.RuntimeException: Class actors.misc.SubClass does not implement Serializable or externalizable
at org.nustaq.serialization.FSTClazzInfo.(FSTClazzInfo.java:144)
at org.nustaq.serialization.FSTClazzInfoRegistry.getCLInfo(FSTClazzInfoRegistry.java:129)
at org.nustaq.serialization.FSTObjectOutput.getFstClazzInfo(FSTObjectOutput.java:534)
at org.nustaq.serialization.FSTObjectOutput.writeObjectWithContext(FSTObjectOutput.java:416)
at org.nustaq.serialization.FSTObjectOutput.writeObjectInternal(FSTObjectOutput.java:327)
at org.nustaq.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:294)
at org.nustaq.serialization.FSTObjectOutput.writeObject(FSTObjectOutput.java:204)
at org.nustaq.serialization.FSTConfiguration.asByteArray(FSTConfiguration.java:1182)
at io.ray.runtime.serializer.FstSerializer.encode(FstSerializer.java:22)
at io.ray.runtime.serializer.MessagePackSerializer.lambda$encode$21(MessagePackSerializer.java:226)
at io.ray.runtime.serializer.MessagePackSerializer.lambda$static$2(MessagePackSerializer.java:57)
at io.ray.runtime.serializer.MessagePackSerializer.pack(MessagePackSerializer.java:211)
at io.ray.runtime.serializer.MessagePackSerializer.encode(MessagePackSerializer.java:225)
... 31 more

SanaIRnI picture SanaIRnI on 9 Nov 2020

Ray can only handle serialization for classes that implement Serializable or Externalizable. Do you think you can add implements Serializable for SubClass?

kfstorm picture kfstorm on 9 Nov 2020

@kfstorm Thanks for the help, it finally worked for one function will be implementing it in whole project. Also I got error in my UI interface which for now I have commented. Will check if that error is due to Ray or in general.
Thanks again for quick help 馃憤

SanaIRnI picture SanaIRnI on 9 Nov 2020
🎉1

Hi @kfstorm ,
After making almost every class Serializable in my application I am getting below error :

java.lang.RuntimeException: java.lang.RuntimeException: Class java.lang.reflect.Method does not implement Serializable or externalizable
at io.ray.runtime.serializer.MessagePackSerializer.encode(MessagePackSerializer.java:243)
at io.ray.runtime.serializer.Serializer.encode(Serializer.java:8)
at io.ray.runtime.object.ObjectSerializer.serialize(ObjectSerializer.java:103)
at io.ray.runtime.task.ArgumentsBuilder.wrap(ArgumentsBuilder.java:44)
at io.ray.runtime.AbstractRayRuntime.createActorImpl(AbstractRayRuntime.java:261)
at io.ray.runtime.AbstractRayRuntime.createActor(AbstractRayRuntime.java:148)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.ray.runtime.RayRuntimeProxy.invoke(RayRuntimeProxy.java:45)
at com.sun.proxy.$Proxy44.createActor(Unknown Source)
at io.ray.api.call.ActorCreator.remote(ActorCreator.java:41)
at simulator.RayBasedSimulator.executeAllTimedActors(RayBasedSimulator.java:104)
at simulator.Simulator.simulate(Simulator.java:48)
at simulator.Configuration.simulate(Configuration.java:141)
at driverLocal.Warehouse.runDriver(Warehouse.java:32)
at driverLocal.Warehouse.run(Warehouse.java:88)
at app.RetailDCApplication.run(RetailDCApplication.java:27)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
at app.RetailDCApplication.main(RetailDCApplication.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

Now i cannot make Class java.lang.reflect.Method class as serializable . How to proceed.

SanaIRnI picture SanaIRnI on 11 Nov 2020

@SanaIRnI Can you try wrapping java.lang.reflect.Method like https://stackoverflow.com/a/13086760/2671861? BTW, what's the root cause you need this proxy approach? Why not create ParentActor actors remotely?

kfstorm picture kfstorm on 11 Nov 2020

Hi @kfstorm I have tried the solution mentioned in the stackoverflow link you provided. But error remains same. This is because this Method has to be serialized inside io.ray.runtime.serializer.MessagePackSerializer.java class as error is thrown by ray class not application code.

I need proxy approach because there is one application project which is depended on core project and ui of core project. SubActor Classes are in Application project and ParentActor is in core Project. I cannot import Application project classes in core project classes as per structure of my project.

Also I tried porting my another application on Ray by creating simple one actor and calling .remote() on its method. But Idid not get any output . Application completes its execution , but it seems it doesn't run the Actor class code.
When I tried putting Rat.get() call , my application hangs for indefinite time. Can you explain me how .remote() and ray.get() are working . And how to correctly run java code on Ray. PFB code of second application:
Ray.init(); { List> containers = new ArrayList<>(); for (int i = 0; i < 5; i++) { containers.add(Ray.actor(Container::new,experimentId,inputMode,outputMode).remote()); } for (ActorHandle container : containers) { container.task(Container::run).remote(); } }
And its a runnable java application so I run on ubuntu server as:
java -jar Application.jar
Am I doing anything wrong?

SanaIRnI picture SanaIRnI on 13 Nov 2020

@chaokunyang Could you please help with the serialization part?

@SanaIRnI You can see how ray.remote() and ray.get() work from the Ray Whitepaper - Example of task scheduling and object storage. For your second application, could you please search in "/tmp/ray/session_latest/logs" and try to find some error logs?

kfstorm picture kfstorm on 17 Nov 2020

@SanaIRnI Could you show the code of ParentActor here? The stack show there are errors when serialize ParentCalss. Ould be great if we can know which fields are being serializing for ParentActor? If some fileds can't be serialized, can we make it lazy and transisent? Or implement Externalizable for ParentActor class?

chaokunyang picture chaokunyang on 17 Nov 2020

Hi @chaokunyang ,

As per your suggestion I implemented Externalizable for ParentActor class, But ended up getting below error :
F1118 07:25:24.495728 23968 23969 reference_count.cc:154] Check failed: object_id_refs_.count(object_id) == 0 Tried to create an owned object that already exists: ffffffffffffffff3bc3c4ff010000c001000000
* Check failure stack trace: *
@ 0x7f9726e3502d google::LogMessage::Fail()
@ 0x7f9726e361e3 google::LogMessage::SendToLog()
@ 0x7f9726e34cfa google::LogMessage::Flush()
@ 0x7f9726e34f31 google::LogMessage::~LogMessage()
@ 0x7f9726e1c4e8 ray::RayLog::~RayLog()
@ 0x7f9726b07641 ray::ReferenceCounter::AddOwnedObject()
@ 0x7f9726b17262 ray::ActorManager::AddNewActorHandle()
@ 0x7f9726ac46b2 ray::CoreWorker::CreateActor()
@ 0x7f9726a64b14 Java_io_ray_runtime_task_NativeTaskSubmitter_nativeCreateActor
@ 0x7f9789534774 (unknown)
Aborted (core dumped)

I found similar issue is raised for AWS raytune application. PFB link to the issue :
https://github.com/ray-project/ray/issues/9551

PFA ParentClassCode inside the text file

ParentClassCode.txt

SanaIRnI picture SanaIRnI on 18 Nov 2020

logs.zip
Error File Name : raylet.err
@kfstorm Hi I am attaching my logs files. I am not able to figure out reason behind : Failed to send local GC request. Request you to kindly help. This error I am getting for another application which I am simply running with one ray actor and calling .remote and .get on its method.

SanaIRnI picture SanaIRnI on 19 Nov 2020

@kfstorm Could you please help with reference counting. The actor can't be created.

chaokunyang picture chaokunyang on 20 Nov 2020

@SanaIRnI About the error you see in raylet.err, it's because you were not using the latest Ray (1.0.1). Actually it was a false alarm and has been resolved in 1.0.0. Could you show us the full version string in pom.xml?

BTW, I see another error: "Error: Could not find or load main class io.ray.runtime.runner.worker.DefaultWorker". It seems that the classpath for worker processes is misconfigured. @SanaIRnI Did you set --code-search-path on ray start? Does it include ray-api and ray-runtime packages?

kfstorm picture kfstorm on 20 Nov 2020
👍1

@SanaIRnI About "Check failed: object_id_refs_.count(object_id) == 0 Tried to create an owned object that already exists", does it fail this way constantly?

@chaokunyang The only reason I can think of is that a duplicated actor ID is generated.

kfstorm picture kfstorm on 20 Nov 2020

Hi @kfstorm

My mistake the pom.xml has version 0.9.0 . I modified it to 1.0.0 and the error changed for both projects and giving similar kind of error. It is unable to load functions from classes I have made as ray actor. For Eg in below stack trace ProxyTimedActor is the ray actor class and its giving error.

I tried to make nested class making ray actor class as static class but got similar error.

2020-11-20 10:37:40.148 INFO 12825 --- [ main] io.ray.runtime.runner.RunManager : Ray runtime started @ 192.168.140.63.
java.lang.RuntimeException: Failed to load functions from class actor.ProxyTimedActor
at io.ray.runtime.functionmanager.FunctionManager$JobFunctionTable.loadFunctionsForClass(FunctionManager.java:230)
at io.ray.runtime.functionmanager.FunctionManager$JobFunctionTable.getFunction(FunctionManager.java:179)
at io.ray.runtime.functionmanager.FunctionManager.getFunction(FunctionManager.java:115)
at io.ray.runtime.functionmanager.FunctionManager.getFunction(FunctionManager.java:93)
at io.ray.runtime.AbstractRayRuntime.createActor(AbstractRayRuntime.java:152)
at io.ray.api.call.ActorCreator.remote(ActorCreator.java:41)
at simulator.RayBasedSimulator.executeAllTimedActors(RayBasedSimulator.java:104)
at simulator.Simulator.simulate(Simulator.java:35)
at simulator.Configuration.simulate(Configuration.java:141)
at driverLocal.Warehouse.runDriver(Warehouse.java:32)
at driverLocal.Warehouse.run(Warehouse.java:88)
at app.RetailDCApplication.run(RetailDCApplication.java:27)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
at app.RetailDCApplication.main(RetailDCApplication.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:109)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.ClassNotFoundException: actor.ProxyTimedActor
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at io.ray.runtime.functionmanager.FunctionManager$JobFunctionTable.loadFunctionsForClass(FunctionManager.java:206)
... 24 more

SanaIRnI picture SanaIRnI on 20 Nov 2020

SanaIRnI About "Check failed: object_id_refs_.count(object_id) == 0 Tried to create an owned object that already exists", does it fail this way constantly?
For this error , the application runs for few objects or few instances and then later fails in between .
But now this error has been replaced by above comment error . It was earlier using 0.9.0 ray jars. Now it is using 1.0.0

SanaIRnI picture SanaIRnI on 20 Nov 2020

@chaokunyang Is it because of https://github.com/ray-project/ray/pull/12001?

kfstorm picture kfstorm on 20 Nov 2020

Seems actor.ProxyTimedActor is not in classpath. @SanaIRnI How do you run the application? Since you are using spring boot, is actor.ProxyTimedActor packaged into spring boot jar in a spring-boot fat-jar format, thus can only be loaded by spring classloader?

chaokunyang picture chaokunyang on 20 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

robertnishihara picture robertnishihara  路  3Comments
austinmw picture austinmw  路  3Comments
xudongliao picture xudongliao  路  3Comments
dragon28 picture dragon28  路  3Comments
Khalilsqu picture Khalilsqu  路  3Comments