Orientdb: Distributing Shards Between Servers
OrientDB Version: 2.2.33
OS: Linux
Expected behavior
I have 3 servers and use the following configs (default-distributed-db-config.json):
{
"autoDeploy": true,
"readQuorum": 1,
"writeQuorum": "majority",
"executionMode": "undefined",
"readYourWrites": true,
"newNodeStrategy": "static",
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"shard_01": {
"servers": ["n01"]
},
"shard_02": {
"servers": ["n01","n02"]
},
"shard_03": {
"servers": ["n03"]
},
"*": {
"servers": ["<NEW_NODE>"]
}
}
}
I insert some nodes (Using java api, graph model) I have 2 problem (Actually Three!)
1) I expect the nodes go to every shard according to the policy "e.g., round-robin"
2) I expect the size of database folder for 'n01' is more than the 'n02' and 'n03'
Actual behavior
Here's my problems:
First:
All the nodes go to the one server. e.g., if I use this OrientGraph graph = new OrientGraph(n01-ip) all the nodes go to the shard_01 (not even shard_02!)
My code is: graph.addVertex("class:" + className, properties)
Second:
The size of database folder for servers 'n02' and 'n03' had been less than the size of database folder server 'n01' before I closed 'n02' and 'n03'. When I shutdown the 'n02' and 'n03' and delete their databases, the database is copied from n01 to n02 and n03 and the size of all the database folders are equal.
Third:
I can't read data! I use this code: ( FYI: keyName and keyValue is unique property)
Iterator<Vertex> iterator = graph.getVertices(className + "." + keyName, keyValue).iterator();
And I get this error from my java output:
Exception in thread "main" com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select rid from index:User.idNumber where key = ?
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:449)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:400)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:283)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:167)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2362)
at com.orientechnologies.orient.client.remote.OStorageRemote$27.execute(OStorageRemote.java:1209)
at com.orientechnologies.orient.client.remote.OStorageRemote$2.execute(OStorageRemote.java:206)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:251)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetry(OStorageRemote.java:203)
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:214)
at com.orientechnologies.orient.client.remote.OStorageRemote.command(OStorageRemote.java:1185)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
at com.orientechnologies.orient.core.index.OIndexRemoteOneValue.get(OIndexRemoteOneValue.java:46)
at com.orientechnologies.orient.core.index.OIndexRemoteOneValue.get(OIndexRemoteOneValue.java:36)
at com.orientechnologies.orient.core.index.OIndexAbstractDelegate.get(OIndexAbstractDelegate.java:58)
at com.orientechnologies.orient.core.index.OIndexTxAwareOneValue.get(OIndexTxAwareOneValue.java:262)
at com.orientechnologies.orient.core.index.OIndexTxAwareOneValue.get(OIndexTxAwareOneValue.java:40)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.getVertices(OrientBaseGraph.java:780)
at graphdb.Orientdb.findVertex(Orientdb.java:10)
And this is from "dserver.sh":
11:38:42:910 WARNI I/O Error on distributed channel (clientId=8 reqType=121 error=java.io.InvalidClassException: com.orientechnologies.orient.core.sql.query.OBasicResultSet; no valid constructor)
11:38:53:001 WARNI [n01] Timeout (10001ms) on waiting for synchronous responses from nodes=[n03, n02] responsesSoFar=[n02] request=(id=0.56 task=gossip timestamp: 1524467322999 lockManagerServer: n01)
11:38:53:002 WARNI [n01]->[n03] Server 'n03' did not respond to the gossip message (db=myDB, timeout=10000ms), but cannot be set OFFLINE by configuration
Does anyone run the Orientdb in the cloud way that all the servers haven't all the shards?
Thanks
hossein-md
All 19 comments
More information!
I test this:
Using the 'Console.sh' I have connected to the 'n01' and run this query:
select * from cluster:shard_01 limit 10
The output is correct!
And I run this
select * from MyClass limit 10 (FYI: shard_01, shard_02, shard_03 are the shards of myClass)
It returns the following error:
Error: com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select * from MyClass limit 10
I found an issue that has one part of my problem! (issue: #7887)
hossein-md
on 23 Apr 2018
Hi @hossein-md ,
Probable reason to your first issue: I do not know if you have gone through this issue: https://github.com/orientechnologies/orientdb/issues/7110 where @tglman mentions about this:
From what you say it seems that you are looking for the ROUND_ROBIN_REQUEST setting instead of ROUND_ROBIN_CONNECT, the difference is that ROUND_ROBIN_CONNECT choose a server each new session is created (that in your case is limitated by the pool) instead ROUND_ROBIN_REQUEST each time you execute an remote operation (like a query).
And there is another concept in sharding which you are doing: https://orientdb.com/docs/last/Distributed-Sharding.html#cluster-locality
bipulkarnani
on 23 Apr 2018
Thanks @bipulkarnani
I think one of my problem is half-solved!
I thought I had to create the database in not-distributed mode (As https://orientdb.com/docs/last/Distributed-Architecture.html) because here it says "With releases < v2.2.6 the creation of a database on multiple nodes could cause synchronization problems when clusters are automatically created. Please create the databases before to run in distributed mode." and I thought we are in version 2.2.33 < 2.2.6 !!!!!
When I build my database in "dserver.sh" the nodes go over servers "n01" and "n02" and according to the "Cluster Locality" it seems correct. But what should I do if I want to sent the nodes into the 'n03'?
Thanks
hossein-md
on 23 Apr 2018
Hi @hossein-md ,
I also tried to rotate my connection to different nodes in cluster and saw the similar behaviour that it always connects to first node in the connection string. Hence I am not sure about how to write in node n03.
bipulkarnani
on 23 Apr 2018
@hossein-md
Just i test your needed scenario with ODB v3.0.0 and it's working normally .
I attach some output in here
I guess you have misconfiguration problem for distribution mode .
Saeed Tabrizi
saeedtabrizi
on 16 May 2018
Hi @luigidellaquila
I think this issue can be close .
Thanks
saeedtabrizi
on 16 May 2018
Hi @saeedtabrizi
Thank you for your test.
I use version 2.2.33 and I haven't tested version 3.0.0. But I share all my steps with you. Could you please tell me my misconfigured part(s)?
1) I've copied two instance of Orientdb in my PC (orientdb-community-importers-2.2.33.zip)
2) I run n01, n02 with 'server.bat' and set the password 123. the servers are stopped by using ctrl + c
Output_n01
Output_n02
3) I've changed the 'default-distributed-db-config.json':
{
"autoDeploy": true,
"readQuorum": 1,
"writeQuorum": "majority",
"executionMode": "undefined",
"readYourWrites": true,
"newNodeStrategy": "static",
"servers": {
"*": "master"
},
"clusters": {
"internal": {
},
"user":{
"servers" : [ "n01" ]
},
"user_1":{
"servers" : [ "n02" ]
},
"*": {
"servers": ["<NEW_NODE>"]
}
}
}
4) I've changed the 'hazelcast.xml'. Just the network part. (Add tcp-ip, enable = "false" for multicast) for both n01 and n02.
<network>
<port auto-increment="true">2434</port>
<join>
<multicast enabled="false">
<multicast-group>235.1.1.1</multicast-group>
<multicast-port>2434</multicast-port>
</multicast>
<tcp-ip enabled="true">
<member>localhost:2434</member>
<member>localhost:2435</member>
</tcp-ip>
</join>
</network>
5) I run 'dserver.bat'. First n01 then n02
Set n01 and n02
Node name [BLANK=auto generate it]: n01
Node name [BLANK=auto generate it]: n02
n02 output
|Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|n02(*)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2 |06:32:53 |192.168.56.1:2425|192.168.56.1:2481|107.67MB/3.56GB (2.96%)|
|n01(@)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|1 |06:31:36 |192.168.56.1:2424|192.168.56.1:2480|108.83MB/3.56GB (2.99%)|
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
2018-05-17 06:33:11:857 WARNI Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql]
2018-05-17 06:33:11:986 INFO OrientDB Studio available at http://192.168.56.1:2481/studio/index.html
2018-05-17 06:33:11:987 INFO OrientDB Server is active v2.2.33 (build 77584cd6827f647cf4aa231cf27bd6f10bc04e2c, branch 2.2.x).
n01 output
CLUSTER CONFIGURATION [wQuorum: true] (LEGEND: X = Owner, o = Copy)
+-------------+----+-----------+----------+------+------+
| | | | |MASTER|MASTER|
| | | | |ONLINE|ONLINE|
| | | | |static|static|
+-------------+----+-----------+----------+------+------+
|CLUSTER | id|writeQuorum|readQuorum| n01 | n02 |
+-------------+----+-----------+----------+------+------+
|* | | 2 | 1 | X | o |
|e_4 | 21| 2 | 1 | o | X |
|e_5 | 22| 2 | 1 | o | X |
|e_6 | 23| 2 | 1 | o | X |
|e_7 | 24| 2 | 1 | o | X |
|followed_by_1| 26| 2 | 1 | o | X |
|followed_by_2| 27| 2 | 1 | o | X |
|followed_by_3| 28| 2 | 1 | o | X |
|followed_by_4| 29| 2 | 1 | o | X |
|internal | 0| 2 | 1 | | |
|sung_by | 41| 2 | 1 | o | X |
|sung_by_1 | 42| 2 | 1 | o | X |
|sung_by_2 | 43| 2 | 1 | o | X |
|sung_by_3 | 44| 2 | 1 | o | X |
|v_3 | 12| 2 | 1 | o | X |
|v_5 | 14| 2 | 1 | o | X |
|v_6 | 15| 2 | 1 | o | X |
|v_7 | 16| 2 | 1 | o | X |
|written_by | 33| 2 | 1 | o | X |
|written_by_4 | 37| 2 | 1 | o | X |
|written_by_6 | 39| 2 | 1 | o | X |
|written_by_7 | 40| 2 | 1 | o | X |
+-------------+----+-----------+----------+------+------+
2018-05-17 06:33:12:465 INFO [n01] Distributed servers status (*=current @=lockManager[n01]):
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|n02 |ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2 |06:32:53 |192.168.56.1:2425|192.168.56.1:2481|107.67MB/3.56GB (2.96%)|
|n01(*)(@)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|3 |06:31:36 |192.168.56.1:2424|192.168.56.1:2480|70.22MB/3.56GB (1.93%) |
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
6) Create a database using studio
CLUSTER CONFIGURATION [wQuorum: true] (LEGEND: X = Owner, o = Copy)
+--------+----+-----------+----------+------+------+
| | | | |MASTER|MASTER|
| | | | |ONLINE|ONLINE|
| | | | |static|static|
+--------+----+-----------+----------+------+------+
|CLUSTER | id|writeQuorum|readQuorum| n01 | n02 |
+--------+----+-----------+----------+------+------+
|* | | 2 | 1 | X | o |
|e_4 | 21| 2 | 1 | o | X |
|e_5 | 22| 2 | 1 | o | X |
|e_6 | 23| 2 | 1 | o | X |
|e_7 | 24| 2 | 1 | o | X |
|internal| 0| 2 | 1 | | |
|user | | 2 | 1 | X | |
|user_1 | | 2 | 1 | | X |
|v_3 | 12| 2 | 1 | o | X |
|v_5 | 14| 2 | 1 | o | X |
|v_6 | 15| 2 | 1 | o | X |
|v_7 | 16| 2 | 1 | o | X |
+--------+----+-----------+----------+------+------+
2018-05-17 06:41:22:478 INFO [n01] Distributed servers status (*=current @=lockManager[n01]):
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|n02 |ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2 |06:32:53 |192.168.56.1:2425|192.168.56.1:2481|138.76MB/3.56GB (3.81%)|
| | |MyDB=ONLINE (MASTER) | | | | | |
|n01(*)(@)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|6 |06:31:36 |192.168.56.1:2424|192.168.56.1:2480|73.57MB/3.56GB (2.02%) |
| | |MyDB=ONLINE (MASTER) | | | | | |
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
CLUSTER CONFIGURATION [wQuorum: true] (LEGEND: X = Owner, o = Copy)
+--------+----+-----------+----------+------+-------------+
| | | | |MASTER| MASTER |
| | | | |ONLINE|SYNCHRONIZING|
| | | | |static| static |
+--------+----+-----------+----------+------+-------------+
|CLUSTER | id|writeQuorum|readQuorum| n01 | n02 |
+--------+----+-----------+----------+------+-------------+
|* | | 2 | 1 | X | o |
|internal| 0| 2 | 1 | | |
|user | | 2 | 1 | X | |
|user_1 | | 2 | 1 | | X |
+--------+----+-----------+----------+------+-------------+
2018-05-17 06:41:14:526 INFO [n02] Installed database 'MyDB' (LSN=LSN{segment=6, position=52})
2018-05-17 06:41:14:529 INFO [n02] Publishing ONLINE status for database n02.MyDB...
2018-05-17 06:41:14:535 INFO [n02] Reassigning ownership of clusters for database MyDB...
2018-05-17 06:41:14:541 INFO [n02] Reassignment of clusters for database 'MyDB' completed (classes=10)
2018-05-17 06:41:14:545 INFO [n02] Setting new distributed configuration for database: MyDB (version=13)
CLUSTER CONFIGURATION [wQuorum: true] (LEGEND: X = Owner, o = Copy)
+--------+-----------+----------+------+------+
| | | |MASTER|MASTER|
| | | |ONLINE|ONLINE|
| | | |static|static|
+--------+-----------+----------+------+------+
|CLUSTER |writeQuorum|readQuorum| n01 | n02 |
+--------+-----------+----------+------+------+
|* | 2 | 1 | X | o |
|e_4 | 2 | 1 | o | X |
|e_5 | 2 | 1 | o | X |
|e_6 | 2 | 1 | o | X |
|e_7 | 2 | 1 | o | X |
|internal| 2 | 1 | | |
|user | 2 | 1 | X | |
|user_1 | 2 | 1 | | X |
|v_3 | 2 | 1 | o | X |
|v_5 | 2 | 1 | o | X |
|v_6 | 2 | 1 | o | X |
|v_7 | 2 | 1 | o | X |
+--------+-----------+----------+------+------+
2018-05-17 06:41:14:551 INFO [n02] Broadcasting new distributed configuration for database: MyDB (version=13)
2018-05-17 06:41:14:557 INFO [n02]<-[n01] Received new status n02.MyDB=SYNCHRONIZING
2018-05-17 06:41:14:562 INFO [n02]<-[n01] Received updated status n01.MyDB=BACKUP
2018-05-17 06:41:14:569 INFO [n02]<-[n01] Received updated status n01.MyDB=ONLINE
2018-05-17 06:41:14:571 INFO [n02] Received updated status n02.MyDB=ONLINE
2018-05-17 06:41:22:478 INFO [n02] Distributed servers status (*=current @=lockManager[n01]):
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|n02(*)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2 |06:32:53 |192.168.56.1:2425|192.168.56.1:2481|138.76MB/3.56GB (3.81%)|
| | |MyDB=ONLINE (MASTER) | | | | | |
|n01(@)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|4 |06:31:36 |192.168.56.1:2424|192.168.56.1:2480|48.57MB/3.56GB (1.33%) |
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
2018-05-17 06:41:32:117 INFO [n02] Distributed servers status (*=current @=lockManager[n01]):
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|n02(*)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2 |06:32:53 |192.168.56.1:2425|192.168.56.1:2481|139.09MB/3.56GB (3.82%)|
| | |MyDB=ONLINE (MASTER) | | | | | |
|n01(@)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|6 |06:31:36 |192.168.56.1:2424|192.168.56.1:2480|73.57MB/3.56GB (2.02%) |
| | |MyDB=ONLINE (MASTER) | | | | | |
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
md5-537f6bfbfd939034d822552f0e47c36f
orientdb> connect remote:localhost:2424/MyDB root 123
Disconnecting from the database [null]...OK
Connecting to database [remote:localhost:2424/MyDB] with user 'root'...OK
CONFIGURED SERVERS
+----+----+------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
|# |Name|Status|Connections|StartedOn |Binary |HTTP |UsedMemory |FreeMemory |MaxMemory|
+----+----+------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
|0 |n02 |ONLINE|2 |2018-05-17 06:32:53|192.168.56.1:2425|192.168.56.1:2481|149.36MB (4.10%)|152.14MB (4.18%)|3.56GB |
|1 |n01 |ONLINE|5 |2018-05-17 06:31:36|192.168.56.1:2424|192.168.56.1:2480|89.34MB (2.45%) |226.66MB (6.23%)|3.56GB |
+----+----+------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
md5-ea2d860a03b212dee73b7fbd9dff7f4f
orientdb {db=MyDB}> ALTER DATABASE MINIMUMCLUSTERS 2
Database updated successfully.
orientdb {db=MyDB}> create class User extends V
Class created successfully. Total classes in database now: 12.
orientdb {db=MyDB}> create property User.name string
Property created successfully with id=1.
orientdb {db=MyDB}> clusters
CLUSTERS (collections)
+----+---------+----+---------+-----+------------+-------------+--------------------+
|# |NAME | ID|CLASS |COUNT|OWNER_SERVER|OTHER_SERVERS|AUTO_DEPLOY_NEW_NODE|
+----+---------+----+---------+-----+------------+-------------+--------------------+
|0 |_studio | 25|_studio | 1| n01 | [n02] | true |
|1 |default | 3| | 0| n01 | [n02] | true |
|2 |e | 17|E | 0| n01 | [n02] | true |
|3 |e_1 | 18|E | 0| n01 | [n02] | true |
|4 |e_2 | 19|E | 0| n01 | [n02] | true |
|5 |e_3 | 20|E | 0| n01 | [n02] | true |
|6 |e_4 | 21|E | 0| n02 | [n01] | true |
|7 |e_5 | 22|E | 0| n02 | [n01] | true |
|8 |e_6 | 23|E | 0| n02 | [n01] | true |
|9 |e_7 | 24|E | 0| n02 | [n01] | true |
|10 |index | 1| | 0| n01 | [n02] | true |
|11 |internal | 0| | 3| | | |
|12 |manindex | 2| | 0| n01 | [n02] | true |
|13 |ofunction| 6|OFunction| 0| n01 | [n02] | true |
|14 |orole | 4|ORole | 3| n01 | [n02] | true |
|15 |oschedule| 8|OSchedule| 0| n01 | [n02] | true |
|16 |osequence| 7|OSequence| 0| n01 | [n02] | true |
|17 |ouser | 5|OUser | 3| n01 | [n02] | true |
|18 |user | 26|User | 0| n01 | | false |
|19 |user_1 | 27|User | 0| n02 | | false |
|20 |v | 9|V | 0| n01 | [n02] | true |
|21 |v_1 | 10|V | 0| n01 | [n02] | true |
|22 |v_2 | 11|V | 0| n01 | [n02] | true |
|23 |v_3 | 12|V | 0| n02 | [n01] | true |
|24 |v_4 | 13|V | 0| n01 | [n02] | true |
|25 |v_5 | 14|V | 0| n02 | [n01] | true |
|26 |v_6 | 15|V | 0| n02 | [n01] | true |
|27 |v_7 | 16|V | 0| n02 | [n01] | true |
+----+---------+----+---------+-----+------------+-------------+--------------------+
| |TOTAL | | | 10| | | |
+----+---------+----+---------+-----+------------+-------------+--------------------+
md5-48337992d0290022ff5f16c334362dfa
orientdb {db=MyDB}> insert into User set name = 'name01'
Inserted record 'User#26:0{name:name01} v1' in 0.004000 sec(s).
orientdb {db=MyDB}> insert into User set name = 'name02'
Inserted record 'User#26:1{name:name02} v1' in 0.002000 sec(s).
orientdb {db=MyDB}> clusters
CLUSTERS (collections)
+----+---------+----+---------+-----+------------+-------------+--------------------+
|# |NAME | ID|CLASS |COUNT|OWNER_SERVER|OTHER_SERVERS|AUTO_DEPLOY_NEW_NODE|
+----+---------+----+---------+-----+------------+-------------+--------------------+
|0 |_studio | 25|_studio | 1| n01 | [n02] | true |
|1 |default | 3| | 0| n01 | [n02] | true |
|2 |e | 17|E | 0| n01 | [n02] | true |
|3 |e_1 | 18|E | 0| n01 | [n02] | true |
|4 |e_2 | 19|E | 0| n01 | [n02] | true |
|5 |e_3 | 20|E | 0| n01 | [n02] | true |
|6 |e_4 | 21|E | 0| n02 | [n01] | true |
|7 |e_5 | 22|E | 0| n02 | [n01] | true |
|8 |e_6 | 23|E | 0| n02 | [n01] | true |
|9 |e_7 | 24|E | 0| n02 | [n01] | true |
|10 |index | 1| | 0| n01 | [n02] | true |
|11 |internal | 0| | 3| | | |
|12 |manindex | 2| | 0| n01 | [n02] | true |
|13 |ofunction| 6|OFunction| 0| n01 | [n02] | true |
|14 |orole | 4|ORole | 3| n01 | [n02] | true |
|15 |oschedule| 8|OSchedule| 0| n01 | [n02] | true |
|16 |osequence| 7|OSequence| 0| n01 | [n02] | true |
|17 |ouser | 5|OUser | 3| n01 | [n02] | true |
|18 |user | 26|User | 2| n01 | | false |
|19 |user_1 | 27|User | 0| n02 | | false |
|20 |v | 9|V | 0| n01 | [n02] | true |
|21 |v_1 | 10|V | 0| n01 | [n02] | true |
|22 |v_2 | 11|V | 0| n01 | [n02] | true |
|23 |v_3 | 12|V | 0| n02 | [n01] | true |
|24 |v_4 | 13|V | 0| n01 | [n02] | true |
|25 |v_5 | 14|V | 0| n02 | [n01] | true |
|26 |v_6 | 15|V | 0| n02 | [n01] | true |
|27 |v_7 | 16|V | 0| n02 | [n01] | true |
+----+---------+----+---------+-----+------------+-------------+--------------------+
| |TOTAL | | | 12| | | |
+----+---------+----+---------+-----+------------+-------------+--------------------+
md5-309380130144f13a5918eb74a2cc7c46
orientdb {db=MyDB}> connect remote:localhost:2425/MyDB root 123
Disconnecting from the database [MyDB]...OK
Connecting to database [remote:localhost:2425/MyDB] with user 'root'...OK
CONFIGURED SERVERS
+----+----+------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
|# |Name|Status|Connections|StartedOn |Binary |HTTP |UsedMemory |FreeMemory |MaxMemory|
+----+----+------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
|0 |n02 |ONLINE|2 |2018-05-17 06:32:53|192.168.56.1:2425|192.168.56.1:2481|182.65MB (5.02%)|118.85MB (3.26%)|3.56GB |
|1 |n01 |ONLINE|6 |2018-05-17 06:31:36|192.168.56.1:2424|192.168.56.1:2480|141.34MB (3.88%)|174.66MB (4.80%)|3.56GB |
+----+----+------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
md5-0157fb70c96b171bd87818fc189b3ef1
orientdb {db=MyDB}> insert into User set name = 'name03'
Inserted record 'User#27:0{name:name03} v1' in 0.009000 sec(s).
orientdb {db=MyDB}> insert into User set name = 'name04'
Inserted record 'User#27:1{name:name04} v1' in 0.002000 sec(s).
orientdb {db=MyDB}> insert into User set name = 'name05'
Inserted record 'User#27:2{name:name05} v1' in 0.002000 sec(s).
orientdb {db=MyDB}> clusters
CLUSTERS (collections)
+----+---------+----+---------+-----+------------+-------------+--------------------+
|# |NAME | ID|CLASS |COUNT|OWNER_SERVER|OTHER_SERVERS|AUTO_DEPLOY_NEW_NODE|
+----+---------+----+---------+-----+------------+-------------+--------------------+
|0 |_studio | 25|_studio | 1| n01 | [n02] | true |
|1 |default | 3| | 0| n01 | [n02] | true |
|2 |e | 17|E | 0| n01 | [n02] | true |
|3 |e_1 | 18|E | 0| n01 | [n02] | true |
|4 |e_2 | 19|E | 0| n01 | [n02] | true |
|5 |e_3 | 20|E | 0| n01 | [n02] | true |
|6 |e_4 | 21|E | 0| n02 | [n01] | true |
|7 |e_5 | 22|E | 0| n02 | [n01] | true |
|8 |e_6 | 23|E | 0| n02 | [n01] | true |
|9 |e_7 | 24|E | 0| n02 | [n01] | true |
|10 |index | 1| | 0| n01 | [n02] | true |
|11 |internal | 0| | 3| | | |
|12 |manindex | 2| | 0| n01 | [n02] | true |
|13 |ofunction| 6|OFunction| 0| n01 | [n02] | true |
|14 |orole | 4|ORole | 3| n01 | [n02] | true |
|15 |oschedule| 8|OSchedule| 0| n01 | [n02] | true |
|16 |osequence| 7|OSequence| 0| n01 | [n02] | true |
|17 |ouser | 5|OUser | 3| n01 | [n02] | true |
|18 |user | 26|User | 0| n01 | | false |
|19 |user_1 | 27|User | 3| n02 | | false |
|20 |v | 9|V | 0| n01 | [n02] | true |
|21 |v_1 | 10|V | 0| n01 | [n02] | true |
|22 |v_2 | 11|V | 0| n01 | [n02] | true |
|23 |v_3 | 12|V | 0| n02 | [n01] | true |
|24 |v_4 | 13|V | 0| n01 | [n02] | true |
|25 |v_5 | 14|V | 0| n02 | [n01] | true |
|26 |v_6 | 15|V | 0| n02 | [n01] | true |
|27 |v_7 | 16|V | 0| n02 | [n01] | true |
+----+---------+----+---------+-----+------------+-------------+--------------------+
| |TOTAL | | | 13| | | |
+----+---------+----+---------+-----+------------+-------------+--------------------+
md5-82af4945970f7b77503fa80708fd761d
orientdb {db=MyDB}> select * from cluster:user_1
+----+-----+------+------+
|# |@RID |@CLASS|name |
+----+-----+------+------+
|0 |#27:0|User |name03|
|1 |#27:1|User |name04|
|2 |#27:2|User |name05|
+----+-----+------+------+
3 item(s) found. Query executed in 0.003 sec(s).
orientdb {db=MyDB}> select from User
Error: com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select from User
orientdb {db=MyDB}>
md5-c0ac5f779bff08d66190ec6b106b9de2
2018-05-17 06:41:32:117 INFO [n02] Distributed servers status (*=current @=lockManager[n01]):
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|n02(*)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2 |06:32:53 |192.168.56.1:2425|192.168.56.1:2481|139.09MB/3.56GB (3.82%)|
| | |MyDB=ONLINE (MASTER) | | | | | |
|n01(@)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|6 |06:31:36 |192.168.56.1:2424|192.168.56.1:2480|73.57MB/3.56GB (2.02%) |
| | |MyDB=ONLINE (MASTER) | | | | | |
+------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
2018-05-17 07:02:04:854 WARNI I/O Error on distributed channel (clientId=5 reqType=121 error=java.io.InvalidClassException: com.orientechnologies.orient.core.s
l.query.OBasicResultSet; no valid constructor)
2018-05-17 07:04:04:852 WARNI I/O Error on distributed channel (clientId=2 reqType=121 error=java.io.InvalidClassException: com.orientechnologies.orient.core.s
l.query.OBasicResultSet; no valid constructor)
2018-05-17 07:06:04:845 WARNI [n02] Timeout (120001ms) on waiting for synchronous responses from nodes=[n01] responsesSoFar=[] request=(id=1.313 task=command_s
l(select from User))
md5-b0e4bd08355c58dc1cb2579c1d3e7666
2018-05-17 06:41:22:478 INFO [n01] Distributed servers status (*=current @=lockManager[n01]):
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name |Status|Databases |Conns|StartedOn|Binary |HTTP |UsedMemory |
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|n02 |ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2 |06:32:53 |192.168.56.1:2425|192.168.56.1:2481|138.76MB/3.56GB (3.81%)|
| | |MyDB=ONLINE (MASTER) | | | | | |
|n01(*)(@)|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|6 |06:31:36 |192.168.56.1:2424|192.168.56.1:2480|73.57MB/3.56GB (2.02%) |
| | |MyDB=ONLINE (MASTER) | | | | | |
+---------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
2018-05-17 07:06:05:098 WARNI [n01]->[n02] Error on sending message to distributed node (java.net.SocketException: Software caused connection abort: socket writ
e error) retrying (1/3)
md5-7b96134833f21b89e045219b6ade6ec1
orientdb {db=MyDB}> select * from cluster:user
Error: com.orientechnologies.orient.server.distributed.ODistributedException: No active nodes found to execute command: sql.select * from cluster:user
orientdb {db=MyDB}>
md5-c0ac5f779bff08d66190ec6b106b9de2
2018-05-17 07:14:50:365 WARNI I/O Error on distributed channel (clientId=6 reqType=121 error=java.io.InvalidClassException: com.orientechnologies.orient.core.sq
l.query.OBasicResultSet; no valid constructor)
2018-05-17 07:15:05:166 WARNI [n02] Timeout (10001ms) on waiting for synchronous responses from nodes=[n01] responsesSoFar=[] request=(id=1.421 task=gossip time
stamp: 1526525095165 lockManagerServer: n01)
2018-05-17 07:15:05:166 WARNI [n02]->[n01] Server 'n01' did not respond to the gossip message (db=GratefulDeadConcerts, timeout=10000ms), but cannot be set OFFL
INE by configuration
md5-b0e4bd08355c58dc1cb2579c1d3e7666
2018-05-17 07:06:05:098 WARNI [n01]->[n02] Error on sending message to distributed node (java.net.SocketException: Software caused connection abort: socket writ
e error) retrying (1/3)
2018-05-17 07:15:05:167 WARNI [n01]->[n02] Error on sending message to distributed node (java.net.SocketException: Software caused connection abort: socket writ
e error) retrying (1/3)
Thanks.
hossein-md
on 17 May 2018
Hi @hossein-md
Please send your configured solution as zip file to my personal email as you know.
Note: Please don't send large or any sensitive data by email .
Thanks
saeedtabrizi
on 17 May 2018
Hi @saeedtabrizi,
I sent it to your email.
Thanks
hossein-md
on 17 May 2018
@hossein-md ,
I checked your database and i found a problem with write quorum value and also a problem to find LSN in your distributed database . (LSN means Log Sequence Number )
Just when i recreate the database and set right quorum value , database syncing and i can retrieve all or part of data from any cluster that be online .
If you feel this answer is enough for you , please close this issue .
@lvca, @laa This issue occurred when the LSN not recovered truly after a failed transaction in each node . so after any CRUD operation an exception trowed in console .
Thanks
Saeed
saeedtabrizi
on 17 May 2018
Hi @saeedtabrizi,
I tested the writequroum = 1 and I got the same errors.
Also I tested the writequroum = 1 and readquroum = 2 and in this case the query select * from cluster:user and select * from user didn't work.
Thanks
hossein-md
on 19 May 2018
Hi @hossein-md
I will send an email about my tested instances asap.
saeedtabrizi
on 19 May 2018
@saeedtabrizi I tried the scenario that u implemented, because i had the same problem as @hossein-md,
It works when i use select from cluster:myclass_a1 but when i use select * from myclass it throws
Error: com.orientechnologies.orient.enterprise.channel.binary.OResponseProcessingException: Exception during response processing
Error: java.lang.ArrayIndexOutOfBoundsException: -1
HassounDev
on 6 Aug 2018
Hey @saeedtabrizi, how did you reach that the records are stored on the local cluster? That does not work for me... It stores all records to a specific node, even if the local node owns the matching cluster...
berevesero
on 8 Aug 2018
Hi guys!
@adler4566 I think I also encountered this problem. I was trying to keep only one cluster in each node that will also be the owner of that cluster. The idea was that none of the nodes should have locally the entire database, only the cluster they own.
I didn't manage to achieve this. If I use autoDeploy: true in distributed-config.json each node is always syncing the entire database, not just its cluster.
georgiana-b
on 9 Aug 2018
Hey @georgiana-b, this is very interesting, because I have exactly the same target as you! But why does this happen? This can鈥榯 be the desired behavour of a distributed database... I work with big data, but with this behavour I can鈥榯 use this database. Maybe someone encountered the same problem and reached to fix it?
berevesero
on 9 Aug 2018
Hi @luigidellaquila!
Just to know if it's worth looking more into this issue, does this feature exist in OrientDB? Is it possible to shard the database so that each node keeps only one cluster locally not the entire database?
If not, are you guys interested in developing this?
georgiana-b
on 21 Aug 2018
@georgiana-b It might be worth taking a look at these issues:
- https://github.com/orientechnologies/orientdb/issues/8015
- https://github.com/orientechnologies/orientdb/issues/6256
They both very much relate to this topic. One thing to consider is that while you definitely want to be able to either distribute data across clusters, and / or clusters across the server in some distributed hash table method, you may ALSO want some sort of replication factor in case you loose a server. So you might want each cluster to exist on N servers where N >= 1. That way if you loose a single server you don't loose availability of data.
mmacfadden
on 25 Sep 2018
Related issues
saeedtabrizi
路
3Comments
PandaMonkey
路
3Comments
andreyvk
路
4Comments
bbourgois
路
3Comments
saeedtabrizi
路
5Comments
Most helpful comment
Hi @hossein-md
I will send an email about my tested instances asap.