Orientdb: Massive Insertion using NodeJS

[hosseinGanjyar]

OrientDB Version, operating system, or hardware.

• 2.2.12

Operating System

• Windows

I using NodeJS and I want insert massive data (~8000000 records) into OrientDb.
I read Documentation, but I don't know 'How to use below code into NodeJS (JavaScript)':

db.declareIntent( new OIntentMassiveInsert() );

// YOUR MASSIVE INSERTION

db.declareIntent( null );

[StarpTech]

@luigidellaquila do you have personal contact with @hosseinGanjyar ? I would leave it open until @hosseinGanjyar can describe his problem in detail.

[luigidellaquila]

Oh, sorry, I think I closed it when it was still incomplete...

Reopening

Thanks @StarpTech !

Luigi

[luigidellaquila]

Hi @hosseinGanjyar

MassiveInsert intent is only available in Java Embedded mode, so unfortunately it's not available in Node.js (and not even in Java remote mode...)

Thanks

Luigi

[hosseinGanjyar]

@StarpTech thanks for support of me... ;)
@luigidellaquila thanks for answer, So no way for massive insert (8.000.000 records) into OrientDb without increase ram memory usage?!

[wolf4ood]

@hosseinGanjyar

if you can forward me your importer script i can help you.

Thanks

[StarpTech]

I think its time to support streaming.

[wolf4ood]

@StarpTech

Which streaming do you suggest?

i think this is not the case as it is in insert with raw document.

[hosseinGanjyar]

@maggiolo00
http://stackoverflow.com/questions/41675947/orientdb-dont-release-memory-after-insert/41694598#41694598

[hosseinGanjyar]

My code is :
```
var dbConn = [];
var dbNext = 0;
var dbMax = 25;
for (var i = 0; i <= dbMax; i++) {
var db = new ODatabase({
host: orientdb.host,
port: 2424,
username: 'root',
password: orientdb.password,
name: 'test',
});
dbConn.push(db);
}
//---------------------------------------------------
//Start loop
// record = {name: 'test'}
record["@class"] = "table";
var db = nextDB();
db.open().then(function () {
return db.record.create(record);
}).then(function (res) {
db.close().then(function () {
//----resume loop
});
}).error(function (err) {
//------
});
// end loop - iteration loop
//---------------------------------------------------
function nextDB() {
if (++dbNext >= dbMax) {
dbNext -= dbMax;
}
return dbConn[dbNext];
}

[wolf4ood]

@hosseinGanjyar
where do you starts the loop?

is the record empty?

[StarpTech]

@maggiolo00 I think its time to support streaming.

[wolf4ood]

@StarpTech

streaming in query is coming in OrientJS v3.

What about insert?

[hosseinGanjyar]

@StarpTech @maggiolo00
code is updated.

[wolf4ood]

@hosseinGanjyar
Here it's a sample code, with async library to handle async looping.
I was able to import 8M records in ~ 25 min on my box.
The ODB heap config was even 512mb.

https://gist.github.com/maggiolo00/34052268aad86ecbd318052ad85d07e9

[saeedtabrizi]

Hi @hosseinGanjyar

First of all i need to know your problem exactly .
as you said , you want to achieve to 8000000 insertion and more in the infinite time . the short answer is yes , you can in the infinite time . but the right answer is your problem definition not completed yet . so define the time , memory , cpu usage , indexing & resource conditions for your problem .

I have similar experience with orient db and i can to achive more than 100000 record per second in java environment locally ( as the @luigidellaquila said) with the some configuration exactly . (NOT THROUGH REMOTE CONNECTION like the orientjs) .

welcome to your comments here .

[hosseinGanjyar]

Hi @saeedtabrizi ,
Please see this page:
https://gist.github.com/maggiolo00/34052268aad86ecbd318052ad85d07e9

[saeedtabrizi]

hi @hosseinGanjyar as i said . your must use diffrent solution .

[saeedtabrizi]

@luigidellaquila i think you can close this issue now . i contacted with @hosseinGanjyar and explained some notes about orientdb concepts .

[luigidellaquila]

Thank you very much @saeedtabrizi , closing then

Luigi

[hosseinGanjyar]

OrientJS wasn't efficient for insert massive data from SqlServer to OrientDB. I used ETL module for massive insert, that is good idea for transpot massive data without increase memory more than 2GB.

Thanks from @saeedtabrizi for help me.

My ETL's config.json:

{
  "config": {
    log : "debug"
  },
  "extractor" : {
    "jdbc": { "driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
              "url": "jdbc:sqlserver://10.10.10.10;databaseName=My_DB;",
              "userName": "sa",
              "userPassword": "123",
              "query": "select * from My_Table" 
            }
  },

  "transformers" : [
    { "vertex": { "class": "Company"} }
  ],
   "loader" : {
    "orientdb": {
      "dbURL": "plocal:D:\DB\Orient_DB",
      dbUser: "admin",
      dbPassword: "admin",
      "dbAutoCreate": true,
      "tx": false,
      "batchCommit": 1000,
      "wal" : false,
      "dbType": "graph"
    }
  }
}

[saeedtabrizi]

@hosseinGanjyar As i said to you , if you be familiar with OrientDB Concepts the you can write efficient code and solution for your problems . wal: false and tx:false configuration is very useful for massive insertion .