Orientdb: Error on creating ETL processor

Created on 23 Jan 2017  路  8Comments  路  Source: orientechnologies/orientdb

Hi, I'm trying to upload graph from JSON with JSON ETL.
System parameters: OrientDB 2.2.14, ubuntu 14.04 64 bit, JAVA 1.8

I am gating the followings errors -
Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on creating ETL processor
DB name="plocal:/tmp/databases/testdb"
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.java:245)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.java:186)
at com.orientechnologies.orient.etl.OETLProcessor.parseConfigAndParameters(OETLProcessor.java:176)
at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:140)
Caused by: java.lang.NullPointerException
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.get(ODatabaseDocumentTx.java:1468)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.readDatabaseConfiguration(OrientBaseGraph.java:1851)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.(OrientBaseGraph.java:146)
at com.tinkerpop.blueprints.impls.orient.OrientGraphNoTx.(OrientGraphNoTx.java:62)
at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory$2.getGraph(OrientGraphFactory.java:116)
at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory.getNoTx(OrientGraphFactory.java:240)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configureGraphDB(OOrientDBLoader.java:407)
at com.orientechnologies.orient.etl.loader.OOrientDBLoader.configure(OOrientDBLoader.java:345)
at com.orientechnologies.orient.etl.OETLProcessor.configureComponent(OETLProcessor.java:484)
at com.orientechnologies.orient.etl.OETLProcessor.configureLoader(OETLProcessor.java:297)
at com.orientechnologies.orient.etl.OETLProcessor.parse(OETLProcessor.java:229)
... 3 more

bug waiting reply
Source
picture ahr12

Most helpful comment

Just pushed the fix, you can download the etl module 2.2.16-SNAPSHOT and drop it inside the lib folder. Delete the 2.2.15 jar before!

https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-etl/2.2.16-SNAPSHOT/

Please close the issue if the fix solves your problem

picture robfrank on 30 Jan 2017
🎉1 👍1

All 8 comments

Could you provide the configuration Json and some sample data?

robfrank picture robfrank on 23 Jan 2017

configuration JSON file

{
"config": {
"log": "debug"
},
"source" : {
"file": { "path": "/tmp/test.json" }
},
"extractor" : {
"json": {}
},
"transformers" : [
{ "merge": { "joinFieldName": "id", "lookup": "Account.id" } },
{ "vertex": { "class": "Account"} },
{ "edge": {
"class": "Friend",
"joinFieldName": "friends",
"lookup": "Account.id",
"unresolvedLinkAction": "CREATE"
} },
{ "edge": {
"class": "Enemy",
"joinFieldName": "enemies",
"lookup": "Account.id",
"unresolvedLinkAction": "CREATE"
} }
],
"loader" : {
"orientdb": {
"dbURL": "plocal:/tmp/databases/testdb",
"dbUser": "admin",
"dbPassword": "admin",
"dbAutoDropIfExists": true,
"dbAutoCreate": true,
"standardElementConstraints": false,
"tx": false,
"wal": false,
"batchCommit": 1000,
"dbType": "graph",
"classes": [{"name": "Account", "extends":"V"}, {"name": "Friend", "extends":"E"}, {"name": 'Enemy', "extends":"E"}],
"indexes": [{"class":"Account", "fields":["id:integer"], "type":"UNIQUE_HASH_INDEX" }]
}
}
}

Sample data - test.json

[
{
"name": "Joe",
"id": 1,
"friends": [2,4,5],
"enemies": [6]
},
{
"name": "Suzie",
"id": 2,
"friends": [1,4,6],
"enemies": [5,2]
}
]

ahr12 picture ahr12 on 23 Jan 2017

Hi @robfrank , i have same problem in odb 2.2.15

osm data can take from OpenStreetMap as test.osm

configuration :
{ "config": { "log": "debug" }, "begin": [ { "let": { "name": "$filePath", "value": "6565" } }, { "let": { "name": "$className", "value": "hello" } } ], "source": { "file": { "path": "/mnt/hdd2/test.osm" } }, "extractor": { "xml": { "rootNode": "osm.node" } }, "transformers": [ { "vertex": { "class": "Osm_Node" } } ], "loader": { "output": {} } }

my output log :


 OrientDB etl v.2.2.15 (build 2.2.x@rebf7b80c08613e871633edeb39c5c985e26b176c; 2017-01-19
07:39:00+0000) www.orientdb.com
BEGIN ETL PROCESSOR
[file] INFO Reading from file /mnt/hdd2/test.osm with encoding UTF-8
Started execution with 1 worker threads
ETL process has problem: java.lang.NullPointerException
END ETL PROCESSOR
+ extracted 0 entries (0 entries/sec) - 0 entries -> loaded 0 bytes (0 bytes/sec) Total t
ime: 77ms [0 warnings, 0 errors]

but wen i use following configuration , everything is ok :

{
    "config": {
        "log": "debug"
    },
    "begin": [
        {
            "let": {
                "name": "$filePath",
                "value": "6565"
            }
        },
        {
            "let": {
                "name": "$className",
                "value": "hello"
            }
        }
    ],
    "source": {
        "file": {
            "path": "/mnt/hdd2/test.osm"
        }
    },
    "extractor": {
        "xml": {
            "rootNode": "osm.node"
        }
    },
    "transformers": [

    ],
    "loader": {
        "output": {}
    }
}

output log : 


OrientDB etl v.2.2.15 (build 2.2.x@rebf7b80c08613e871633edeb39c5c985e26b176c; 2017-01-19 07:39:00+0000) www.orientdb.com
BEGIN ETL PROCESSOR
[file] INFO Reading from file /mnt/hdd2/test.osm with encoding UTF-8
Started execution with 1 worker threads
Start extracting
{changeset:45332095,id:4620980938,lat:35.7086218,lon:51.4109069,timestamp:2017-01-20T17:20:29Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979384,lat:35.7116648,lon:51.4105953,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979386,lat:35.7111156,lon:51.4106510,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979387,lat:35.7110695,lon:51.4100838,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620979388,lat:35.7106872,lon:51.4106944,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980189,lat:35.7106470,lon:51.4100919,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980190,lat:35.7105810,lon:51.4095878,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980191,lat:35.7098157,lon:51.4107827,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980192,lat:35.7098283,lon:51.4112547,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:45332039,id:4620980195,lat:35.7092527,lon:51.4108399,timestamp:2017-01-20T17:18:10Z,uid:3360520,user:madjidf,version:1,visible:true}
{changeset:42880693,id:2021254176,lat:35.7105896,lon:51.4092324,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:2021254177,lat:35.7112705,lon:51.4091419,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:2021254178,lat:35.7104456,lon:51.4068835,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:4,visible:true}
{changeset:42880693,id:2021254175,lat:35.7105020,lon:51.4083992,timestamp:2016-10-13T20:24:07Z,uid:4449060,user:Khalil Laleh,version:4,visible:true}
{changeset:35751999,id:3876630879,lat:35.7112496,lon:51.4088079,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630880,lat:35.7110050,lon:51.4088303,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630881,lat:35.7112280,lon:51.4084634,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630882,lat:35.7105069,lon:51.4084457,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630883,lat:35.7113056,lon:51.4097014,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:35751999,id:3876630884,lat:35.7106065,lon:51.4097744,timestamp:2015-12-04T17:04:19Z,uid:25975,user:AmZaf,version:1,visible:true}
{changeset:36303811,id:3922337347,lat:35.7106629,lon:51.4081717,timestamp:2016-01-01T18:40:46Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:36303811,id:3922337348,lat:35.7106651,lon:51.4082522,timestamp:2016-01-01T18:40:46Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:36303811,id:3922337349,lat:35.7105148,lon:51.4082522,timestamp:2016-01-01T18:40:47Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:36303811,id:3922337350,lat:35.7104936,lon:51.4081726,timestamp:2016-01-01T18:40:47Z,uid:3012890,user:HADiMozafari,version:1,visible:true}
{changeset:38168129,id:4086609684,lat:35.7108022,lon:51.4084778,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:38168129,id:4086609685,lat:35.7108057,lon:51.4087093,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:38168129,id:4086609686,lat:35.7111975,lon:51.4087007,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:38168129,id:4086609687,lat:35.7111852,lon:51.4084887,timestamp:2016-03-30T14:29:32Z,uid:3312986,user:Amir Ahmad Mohamadi,version:1,visible:true}
{changeset:42880322,id:1649546841,lat:35.7112249,lon:51.4072513,timestamp:2016-10-13T20:07:26Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748508,lat:35.7086936,lon:51.4091692,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748523,lat:35.7087626,lon:51.4092329,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748528,lat:35.7088236,lon:51.4108834,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748534,lat:35.7088531,lon:51.4107725,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748574,lat:35.7093578,lon:51.4091232,timestamp:2016-10-13T20:13:28Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748594,lat:35.7098728,lon:51.4067574,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:4,visible:true}
{changeset:42880693,id:1730748608,lat:35.7099457,lon:51.4091273,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748628,lat:35.7100022,lon:51.4090786,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748632,lat:35.7100362,lon:51.4106669,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748634,lat:35.7100960,lon:51.4107543,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748641,lat:35.7101567,lon:51.4117777,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:44853153,id:1730748428,lat:35.7081525,lon:51.4109616,timestamp:2017-01-02T19:16:20Z,uid:3360520,user:madjidf,version:3,visible:true}
{changeset:42880693,id:1730748734,lat:35.7111627,lon:51.4074215,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748748,lat:35.7113635,lon:51.4106259,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748800,lat:35.7120109,lon:51.4105602,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748814,lat:35.7122042,lon:51.4105406,timestamp:2016-10-13T20:13:29Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:42880693,id:1730748893,lat:35.7127288,lon:51.4104874,timestamp:2016-10-13T20:13:30Z,uid:4449060,user:Khalil Laleh,version:2,visible:true}
{changeset:44340601,id:4456355194,lat:35.7108694,lon:51.4109623,timestamp:2016-12-12T09:21:40Z,uid:4449060,user:Khalil Laleh,version:2,visible:true,tag:[2]}
Pipeline worker done without errors:: true
all items extracted
END ETL PROCESSOR
+ extracted 47 entries (0 entries/sec) - 47 entries -> loaded 47 bytes (0 bytes/sec) Total time: 65ms [0 warnings, 0 errors]

what can i do ?

saeedtabrizi picture saeedtabrizi on 23 Jan 2017

@saeedtabrizi can you see where the NPE originated from?

lvca picture lvca on 27 Jan 2017

Hi @lvca as i explained in https://github.com/orientechnologies/orientdb/issues/7107#issuecomment-275579245 , after some tries and spending some hours , now i can to transfer my data from .osm xml files to orientdb but i realized the documentaion of ETL is not suitable and true .
I recommend review the documentation of ETL again .

but in the title of this issue , i found if you dont use orientdb as loader or using output as a loader you got this this exception as i explaind .

Thanks
Saeed

saeedtabrizi picture saeedtabrizi on 27 Jan 2017

Hi, I replicated the problem. The workaround is to avoid "dbAutoDropIfExists": true,

I tested with previous versions and the same problem is there since a long time. I'll fix it for next release.

robfrank picture robfrank on 30 Jan 2017
👍1

Just pushed the fix, you can download the etl module 2.2.16-SNAPSHOT and drop it inside the lib folder. Delete the 2.2.15 jar before!

https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-etl/2.2.16-SNAPSHOT/

Please close the issue if the fix solves your problem

robfrank picture robfrank on 30 Jan 2017
🎉1 👍1

Many Thanks @robfrank . i will check it .

saeedtabrizi picture saeedtabrizi on 31 Jan 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

saeedtabrizi picture saeedtabrizi  路  3Comments
electricjones picture electricjones  路  3Comments
tglman picture tglman  路  5Comments
fpertl picture fpertl  路  4Comments
careerscale picture careerscale  路  4Comments