Azure-cosmos-dotnet-v2: Question: bulk insert graph

[eosfor]

Hello colleagues,

Is there any option to bulk insert the graph, or import from GraphML or DOT?

Thanks

[olivertowers]

A bulk import .net library for COSMOSDB graph is in the works. Can't give a firm ETA, but expecting a preview release by the end of December.

[burikella]

We're successfully using https://docs.microsoft.com/en-us/azure/cosmos-db/import-data for importing graph.

[eosfor]

Thanks very much!

[grahamehorner]

I’ve been attempting to import the OSM dataset into a graph without success, I believe a batch feature would help as the number of RUs would as I understand it be reduced as a result

FYI https://planet.openstreetmap.org/

[mcharters]

@olivertowers is there a preview release of a bulk import tool yet? If so, can I preview it please? 😄

[Raghu-Gurrala]

@burikella I'm trying to bulk load a csv file into CosmosDB, I'm able to insert vertex's but got stuck on creating edges between vertex's. Thanks in advance.

[burikella]

@Raghu-Gurrala, yes, I'm using json. Each edge stored as separate document so you need to inser it. It looks like following:
{"label":"<edge-label>","id":"<edge-id>","_sink":"<target-vertex-id>","_sinkLabel":"<target-vertex-label>","_vertexId":"<source-vertex-id>","_vertexLabel":"<source-vertex-label>","_isEdge":true}

[mcharters]

Has anyone successfully bulk inserted graph JSON with partition keys? I was able to upload edges and vertices through the bulk insert tool but things broke down when I tried to add partitioning into the mix.

[jayantacs]

@mcharters @Raghu-Gurrala : Please try this for Importing graph. It has partition key support https://github.com/Microsoft/Microsoft.Azure.Graphs.BulkImport

It's in preview and would be released soon. I have given you access to the repository.

[QuantMonk]

@jayantacs The link seems to be broken (I see 404). Can you please check?

@mcharters I could get it working with Partition keys but the throughput hasn't increased a lot (vs single partition)

[gauravhub]

@jayantacs I am too getting 404 error. Can you please update the link and provide access to me as well?

[jayantacs]

@gauravhub @QuantMonk : Can you check now.

[riparkms]

@jayantacs - any chance you can add me as well?

[jayantacs]

@riparkms : You should be good to go now!!

[QuantMonk]

Can confirm that I can access now. Thank you @jayantacs

[zebarooo]

@jayantacs - can you add me too please?

[gauravhub]

thanks @jayantacs, confirmed that I can access now.

[pdekkers]

@jayantacs - can you add me too please?

[punitkshah]

@jayantacs : can you provide me an access as well?

[jayantacs]

@punitkshah @pdekkers : Done.

[zebarooo]

@jayantacs i'm still seeing a 404 :(

[punitkshah]

Works for me. I didn't have 2-factor Authentication set-up - had to therefore enable it.

[zebarooo]

@punitkshah i've enabled 2FA, no change. @jayantacs can you please confirm you added access for me?

[jayantacs]

@zebarooo : I added you.

[zebarooo]

awesome, thanks!

[jmuthuku]

@jayantacs, can you please add me as well. Thanks

[jayantacs]

@jmuthuku : Added you.

[jmuthuku]

thanks @jayantacs , but I am still not able to access it. Getting 404 Error.

[jayantacs]

@jmuthuku : can you try now?

[jmuthuku]

thanks again @jayantacs . I am able to access now (after enabling 2FA). After filling all the details in App.Config I tried to run GraphBulkImportTest but I am running into this error:

System.AggregateException: One or more errors occurred. ---> System.TypeLoadException: Could not resolve type with token 0100005e (from typeref, class/assembly System.Diagnostics.Eventing.EventProviderTraceListener, System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
at Microsoft.Azure.Documents.Client.DocumentClient.Initialize (System.Uri serviceEndpoint, Microsoft.Azure.Documents.Client.ConnectionPolicy connectionPolicy, System.Nullable1[T] desiredConsistencyLevel) [0x00014] in <8b43155ec86d4fe3bef147f5d0a40526>:0 at Microsoft.Azure.Documents.Client.DocumentClient..ctor (System.Uri serviceEndpoint, System.String authKeyOrResourceToken, Microsoft.Azure.Documents.Client.ConnectionPolicy connectionPolicy, System.Nullable1[T] desiredConsistencyLevel) [0x00070] in <8b43155ec86d4fe3bef147f5d0a40526>:0
at GraphBulkImportTest.Program+d__3.MoveNext () [0x00051] in /Users/jayarajm/temp/Microsoft.Azure.Graphs.BulkImport-master/GraphBulkImportTest/GraphBulkImportTest/Program.cs:40
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional (System.Boolean includeTaskCanceledExceptions) [0x00011] in /Users/builder/data/lanes/5533/mono-mac-sdk/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:2157
at System.Threading.Tasks.Task.Wait (System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00043] in /Users/builder/data/lanes/5533/mono-mac-sdk/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:3189
at System.Threading.Tasks.Task.Wait () [0x00000] in /Users/builder/data/lanes/5533/mono-mac-sdk/external/bockbuild/builds/mono-x64/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:3054
at GraphBulkImportTest.Program.Main (System.String[] args) [0x00001] in /Users/jayarajm/temp/Microsoft.Azure.Graphs.BulkImport-master/GraphBulkImportTest/GraphBulkImportTest/Program.cs:22

Any ideas how to get past this. I'd appreciate if you can let me know. Thanks

[jasonchester]

@jayantacs can you add me to the bulk import repo?

[jayantacs]

@jasonchester: You should have access now.

[gauravhub]

@jayantacs, For Java based applications, is there a similar library that we can use for bulk import?

[AbhishekTripathi]

@jayantacs Please provide access to me as well.

[J0nnie]

@jayantacs Can you provide med access as well. Thanks!

[gaport]

@jayantacs Can I get access to this as well. Thanks

[daviwu]

Can I also get access to the preview please, thanks.

[FSou1]

@jayantacs can you add me to the bulk import repo? Thanks

[whilke]

@jayantacs Can i also get access to the bulk import repo? Thanks

[tknandu]

@AbhishekTripathi @J0nnie @daviwu @FSou1 @whilke Have provided access.

To be noted - we're going to be releasing Graph Import functionality in the BulkExecutor library (which was GAed) very shortly to make access easier - will update details here soon!

[PhilStollery]

Just at the point of needing to do this- perfect timing. Is this where it's being added? https://www.nuget.org/packages/Microsoft.Azure.CosmosDB.BulkExecutor/

[Tameshiwari]

FYI this provided link

https://github.com/Microsoft/Microsoft.Azure.Graphs.BulkImport

is broken [404...] or someone just simply decided to remove the linked content from the map😒

[dagarkatyal]

@jayantacs Can you please provide me access to https://github.com/Microsoft/Microsoft.Azure.Graphs.BulkImport

[ainurt]

Hi @jayantacs @tknandu Can I also get access to https://github.com/Microsoft/Microsoft.Azure.Graphs.BulkImport

Thanks!

[hugovanmens]

Hello @jayantacs, I would like access as well.
Thanks in advance!

[PhilStollery]

@tknandu any update?

[danvanderboom]

Could I get access to the bulk import tool as well? Thanks.

[jayantacs]

The tool is public now.

https://github.com/Azure-Samples/azure-cosmosdb-graph-bulkexecutor-dotnet-getting-started/

[dmelgarejoPDESGO]

@mcharters @Raghu-Gurrala : Please try this for Importing graph. It has partition key support https://github.com/Microsoft/Microsoft.Azure.Graphs.BulkImport

It's in preview and would be released soon. I have given you access to the repository.

Can you give me access to the repository? dmelgarejoPDESGO