Dexie.js: Very SLOW For BigData Store

[mesutyigit]

I store a video chunks on indexeddb via dexie , approx 2GB video chunks (~600 rows).

!!!!! Time of First lookup and fetch row with primary key -> 40 Second (db.chunks.get(1)-> took 40 second)
!!! Second and others fetches took in millisecond.
What happening in this period.
When database size increased , first fetch take a long time linearly.

Note: Tried Latest version of dexie and browser of latest Chrome&Opera&Firefox

DB Schema:
db.version(1).stores({
videos: "++id,url,content,size,httpUrl,hiveCdnUrl,createdDate"
}); //* content propert is ArrayBuffer

[dfahlander]

What browser type and version are you using? Device type desktop or mobile? Can you try fetching your data directly from native indexedDB just to check what time you would get there?

db.transaction ('r', 'chunks', trans => {
  return new Promise (resolve => {
    var store = trans.idbtrans.objectStore('chunks');
    var tick = Date.now();
    store.get(1).onsuccess = ev => {
      console.log("Time taken: ", Date.now() - tick);
      resolve();
    };
  });
});

[dfahlander]

I updated the code snippet in my last comment to measure time taken (milliseconds).

Side note: It is also interesting to know whether you store your gigabyte chunks in Blob or Uint8Array - just for curiousity. I have no experience of storing that large amount of data in indexedDB and chances are that it would not be optimized for it. It may be worth it to try divide your data into smaller chunks than 2 GB - maybe in the MB range.

[dfahlander]

@mesutyigit Do I understand you correctly: Are you are really storing 2 Gigabyte (!!) into a single indexedddb row (object)? That will not work well as each row needs to be loaded into memory. Should really divide your data into much smaller chunks as suggested in my previous comment. Client devices may have very limited RAM memory to work with.

[mesutyigit]

Not single row. There are thousands rows. Each row approx. 900Kb size.
Dexie db.open function resolving very long time. Why?

db scheme is here

my metrics is here:
first read time (including db opening time): readtime(378352ms)
and first is completed then second readtime(db has opened): readtime(20639ms)
and second is completed then third time : readtime(1418ms)
and third is completed then fourth time : readtime(311ms).

other notes:
dexie version : dexie@next
browser chrome: 67

Dexie initialize time for metric my all client (4K client )
[Dexie initialize time: db must be open and lookup with primary key must spend time 1second or below. If it spend more 1 sec time , try second lookup , recursive try with this method. If first catch lookup resolve time 1 second below, dexie is initialized. ]
This metric visualization is here (collected from avg 4k online client,every day )

Avg Initialize time take 1min.

[dfahlander]

You index a property named 'content'. What data is stored into it? I hope no large content. I hope you know that you don't need to index all properties. I've had other issues reported where people indexed properties containing images or videos. That can starve out the db completely. Indexes are only for properties to use when searching whitch is typically numbers or small strings.

[mesutyigit]

schema : "&url:String,content:ArrayBuffer,size:Int,createdDate:Long" url:Primary Key (index is here )
content is arrayBuffer of this object (a small chunk of video (500Kb- max 1mb)), no index in here.

[dfahlander]

Aha. You say 'no index here' but in your code you list it among other indexed properties (!) Please add a version 2 with 'content' from videos table to remove that index (!)

db.version (2).stores ({
  videos: "++id,url,size,httpUrl,hiveCdnUrl,createdDate"
}) ;

[dfahlander]

This seems to have become a common pitfall, that people think they have to list all properties in the schema (as they were SQL columns), not aware that they should only list those properties that they wish to index.

[mesutyigit]

I opened this issue a few months ago so schema changed more times.
you say videos:"++id,url,size,httpUrl,hiveCdnUrl,createdDate" not usefull for my purpose.

Please tell me best effective indexedDB schema for my purpose.

• I want to store chunks of videos.
  think that VideoChunk(url:String , size:Int,createDate:Long, data: ArrayBuffer).

• I want to lookup chunk with URL as soon as fast like this

  • db.videos.get('www.example.com/1.ts') !== undefined

• I want to read chunk object with URL as soon as fast

• db.videos.get('www.example.com/1.ts')

• I want to store thousands of row in this table ( Max limit is client Qouta Exceed)

My current schema is:
db.version (1).stores ({
videos: "&url,data,size,createdDate"
}) ;

I don't use ++id column because URL column unique for me. All lookup queries using URL.

How I can do it.

[sumnulu]

http://dexie.org/docs/Version/Version.stores() clearly states schemaDefinition is just for primary and secondary indexes and there are few more warning below.

If one reads the documentation and remember all there are no problems, however I think it is still a pitfall. schemaDefinition parameter name can be changed to indexDefinition or indexedFields or something similar to reflect that, that parameter is just for indexes ,and dexie is a schemaless database.