Dexie.js: tables only populated with first 50 ID's

[teachtyler]

In my GTFS collection I have 5 tables, all are being populated with data now (which is awesome). However, at the end of my transaction when I am adding items to my tables, it seems to only update to the first 50 keys (49 Key_ID) and this happens in all tables. I tried removing ++id from table stores. That only errors saying key already exists in object store, preventing me from writing anything to my tables. Without checking for existing db, I can rewrite what's already been created, so 100 entries of the same thing twice. Am I missing something or going at this the wrong way?

---

gist

db.version(1).stores({
  'calendar': "service_id,monday,tuesday,wednesday,thursday,friday,saturday,sunday,start_date,end_date",
  'calendar_dates': "service_id,date,exception_type",
  'stop_times': "trip_id,arrival_time,departure_time,stop_id,stop_sequence,pickup_type,drop_off_type",
  'stops': "stop_id,stop_code,stop_name,stop_lat,stop_lon,zone_id,stop_url,location_type,parent_station,platform_code,wheelchair_boarding",
  'trips': "route_id,service_id,trip_id,trip_headsign,trip_short_name,direction_id,shape_id,wheelchair_accessible,bikes_allowed"
});

return db.transaction('rw', db[name], function () {
  res.forEach(function (item) {
    // console.log("Adding object: " + JSON.stringify(item));
    db[name].add(item);
    resolve(data);
  })
})

---

Example from stop_times.txt for reference

trip_id,arrival_time,departure_time,stop_id,stop_sequence,pickup_type,drop_off_type
23a,7:33:00,7:33:00,777403,1,0,0
23a,7:45:00,7:45:00,777402,2,0,0

[dfahlander]

I was briefly looking at the gist. It seems you are not using promises correctly. I would do it like this:

    // Populate from AJAX:
    db.on('ready', function () {
      return db.calendar.count(function (count) {
        if (count > 0) {
          console.log("Already populated, 'gtfs' is already created at this domain.");
        } else {
          console.log("Database is empty. Populating from ajax call...");
          return Dexie.Promise.all (gtfs.map(name => Dexie.Promise.resolve($.ajax('gtfs/' + name + '.txt', {
            dataType: 'text'
          })).then(data => {
            console.log("Got ajax response for " + name);
            return JSON.parse(csvJSON(data));
          }).then(res => {
            return db[name].bulkPut(res);
          }).then(()=>{
            console.log("Done importing " + name);
          }))).then(()=>{
            console.log("All files successfully imported");
          }).catch(err => {
            console.error("Error importing data: " + (err.stack || err));
            throw err;
          });
        }
      });
    });

Here's what I do:

1. Wait in parallell for all ajax requests asynchronically
2. Convert each jQUery promise to a Dexie.Promise so we can chain it.
3. When each ajax request completes, convert the data and put it in the database using bulkPut (will do an internal transaction for each bulkPut()).

Could have used bulkAdd() instead but if using put, you can update existing items if needed in a future version of your app.

[teachtyler]

Awesome, this snippet has really helped me understand promises better! I had to add a few lines [1] to get importing promise. However, I seem to be getting an error [2], using bulkPut or bulkAdd; add still works but only first 50 keys. Seems like a good error to have, since it's trying to write the entirety of all the text files, but I'm not sure where the error is coming from. transitFactory.js:9:19 is simply
var transit = new Dexie('gtfs');

---

[1]

 }).then(res => {
            res.map(item => {
              return db[name].bulkPut(item);
            })

[2]

[dfahlander]

That's weird. Is the gist uptodate to reproduce this?

[teachtyler]

yes, i also just pushed an update to the repo as well. Thanks for looking into this

[dfahlander]

The reason for the strange error was that you sent an object instead of an array into bulkPut(). You should remove the map part in bulkPut. I make a pull request with suggested fixes in your repo.

[dfahlander]

Also make sure to delete your db. Your code didnt import anything. the 50 records was there from a previous import.

[teachtyler]

Thanks a bunch! I see now how bulkPut and bulkAdd work, so we don't have to iterate over every object in the response. This library is the best, thanks for helping me understand how to use it better.