Litedb: Transaction cancelled when index does not exist

[baSSiLL]

If there is a query inside a transaction, and index on the field is not created yet, then after query executes transaction is in the Canceled state.

That happens because when there is no index, IndexNotFoundException is thrown first, and inner transaction in DbEngine.Find is rolled back (and outer transaction cancelled). Upper in the stack exception get caught and index created, and query succeeds. But outer transaction could not be committed because it's cancelled.

Test to reproduce:

    [TestMethod]
    public void TestIndexCreationInsideTransaction()
    {
        using (var tmp = new TempFile())
        {
            using (var db = new LiteDatabase(tmp.ConnectionString))
            {
                var col = db.GetCollection("col");

                using (var trans = db.BeginTrans())
                {
                    col.Insert(new BsonDocument().Add("test", 0));

                    var query = Query.EQ("test", 0);
                    // Index is created here
                    col.Find(query).ToList();

                    // Fails with TransactionCanceledException
                    trans.Commit();
                }
            }
        }
    }

[mbdavid]

Hi @baSSiLL, I will try to take a look on this. I'm out of time for last days. Transactions will be die :) To be thread safe, next version will not support transactions (ACID will be only in document level).

[sherry-ummen]

Yeah I have been wanting this for long. Finally you are pulling the plug :)

[baSSiLL]

Hm. I was sure you've made decision in favor of transactions. Because I saw they appeared back in 2.0.0-rc2 after being removed in 2.0.0-alpha.
Please clarify what thread safety you mean here. Using a single LiteDatabase object from different threads? Using the same database file from different threads?

[mbdavid]

Hi @baSSiLL, I tried to remove in v2 alpha/beta version, but added again in final release (removing thread safe).
Thread safe will be support a single LiteDatabase instance for many threads consuming data. This is better than use current using statement - which open/close datafile (and journal file) each use. Using thread safe I can manage concurrency locks using lock statement (to avoid two write operations at same time). Using a single LiteDatabase instance, I can keep in cache loaded pages for much more fast fetch data results.
Using thread safe will be easy and fast than current "process" concurrency (which are file based locked). For applications like website, mobile, single desktop app, LiteDB will works more like a "server database". If your application are "multi-process" app (like many clients connection to same datafile) you can still using using statement.

[sherry-ummen]

Does it mean with removing the Transaction you will also remove the possibly of multiple process concurrency?

[mbdavid]

@sherry-ummen, no. Concurrency now is "file-lock" based. In next version, will be based in .NET lock statement (because will be same instance). My very first example is here (still using new LiteEngine):

https://github.com/mbdavid/LiteDB/blob/dev/LiteDB.Tests/Tests/EngineTest.cs#L38

[sherry-ummen]

Hmm if u remove file lock based then how will it work in case multiple process share the same database? Will it lead to some inconsistent state?

[mbdavid]

LiteDB will not remove complete file lock. File will be open in ReadWrite mode and ReadShare only. When a second instance try open datafile that will be opened, file lock will block this second connection. Will release to connect only when first instance close datafile. This is the same solution that LiteDB uses today, but will not be recommended anymore.

[sherry-ummen]

When you say will not be recommended, I am curious to know the reasons. Thanks

[mbdavid]

Simple, because will be more efficient works with a single instance. Single instance will open datafile once, will open journal file once too. Single instance can keep, in cache, loaded pages to fast results. It´s works like a server database, when you have a single process managing cache/files (in LiteDB will be a single instance).

[sherry-ummen]

Ok I did not understand how you can have single instance shared between processes? Sorry I kind of lost now. Please explain

[mbdavid]

You will not have some instance across processes. When you have an app that has multiples processes that can access same datafile, you still using same using statement. This will not change. Concurrency will be supported by locking data file.

But, when you have a single process app (like website, mobile or desktop app) will be better use single instance. No file lock, only .NET internal lock implementation.

[sherry-ummen]

All clear. I understand now. Thanks for clearing this up

[henon]

@mbdavid: I don't know how you plan to do this, just want to say that the file lock would be a very bad thing if it blocks the thread. It causes the second process to block until the first process releases the file lock. This is how it currently is with DBLite (and I mean DBLite not LiteDB). I would suggest that you allow the file lock to be awaited using async and allow a timeout to be specified. That way we don't have to waste a thread or ThreadPool-thread for blocked waiting on file lock.

[mbdavid]

Hi @henon. In v1 I used FileStream.Lock and Unlock methods to control concurrency. Using a simple "while timeout, try lock". Locks are only during write operations.
In v2, I changed this lock for open data file in write exclusive when write operations. Datafile now are released after each operation.

Both ways causes some problem: concurrency are based in lock file. Almost all time are spend open/close datafile every time. If you see many issues create a simple solution: create a LiteDB wrapper using .NET lock statment and re-use same LiteDatabase instance.

In this next v3, I'm working on it: create a single instance LiteDB (when possible). If needs multi processes access, so will works lite v2 (using same disk lock).

[mbdavid]

Hi @sherry-ummen and @baSSiLL, transactions are back again 😄 in v3. At least for now, I go it how implement transaction locks with ReaderWriterLockSlim in right way. I fixed cache checkpoint (a tricky part of cache management system). I introduced AutoCommit (default is false) - so any Insert/Update/Delete will be made in memory only - and will persist in disk only when database close (Dispose) or call Commit() (I don't know yet if this will be false as default). Speed increases a lot because of that, because now I can manage data in memory for more time that in disk.