Litedb: IOException occured when running transaction concurrently(v5)

[itn3000]

Overview

when testing litedb-5-beta with concurrent transaction, System.IO.IOException was occured.

Environment

• dotnet-sdk-3.0
• TargetFramework: netcoreapp3.0

Steps to reproduce

1. create dotnet console project with TargetFramework=netcoreapp3.0
2. run following program

using System;
using System.Threading.Tasks;
using System.Threading;
using System.Linq;
using System.IO;
using LiteDB;

namespace litedbtest
{
    class EntityA
    {
        public long Id { get; set; }
        public int X { get; set; }
        public string Y { get; set; }
    }
    class LiteDbConflictTest
    {
        static long IdValue = 0;
        public static async Task TestManyThread()
        {
            const int TotalNum = 10000;
            const int TaskNum = 2;
            // ensure using initialized database.
            if (File.Exists("manythread.db"))
            {
                File.Delete("manythread.db");
            }
            if (File.Exists("manythread-log.db"))
            {
                File.Delete("manythread-log.db");
            }
            using (var db = new LiteDB.LiteDatabase("manythread.db"))
            {
                await Task.WhenAll(
                    Enumerable.Range(0, TaskNum).Select(async (idx) =>
                    {
                        // concurrent insert task
                        await Task.Yield();
                        db.BeginTrans();
                        var collection = db.GetCollection<EntityA>("HogeCollection");
                        for (int i = 0; i < TotalNum / TaskNum; i++)
                        {
                            collection.Insert(new EntityA()
                            {
                                Id = Interlocked.Increment(ref IdValue),
                                X = idx * 10000 + i,
                                Y = $"{idx}_{i}"
                            });
                        }
                        db.Commit();
                    })
                ).ConfigureAwait(false);
            }
        }
        static void Main(string[] args)
        {
            TestManyThread().Wait();
        }
    }
}

Expected result

the litedb database file named "manythread.db" which has 10000 records.

Actual result

following exception was thrown and exited abnormally.

Unhandled exception. System.AggregateException: One or more errors occurred. (The process cannot access the file 'G:\src\gitrepos\dotnet-sandbox\litedbtest\litedbtest\manythread.db' because it is being used by another process.)
 ---> System.IO.IOException: The process cannot access the file 'G:\src\gitrepos\dotnet-sandbox\litedbtest\litedbtest\manythread.db' because it is being used by another process.
   at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)
   at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at LiteDB.Engine.FileStreamFactory.GetStream(Boolean canWrite, Boolean sequencial)
   at LiteDB.Engine.StreamPool.<>c__DisplayClass3_0.<.ctor>b__0()
   at System.Lazy`1.PublicationOnlyViaFactory(LazyHelper initializer)
   at System.Lazy`1.CreateValue()
   at System.Lazy`1.get_Value()
   at LiteDB.Engine.StreamPool.get_Writer()
   at LiteDB.Engine.DiskService..ctor(EngineSettings settings)
   at LiteDB.Engine.LiteEngine..ctor(EngineSettings settings)
   at LiteDB.ConnectionString.CreateEngine()
   at System.Lazy`1.PublicationOnlyViaFactory(LazyHelper initializer)
   at System.Lazy`1.CreateValue()
   at litedbtest.LiteDbConflictTest.<>c__DisplayClass1_0.<<TestManyThread>b__0>d.MoveNext() in G:\src\gitrepos\dotnet-sandbox\litedbtest\litedbtest\LiteDbConflictTest.cs:line 39
--- End of stack trace from previous location where exception was thrown ---
   at litedbtest.LiteDbConflictTest.TestManyThread() in G:\src\gitrepos\dotnet-sandbox\litedbtest\litedbtest\LiteDbConflictTest.cs:line 34
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at litedbtest.Program.Main(String[] args) in G:\src\gitrepos\dotnet-sandbox\litedbtest\litedbtest\Program.cs:line 432

if TaskNum was changed to 1, all records were inserted successfully.

[alifpay]

Litedb has problems Thread-safe in v4, concurrently insert fails when we are using in threads.

[mincasoft]

Same problems. In v5 too

[mbdavid]

Hi guys, thanks for this report. Sounds problem is in datafile initialization that are not concurrent support. Just create new LiteDatabase do not open/create your datafile. If you create your database before, there is no problem:

```C#
using (var db = new LiteDB.LiteDatabase("manythread.db"))
{
// force database creation before starts threads
db.GetCollection("HogeCollection").Count();

            await Task.WhenAll(
                Enumerable.Range(0, TOTAL_TASKS).Select(async (idx) =>
                {

```

I will try fix this now.

[mbdavid]

Ok, done. I fix Lazy initialize to thread safe. Thanks again guys

[itn3000]

test in my env(win8.1-x64, netcoreapp3.0, git rev=3a109c3519c2e2987f52831ffc2ed9ff8e55df11), IOException is no longer occured, thank you for fixing.

[mincasoft]

@mbdavid when there is a patch release on nuget?

[Stockholmsnovis]

Hi! I am using LiteDB "behind" an Asp.net WebAPI. I ran into this issue when upgrading from v4 to v5. At first, I didnt have any connection string parameters so I guess IOException was expected since the connection string defaults to Connection=direct right?

I added upgrade=true;connection=shared, but still got the IOException. However if I now remove upgrade=true, shared connection seems to be working again.

It seems like having upgrade=true in the connection string triggers some thread unsafe behavior even though connection=shared is present. Does that make sense? Just thought Id share this in case someone else has this problem.

I guess I can upgrade all existing v4-files in a "pre-initialization step" by simply opening up a connection to the files with upgrade=true. Thanks for reading.