Litedb: Exception when concurrent insert (raspberry pi)

[mwkldeveloper]

Hi

I really love LiteDb. it is easy to use.

I use LiteDb 4.1.0 with .net Core WebApi project at raspberry pi. Mostly works fine.

But when i use ab stresstest tool to test concurrent insert.
ab -n 1000 -c 100 192.168.0.101/api/test/insert

it sometimes (random) throw exception (in window environment no problem) :
The important thing is that once this exception throw, the insertion will become very slow and not recover even I restart the application. I need to delete the database then create again to back to normal speed.

System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Object.GetType()
   at LiteDB.BsonMapper.<>c__DisplayClass44_0.<RegisterDbRefItem>b__0(Object obj, BsonMapper m)
   at LiteDB.BsonMapper.SerializeObject(Type type, Object obj, Int32 depth)
   at LiteDB.BsonMapper.Serialize(Type type, Object obj, Int32 depth)
   at LiteDB.BsonMapper.ToDocument(Type type, Object entity)
   at LiteDB.BsonMapper.ToDocument[T](T entity)
   at LiteDB.LiteCollection`1.Insert(T document)
   at LiteDbTest.Controllers.ValuesController.Get() in C:\Users\Marcus\Documents\Visual Studio 2017\Projects\LiteDbTest\LiteDbTest\Controllers\ValuesController.cs:line 37
   at lambda_method(Closure , Object , Object[] )
   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeInnerFilterAsync>d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()

My code in test controller:

//LiteDbManager.Instance  singleton pattern

static LiteCollection<Employee> ecol = LiteDbManager.Instance.GetCollection<Employee>("employees");
static LiteCollection<Accesslog> acol = LiteDbManager.Instance.GetCollection<Accesslog>("accesslogs");


[HttpGet]
public string Get()
{
                        //random employeeId  for test
            int value = random.Next(0, 998);
            string employeeId = $"B0{value.ToString().PadLeft(3, '0')}";

            var employee = ecol.FindById(employeeId); 

            Accesslog alog = new Accesslog()
            {
                Employee = employee ,   //BsonRef
                //Employee = new Employee() { Id = employeeId },   //<-- no exception if create a new Employee Object with Id 
                LogTime = DateTime.Now
            };

            var id = acol.Insert(alog);
            return id.AsString;

}

My Models:

public class Accesslog
{
        public Guid Id { get; set; } = new Guid();

        [Required]
        public DateTime LogTime { get; set; }

        [BsonRef("employees")] 
        public Employee Employee { get; set; }
}

public class Employee
{
        [JsonProperty("EmployeeId")]
        public string Id { get; set; }

        [StringLength(20)]
        public string LastName { get; set; }

        [StringLength(20)]
        public string FirstName { get; set; }
}

What i found is , if i create a new Employee Object only with an Id instead of using the Employee object returned by collection.FindById(employeeId) for reference insert, like below:

Accesslog alog = new Accesslog()
{
Employee = new Employee() { Id = employeeId },
LogTime = DateTime.Now
}

then No exception will throw and work fine.

what is the correct way to Insert/Update/Upsert a document with reference.

Thanks

[mbdavid]

Hi @leung85, your code are right. I don't see your connection string, but for single instance I recommend you to use exclusive mode to avoid re-check external file changes in each operation.

If you want optimize your code, it's better use only empty instance with Id value (because you avoid an Find).

About the error, it's happening in BsonMapper, RegisterDbRefItem 409 line but maybe is not there the real error. This line only apply id.GetType() and your Id is null (and could not be null). After run var employee = ecol.FindById(employeeId); your employee variable are with null value in Id.

This could be some concurrency problem in BsonMapper that use single instance.... To keep compatible with NET35 I use simple Dictionary instead ConcurrencyDictionary. I will made some updates I will ask you to test in raspberry again.

And about raspberry, that awesome to know that are working on this board. I had one at home but I didn't test on it. Are you using Linux or Windows Core?

[mwkldeveloper]

Hi @mbdavid , thanks for your quick reply and recommendation.

You are right.
i test again with below, employee is not null but and only id is null, FirstName, Lastname is still here.

[HttpGet]
public string Get()
{
    int value = random.Next(0, 998);
    string Id = $"B0{value.ToString().PadLeft(3, '0')}";
    var employee  = ecol.FindById(Id);

    if (employee  != null)
    {

        if (string.IsNullOrEmpty(employee.Id))
        {
            Console.WriteLine($"Employee {employee.FirstName} {Id}'s Id Null");
            return null;
        }

        Accesslog a = new Accesslog()
        {
            Employee = employee ,
            LogTime = DateTime.Now
        };
        var id = acol.Insert(a);
        return id.AsString;
    }
    else
    {
        Console.WriteLine($"Employee {Id} Null");
        return null;
    }
}

I run ecol.FindById(Id) standalone, the id is not null.

For raspberry, i am using Linux (Raspbian) and (Linaro).
I follow .net community's guide and work fine in both webapi, MVC and Console project.

I have't try Windows 10 IoT Core, but it should work. Above guide has instruction too.

Anyway, LiteDb is awesome, and fit for IoT projects . I look forward to your update and test again.

[mwkldeveloper]

Hi @mbdavid

I do a concurrent test on ecol.FindById(Id).
And find that not only Id randomly null, other property also randomly null.

The test is to get the same employee "B0226"
Standalone request result all have value:

{
"id": "B0226",
"lastName": "BB000226",
"firstName": "AA000226",
"internalID": "II000226"
}

My concurrent Test :

ab -n 1000 -c 100 192.168.0.245/api/test/B0226

Code:

[HttpGet("{id}", Name = "Get")]
public Employee Get(string id)
      {
    var employee = ecol.FindById(id);

    foreach (PropertyInfo propertyInfo in employee.GetType().GetProperties())
    {
        if (propertyInfo.GetValue(employee) == null)
        {
            Console.WriteLine($"Property {propertyInfo.Name} is Null");
            break;
        }
    }

    return employee;

}

Result:

Property InternalID is Null
Property Id is Null
Property LastName is Null
Property FirstName is Null
Property Id is Null
Property Id is Null
Property Id is Null
Property Id is Null
Property InternalID is Null
Property Id is Null
Property Id is Null
Property LastName is Null
Property LastName is Null
Property Id is Null
Property Id is Null
Property LastName is Null
Property FirstName is Null
Property Id is Null
Property Id is Null
Property Id is Null
Property InternalID is Null
Property Id is Null
Property Id is Null
Property Id is Null

[mbdavid]

Hi @leung85, I found a bug on BsonReader class that can happen in some cases in v.4.1. Can you get branch version and test again?

[mwkldeveloper]

Hi @mbdavid I do the same test with branch version.

That's amazing!!. What's magic have you done? That's cool.

The "null" value problem is solved and also solved a problem that annoying me:

In 4.1 version, there is a performance issue on insert with index created.

I do the test on raspberry pi 2 model B and my target machine

In raspberry 2, there are no significant drop of performance, but i think it is slow, not good enough.

In my target machine, it is also slow and the worst is after insert around 3000 record, there are significant drop of performance, and never recover after restart the applicant. If i do not create any index, no drop of performance but still slow.

It is glad the above problem is solved using the branch version and also a a huge improvement on performance.

I rebuild your performance Test project LiteDB-Perf to Core project and do a test on raspberry pi 2 model B : (a much different speed)

4.1 version:

pi@raspberrypi:~/core $ ./TestPerfLiteDBCore
LiteDB: default - 5000 records
==============================
Insert         : 227259 ms -       22 records/second
Bulk           :  1985 ms -     2518 records/second
Update         : 104964 ms -       48 records/second
CreateIndex    :  4367 ms -     1145 records/second
Query          :  1685 ms -     2967 records/second
Delete         :  4309 ms -     1160 records/second
Drop           :  2552 ms -     1959 records/second
FileLength     :  7540 kb

LiteDB: encrypted - 5000 records
================================
Insert         : 219739 ms -       23 records/second
Bulk           :  2217 ms -     2255 records/second
Update         : 114672 ms -       44 records/second
CreateIndex    :  1567 ms -     3190 records/second
Query          :  1478 ms -     3382 records/second
Delete         :  5377 ms -      930 records/second
Drop           :   611 ms -     8179 records/second
FileLength     :  7552 kb

LiteDB: exclusive no journal - 5000 records
===========================================
Insert         : 96787 ms -       52 records/second
Bulk           :  2807 ms -     1781 records/second
Update         : 50389 ms -       99 records/second
CreateIndex    :  2057 ms -     2431 records/second
Query          :   782 ms -     6387 records/second
Delete         :   631 ms -     7915 records/second
Drop           :   264 ms -    18923 records/second
FileLength     :  7536 kb

branch version:

pi@raspberrypi:~/core $ ./TestPerfLiteDBCore
LiteDB: default - 5000 records
==============================
Insert         : 22951 ms -      218 records/second
Bulk           :  1695 ms -     2948 records/second
Update         :  5403 ms -      925 records/second
CreateIndex    :  4369 ms -     1144 records/second
Query          :  1579 ms -     3165 records/second
Delete         :   467 ms -    10698 records/second
Drop           :   189 ms -    26316 records/second
FileLength     :  7524 kb

LiteDB: encrypted - 5000 records
================================
Insert         : 29553 ms -      169 records/second
Bulk           :  2034 ms -     2458 records/second
Update         :  8295 ms -      603 records/second
CreateIndex    :  1626 ms -     3075 records/second
Query          :  1576 ms -     3173 records/second
Delete         :   844 ms -     5922 records/second
Drop           :   400 ms -    12489 records/second
FileLength     :  7568 kb

LiteDB: exclusive no journal - 5000 records
===========================================
Insert         : 17848 ms -      280 records/second
Bulk           :  1652 ms -     3026 records/second
Update         :  3377 ms -     1480 records/second
CreateIndex    :  1411 ms -     3543 records/second
Query          :  1048 ms -     4769 records/second
Delete         :   405 ms -    12345 records/second
Drop           :   120 ms -    41396 records/second
FileLength     :  7528 kb

[mbdavid]

Hi @leung85, thanks!! The performance issues was already fixed in this commit:

https://github.com/mbdavid/LiteDB/commit/ad720af8e046c4beee46ecaeca28c5eb6c7946e3

It's about witch Flush method use in FileStream. Using Flush(true) .net ignores OS disk flush system and do your own flush instantly.

About this bug on null in Id, it's was a trick :) - I was working in v5 and got a random error that I was going crazy!! In v4.1 I changed BsonRead class to single instance per LiteEngine instance but I forgot an class variable declaration (at end of file... I never do this!)... so there are re-use this variable an, some concurrency cases, can override:
https://github.com/mbdavid/LiteDB/commit/83f193ae5107b81ac58dca1e467db29455cbb494

[mwkldeveloper]

@mbdavid Will you plan for a launch a release 4.1.x for those fix??

[mbdavid]

Next weekend (6~7/jan). Need run some unit tests and update release notes.