Litedb: [question] How to prevent race condition?

[nictaylr]

I am using LiteDB in a web application.

However, I am having this issue:
I am storing a list of users, and each user has a data usage number. In each request, the user data usage number is loaded from LiteDB, and is increased. But this is my problem: if the client makes many simultaneous requests, the data usage number will be loaded before it is updated, resulting in an incorrect data usage.

To explain again:
In database: [usage: 20]

request 1:

• thread 1: server loads usage (is 20)
• thread 1: does some processsing (adds 10), does not update user in litedb collection yet
  request 2 (happens during request 1's data processing):
• thread 2: server loads usage (is 20) <-- This is my problem, I want to lock the user so other requests can't happen until first request processing is done
• thread 2: server adds 10
• thread 1 saves user (now usage is 30)
• thread 2 saves user (now usage is 30)

the problem is that thread 2 doesn't wait for thread 1 to finish, and incorrect final value is stored. I am using NancyFx with async/await requests; each request spawns a new thread.

i am using transactions only when saving the value.

thanks

[DeanVanGreunen]

For each user create seperate table which is like "user_locks" table.

It can have same user issue / username to keep the locks active (do not
delete locks) have 3 fields. Where fields are "read_lock" "write_lock" "id"

Then set read lock when writing and set write lock when reading. Use a for
loop / do loop or while loop with like 20 retires to access user table by
first reading the user_locks table.
On Jan 6, 2017 2:32 AM, "nictaylr" [e-mail] wrote:

I am using LiteDB in a web application.

However, I am having this issue:
I am storing a list of users, and each user has a data usage number. In
each request, the user data usage number is loaded from LiteDB, and is
increased. But this is my problem: if the client makes many simultaneous
requests, the data usage number will be loaded before it is updated,
resulting in an incorrect data usage.

To explain again:
In database: [usage: 20]

request 1:

• thread 1: server loads usage (is 20)
• thread 1: does some processsing (adds 10), does not update user in
  litedb collection yet
  request 2 (happens during request 1's data processing):
• thread 2: server loads usage (is 20) <-- This is my problem, I want
  to lock the user so other requests can't happen until first request
  processing is done
• thread 2: server adds 10
• thread 1 saves user (now usage is 30)
• thread 2 saves user (now usage is 30)

the problem is that thread 2 doesn't wait for thread 1 to finish, and
incorrect final value is stored. I am using NancyFx with async/await
requests; each request spawns a new thread.

thanks

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mbdavid/LiteDB/issues/415, or mute the thread
https://github.com/notifications/unsubscribe-auth/AVQ2C-nItbQbwnMpFaXbPwvR3C4vxQvAks5rPYuMgaJpZM4LcSIP
.

[DeanVanGreunen]

Implementing an international lock system will ensure data integrity
against data concurrency issues as you have above(git issue) / below
(email)
On Jan 6, 2017 2:43 AM, [email protected] wrote:

For each user create seperate table which is like "user_locks" table.

It can have same user issue / username to keep the locks active (do not
delete locks) have 3 fields. Where fields are "read_lock" "write_lock" "id"

Then set read lock when writing and set write lock when reading. Use a for
loop / do loop or while loop with like 20 retires to access user table by
first reading the user_locks table.
On Jan 6, 2017 2:32 AM, "nictaylr" [e-mail] wrote:

I am using LiteDB in a web application.

However, I am having this issue:
I am storing a list of users, and each user has a data usage number. In
each request, the user data usage number is loaded from LiteDB, and is
increased. But this is my problem: if the client makes many simultaneous
requests, the data usage number will be loaded before it is updated,
resulting in an incorrect data usage.

To explain again:
In database: [usage: 20]

request 1:

• thread 1: server loads usage (is 20)
• thread 1: does some processsing (adds 10), does not update user in
  litedb collection yet
  request 2 (happens during request 1's data processing):
• thread 2: server loads usage (is 20) <-- This is my problem, I want to
  lock the user so other requests can't happen until first request processing
  is done
• thread 2: server adds 10
• thread 1 saves user (now usage is 30)
• thread 2 saves user (now usage is 30)

the problem is that thread 2 doesn't wait for thread 1 to finish, and
incorrect final value is stored. I am using NancyFx with async/await
requests; each request spawns a new thread.

thanks

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mbdavid/LiteDB/issues/415, or mute the thread
https://github.com/notifications/unsubscribe-auth/AVQ2C-nItbQbwnMpFaXbPwvR3C4vxQvAks5rPYuMgaJpZM4LcSIP
.

[nictaylr]

could you please explain in more detail @xX-MLG-Xx

[nictaylr]

should I store the lock table in a singleton or in the database?

[mbdavid]

you can use transaction. When you call BegonTrans() you lock your databse in Reserved lock. So, after BeginTrans call your find and than call your update and than commit.

In lastest commit I expose Lock service too.

[mbdavid]

In LockService you can call :

using(db.Engine.Lock.Reserved())
{
safe zone
}

[nictaylr]

the problem might be that my operations are very long (20 seconds!), and I want to allow concurrent requests. I don't want the concurrent operations to take so long total. is there a good way to implement this? async/await are fine.

[nictaylr]

i found helpful resource: http://softwareengineering.stackexchange.com/questions/263726/how-to-prevent-race-conditions-in-a-web-application, I will try to implement and ask if I have issues

[DeanVanGreunen]

This would stop threads from reading writing simultaneously.

His race is due to internal database read write processing within
application and not file read writing

Using a singleton would solve this issue in general but would still require
customer read write lock logic. As the logic for database login only
resides in the actual file system.
On Jan 6, 2017 3:02 AM, "Mauricio David" [e-mail] wrote:

you can use transaction. When you call BegonTrans() you lock your databse
in Reserved lock. So, after BeginTrans call your find and than call your
update and than commit.

In lastest commit I expose Lock service too.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mbdavid/LiteDB/issues/415#issuecomment-270803334, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVQ2CwjaCIO9h4IYkgtRJAHmHBI4j-8Jks5rPZKEgaJpZM4LcSIP
.

[DeanVanGreunen]

Within the database.

Table: user_locks
Fields :
Bool write_lock default value off
Bool read _lock default value off.

Before processing user data from user table for either read (select) or
write (update / insert)

Do check of contents of the values of user_locks table

In structure of example:
Table: users
Id | username | data...
1| Steve | djdjsjs
2 | John | djdbdbdb
3 | most | dhdjsnsn

Table: user_locks
Id | read_lock | write_lock
1 | 0 | 0
2 | 0 | 0
3 | 0 | 0

Program example

To explain again:
In database: [usage: 20]

request 1:

• thread 1: server loads usage (is 20)
• thread 1: does some processsing (adds 10), does not update user in

litedb collection yet
request 2 (happens during request 1's data processing):

• thread 2: server loads usage
• thread 2: server adds 10
• thread 1 saves user
• thread 2 saves user

On each of user data in thread do the following:
When reading user data
Check read_lock
If set to true do a wait loop for x. seconds then check again
till. read_lock is false
Set write_lock to true
Read user data.
Do processing
Set write_lock to false

When writing (saving / updating)
Check write_lock
Do wait loop till value is false (same as read_lock logic)
Set read_lock to true
Do update / insert command etc..
Set read_lock to false
On Jan 6, 2017 2:51 AM, "nictaylr" [e-mail] wrote:

could you please explain in more detail @xX-MLG-Xx
https://github.com/xX-MLG-Xx

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mbdavid/LiteDB/issues/415#issuecomment-270801691, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVQ2C80Zs82SF8AlecmV1oiQ7-VYxb9iks5rPY_8gaJpZM4LcSIP
.

[nictaylr]

Thanks for the great idea! I will try to implement.

[DeanVanGreunen]

Sure :) ;) here to help - LiteDB made with love xD

[0xFireball]

Yay! Thanks for the design idea! I implemented it like this:

https://github.com/0xFireball/PenguinUpload/blob/quota/PenguinUpload/src/PenguinUpload/Infrastructure/Upload/LocalStorageHandler.cs#L45-L64

https://github.com/0xFireball/PenguinUpload/blob/quota/PenguinUpload/src/PenguinUpload/Infrastructure/Concurrency/UserLock.cs

[0xFireball]

Rather than using loops @xX-MLG-Xx , I used AutoResetEvent because it does effectively the same thing in a simpler and more scalable way

[DeanVanGreunen]

@mbdavid be able to revise this lock into litedb engine?

@0xFireball you could exend this to allow multi reads with only locking write_lock when reading.

So you could update to exclusivereadonly
Exclusive read write async
as your exclusive read does block reading if being read at in some part of web app.

So logins into web app would take on role of calling exclusivereadonly for pass and username verfication etc. And password setting would use write exclusive etc. All asynchronous of course

Which would act like FB micro server database system.

Where many reads and then 1 huge write insteady of many writes.

[0xFireball]

very helpful suggestion, thanks!

[0xFireball]

@xX-MLG-Xx Here's my implementation: https://github.com/0xFireball/PenguinUpload/blob/2bede37e349d824d5188d636edfc535ec6c4d58a/PenguinUpload/src/PenguinUpload/Infrastructure/Concurrency/UserLock.cs#L45-L60

Please let me know if I misunderstood you or did anything wrong :stuck_out_tongue:

UPDATE: I thought this might be nicer (I added a nicer asynchronous API in addition to the existing methods):
https://github.com/0xFireball/PenguinUpload/blob/495591975b3d3f9b97aee7eb7db63589e4e636cb/PenguinUpload/src/PenguinUpload/Infrastructure/Concurrency/UserLock.cs#L67-L90
This will auto-call the Obtain and Release locks, and is awaitable