Litedb: Any way around the 2GB file limit?

[softwareguy74]

While 2GB may sound like a lot, it is not enough when dealing with a DMS that contains 10GB or larger files. I know I could just keep them external to the LiteDB database file and make some sort of reference, but I really was hoping to keep these all self contained. Is there any "hack" or some way around this limitation? Is there any plans to increase this in the future?

[mbdavid]

2Gb are per file in LiteStorage, not datafile. Do you have larger file to be storaged inside a datafile?

[DeanVanGreunen]

There is a possible way to bypass the 2gb limit. Use a 64bit system in
combination of a custom implementation so the default counter for linked
lists uses an unsigned 64bit sized int. That will give you 16gb file
allowance but remember to use a Linux based so as the file storage works
like this. Its the best method to run with you.

On 25 Mar 2017 21:32, "Mauricio David" [e-mail] wrote:
>

2Gb are per file in LiteStorage, not datafile. Do you have larger file to
be storaged inside a datafile?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[softwareguy74]

Yes, I mean PER file. 10GB+ files are common for us.

[DeanVanGreunen]

Use a Linux distro and have mono.net installed.
On 27 Mar 2017 18:33, "softwareguy74" [e-mail] wrote:

Yes, I mean PER file. 10GB+ files are common for us.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mbdavid/LiteDB/issues/549#issuecomment-289508927, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVQ2C26wCcfNVxU8pZAESH1U0FGFwQA7ks5rp-TbgaJpZM4MpGLw
.

[DeanVanGreunen]

Windows plateform has a file size limit.
On 27 Mar 2017 18:42, "Dean Van Greunen" [e-mail] wrote:

Use a Linux distro and have mono.net installed.
On 27 Mar 2017 18:33, "softwareguy74" [e-mail] wrote:

Yes, I mean PER file. 10GB+ files are common for us.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mbdavid/LiteDB/issues/549#issuecomment-289508927,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVQ2C26wCcfNVxU8pZAESH1U0FGFwQA7ks5rp-TbgaJpZM4MpGLw
.

[mbdavid]

Hi @softwareguy74, I was reviewing my code and I guess that I removed this limit too in Storage. There is no limit anymore.

LiteDB uses UInt32 as PageID to link lists. So, datafile limit is theoretically 4,096 * 4,294,967,295 = 16Tb

[softwareguy74]

Nice, that's great news!

[DeanVanGreunen]

add a UInt64 optional build instead of UInt32, you can get much more as it is 64bits wide.
currently working on hardware/software as a project, that you can have a uint4096 or an unsigned integer of the bit size 4096 bits wide. so just think of that storage and memory possibilities.

[mbdavid]

@DeanVanGreunen using UInt64 will consume 8bits per page + per page reference (and need page ref lot of times). It's better use 8192 bytes per page to double disk limit (to 32TB). Has an issue about change page size to 8192 to test if has better results, but it's not conclusive. I prefer now use 4096 to be same size of ntfs

[DeanVanGreunen]

@mbdavid well with that said, then so be it.

On Tue, Mar 28, 2017 at 1:53 PM, Mauricio David [e-mail]
wrote:

@DeanVanGreunen https://github.com/DeanVanGreunen using UInt64 will
consume 8bits per page + per page reference (and need page ref lot of
times). It's better use 8192 bytes per page to double disk limit (to 32TB).
Has an issue about change page size to 8192 to test if has better results,
but it's not conclusive. I prefer now use 4096 to be same size of ntfs

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mbdavid/LiteDB/issues/549#issuecomment-289745906, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AVQ2C9HzYC5FDSVJ3F4D8Hd8ExQDBySgks5rqPSngaJpZM4MpGLw
.

--
Dean Van Greunen
0728855371

[eldiablox]

I have a 2.5gb db file that i can't seem to access with LiteDbExplorer or LiteDbViewer -- any workarounds? to my knowledge, the file is not corrupt.