Sqlite_orm: Building a large storage consumes lots of memory

[TwelveNights]

I'm running into an issue where the build consumes almost all the memory on my device (16GB RAM). I have two storages in the same cpp file. Each storage has ~5 tables, and each table has ~6 columns. The issue arises when I compile with -Ofast.

Have you had this issue before?

[fnc12]

Is storage definition located in header file or cpp file?

[TwelveNights]

The definition is located in the cpp file.

[fnc12]

what compiler do you use?

[TwelveNights]

I'm using g++ 8.2.0, compiling on an x86 host to arm.

[fnc12]

if you comment make_storage call the amount of RAM reduces?

[TwelveNights]

The application compiles when I remove some columns

[fnc12]

5 tables is not so many. It is weird that compilation takes up to 16 Gb RAM. Can you send me your code? I want to compile it on my machine. I have 16 Gb RAM too. You can send it to my email [email protected]

[TwelveNights]

Sent an email :)

[ChineseBoyLY]

error: cc1plus: out of memory allocating 268435144 bytes after a total of 52224000 bytes

[alessandroferraioli]

Hi, I'm facing the same issue.

I have the storage definition into my .h file , and when I try to compile the code I get back this error: cc1plus: out of memory allocating 67108816 bytes after a total of 77172736 bytes

I'd like to point out that I'm currently developing under Qt creator, and If I compile using the so called "debug" everything works. Unfortunately, whenever I try to compile in "release" , I get this error.

Do you have any idea how I can fix this issue ?

[fnc12]

@alessandroferraioli how many tables do you have in a single storage?

[alessandroferraioli]

@fnc12 it's about 19 tables.
I've tried to compile just the init of the storage as a library, but I have the same issue.
I have 8gb of ram, so I guess it should be enough.

[alessandroferraioli]

@fnc12 I genuinely know that it sounds weird, but apparently, I have this issue, as soon as a table becomes bigger than 18 columns. Does it make sense?

Sorry for the two messages in row

[fnc12]

omg this is how templates work. This is a very bad disadvantage. BTW how many RAM do you have on your build machine? I have an idea about how to reduce the amount of templates by using type erasure for column_t but I need to test it first.

[alessandroferraioli]

I have 8gb of ram

[alessandroferraioli]

@fnc12 I'm really sorry to bother you again, but since I'm running out of time I wanted to be sure.
Do you have any idea how long it'll take to figure out if the solution that you proposed actually works? Idk, maybe you have some suggestions for my case that I could try in the meanwhile. Really appreciate your help :)

[fnc12]

Well right now I'm in the middle of my primary job and it means I cannot code. I can take a look at it in the evening. My idea is to remove column_t's member pointers template arguments so there will be only one template argument. This will reduce template recursion and amount of symbols.
Quick solution for now is to split storage up to several storages with dedicated tables. Try to split it by two.

[fnc12]

@alessandroferraioli are you here? Did you try anything?

[alessandroferraioli]

@fnc12 Yes, I'm here. Sorry I have been busy implementing a new feature. Unfortunately, I cannot split the storage, since we have a requirement that tells us that the Db must belong to just one storage. Do you have any update from your idea?

[fnc12]

Idea is the same - column_t refactoring. It will reduce binary's size and probably will erase the current error from this thread. I added investigation cause I am not sure whether it will for 100%.

[fnc12]

Details of the idea: all operators that use member pointers (all three of them: field member pointer, getter member function pointer and setter function pointer) must be replaced with std::function with the same type. E.g. member pointer comparison must be replaced with std::function which compares member pointers inside but captures type in lambda capture and accepts arguments in some kind of common type like const void *. This is type erasure technique. I need to think more to understand how to implement it right cause I can not get how to accept member pointer in a common way. Casting member pointer to const void * leads to undefined behavior (I've already tried to do it). Probably we need to make a dummy object of field type and get address after member dereferencing (man I just figured it out and it sounds like a great solution).

[alessandroferraioli]

@fnc12 Thank you for your comment, as I already said, I really appreciate your support.
Do you think that this week, you will'be able to run some test?

[fnc12]

@alessandroferraioli I am not sure cause I don't know how to add type erasure for member function pointers. If someone has an idea it would be great.

[alessandroferraioli]

I'd like to help you, but unfortunately, I'm not skilled like you. Hopefully, someone will come up with some solution.

[gitmodimo]

I encounterd similar problems with ~20 columns and more. I will post some more details next week.

[alessandroferraioli]

@gitmodimo How much ram do you have?

[gitmodimo]

I have two build machines(win7 and linux). Both have 32GB of ram. GCC the same version on both, but different compiler options.
I will do some more testing next week and post more details.

[fnc12]

Oh 32GB is a lot. @gitmodimo try clang instead of gcc.

[gitmodimo]

Well I actually cannot because I am cross compiling for aarch64 with toolchain provided by third party.

[fnc12]

Damn. The problem exists when people use GCC and doesn't exist with clang. I have 16Gb RAM and I've never faced such a problem.

[alessandroferraioli]

@fnc12 Exactly. I just compiled my code with clang without errors

[fnc12]

Looks like it is a problem of GCC compiler only. If so it must be solved within GCC project by its developers.