Sqlite_orm: Too large size in huge project

[FirasAtaya]

We are using sqlite_orm in a very huge project, we are generating a static library which is being used on different platforms, the problem is that including sqlite_orm is causing the generated .a library to be very huge.

We did a test by removing a single table from the schema which reduced the size by 10MB, this is how we are defining the schema in "Database.h" file:

class Database {

    public:

   ....

    inline static auto createTooltip() {

            return sqlite_orm::make_table("Gi4oJU05SQ==",
                                                 sqlite_orm::make_column("id", &Tooltip::id, sqlite_orm::primary_key()),
                                                 sqlite_orm::make_column("OigzJVw=", &Tooltip::title),
                                                 sqlite_orm::make_column("Jy80IF44TWEhJ10=", &Tooltip::insightType),
                                                 sqlite_orm::make_column("OiQ/PQ==", &Tooltip::text),
                                                 sqlite_orm::foreign_key(&Tooltip::insightType).references(&InsightType::id));

        }

       static auto makeStorage(const int & region) {

            //get database path
            string path = Setting::getDBName(region);

            //connect to database
            auto storage = makeStorageSchema(path);

            return storage;

        }

and in the .cpp files we are including "Database.h" and then when we need the database we are calling

vector<Tooltip> tooltips = Database::makeStorage(region).get_all<Tooltip>(where(c(&Tooltip::insightType) == insightType.id));

Any idea what could be causing this issue and how can we possibly reduce the size?

[fnc12]

The reason is simple - templates. Templates compilation require body of template class/function in every place. The only thing of solving your issue is using extern templates. I've never used it but it must help. Main advantage of extern templates is that you are not including templates everywhere you need it but define used template specialisations with extern keyword which tells compiler not to compile specified specialisation but believe that this specialisation is compiled somewhere else. So you can mark all sqlite_orm template specialisations as extern except one place and your project will still be able to compile but will have reduced size. http://blog.bitwigglers.org/extern-templates/

[FirasAtaya]

@fnc12 thank you very much for your response, actually, the example you provided is clear enough but the code in the library is quite complicated.

How exactly should I add the extern when I'm using the ORM, should I add it in the cpp files where I'm using the library? or where I'm defining the schema?

Would you please provide an example of how to do this?

Thanks in advance for your support

[fnc12]

Let's assume we want to use extern template for std::vector. We must take every compilation unit (.cpp file I suppose) where std::vector is used and write in every cpp file where std::vector is used extern template expression with every specialisation used in that cpp file. So you tell your compiler not to compile these specialisations but find them in another compilation unit during linking. The only thing left: you must compile every specialisation at least once in one of compilation units. You can either omit extern template in one of compilation unit per every specialisation or create one special .cpp file with all specialisations.
Example:
Assume you use std::vector<int>, std::vector<std::string> and std::vector<float> in your project. And you have 10 .cpp files and these files use one or more std::vector specialisations.

namespace std{
    extern template class std::vector<int>;
    extern template class std::vector<std::string>;
    extern template class std::vector<float>;
}

Here we defined all three specialisations in our cpp file where these specialisations are used. It means compiler will not compile them in this compilation unit. Also provide same extern expressions in every cpp file where you do not want to compile it. And then create one dedicated cpp file with explicit specialisation (not extern) to compile these specialisations:

namespace std{
    template class std::vector<int>;
    template class std::vector<std::string>;
    template class std::vector<float>;
}

This last one cpp file is the only file which compiles these specialisations. Profit.
If you understand all this you probably have a question: how to perform the same thing in sqlite_orm. The point is that almost all APIs are templated. So there is a lot of work to do. To understand which template functions and classes are used exactly you can use nm utility just like author used here. nm shows all symbols compiled in a special compilation unit. Look through all sqlite_orm template functions and classes with W type (weak) just like it is described in the article.
Yes it sounds complicated but it is the only option to reduce compilation time and file size with templates. Same issue faces everyone who uses boost lib. And I didn't find any tool to automate extern template creating in code. In a perfect world C++ modules must solve this problem - modules must link with each other with extern technique implicitly. But modules do not exist right now.
I'd like to create a tool which analyses cpp code AST and generates extern template code itself. It can reduce compilation time at my primary job and for people like you who uses my lib. But I need time to do it. Please feel free to ask any issue at any time and thanks for your patience about templates imperfection

[FirasAtaya]

Great explanation, thanks @fnc12.

However, we have tried the nm tool, the problem is that we are not finding any w weak type in the results, do you have an idea what could be the reason?

[fnc12]

show me your output please. You must see some symbols from sqlite_orm:: namespace

[FirasAtaya]

Here is the output of one of the small files which contains only one function:

https://drive.google.com/open?id=1h_NH9F23227bVKNOSubKLqUS1jhwUI0m

As you can see there are symbols from sqlite_orm but nothing is w

[fnc12]

Hm, ok, maybe I made a mistake with w. Please ignore it and take a look at symbols in sqlite_orm_secure namespace. BTW why sqlite_orm_secure not sqlite_orm?
E.g. line 763 has compiled symbol of make_index template function:

00000000000021b0 T sqlite_orm_secure::internal::index_t<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > CerboCore::Answer::*> sqlite_orm_secure::make_index<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > CerboCore::Answer::*>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > CerboCore::Answer::*)

This is a specialisation for function

    template<class ...Cols>
    internal::index_t<Cols...> make_index(const std::string &name, Cols ...cols) {
        return {name, false, std::make_tuple(cols...)};
    }

Where Cols is std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > CerboCore::Answer::* or std::string CerboCore::Answer::* (a member pointer to string from CerboCore class.
To declare it as extern template write

namespace sqlite_orm_secure{
   namespace internal{
        using MemberType = std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > CerboCore::Answer::*;
        extern template< MemberType >
        index_t< MemberType > make_index(const std::string &, MemberType);
    }
}

Also how many compilation units do really use sqlite_orm functions?

[FirasAtaya]

I tried your suggested solution and it seems to be working, it reduced the compilation time and the size, I'll continue working on this and I think it will make a big difference.

About the tool which analyses and generates the extern template, I think we can help to build this tool especially that we really need it now and it's almost impossible to do this work manually on each update for the DB schema, but I need your help on how are you thinking to implement it and what is the scope you have in mind.

Here are the answers to your questions:

why sqlite_orm_secure not sqlite_orm?

It's because we are encrypting/decrypting the data when we are dealing with the DB, so we added a security layer to the functions

how many compilation units do really use sqlite_orm functions?

We have about 32 cpp files including the sqlite_orm library

Thank you very much for your help, it's really appreciated

[fnc12]

I have already tried to build such a tool with C++ and libclang. Main issue I faced is that template variable declaration is ignored within AST visiting. I want to make AST parser which parses every cpp file and constructs AST (abstract source tree). Next tool must analyse source tree and find every template class instantiation and every template function call (including member function calls). Next tool must modify cpp file with extern template declarations in the beginning of cpp file. Next tool must specify explicit specialisations it found in special cpp file. E.g. all std::vector specialisations in vector.cpp file, all sqlite_orm in sqlite_orm.cpp file etc. And we should run this tool after every commit locally of once a day on remote within CI environment. Now I think to use another lib. It is a high level frontend for libclang.

[FirasAtaya]

@fnc12 sorry for the late reply, but I have been trying the suggested solution and actually the results were not that good.

I wrote a file which includes all the possible "extern" generated from the tool, and the file contained about 2000 lines, and the size was reduced only by about 10 MB which is about 370 MB.

I have tried another solution which is generating a single .cpp file which contains all the code including your "sqlite_orm" core and compiled it but with the same results.

I'm a bit lost here, could it be possibly a reason other than templates? shouldn't the issue be resolved by adding the code in a single .cpp file resolve the issue?

[fnc12]

show me your extern code please. Probably you externed only classes not functions.

[FirasAtaya]

the file was removed actually because the results didn't work and unfortunately I don't have it now, but the file contained extern for make_index and make_table and other functions if that what you mean

[fnc12]

ok, I need to think and analyze your output once more. I'll write later

[fnc12]

@FirasAtaya looks like we need to dive even deeper: can you please provide compilation unit sizes and their nm demanle output? I think we need to analyse size of every unit with and without template classes/functions specialisations. You can write to e-mail if it is more comfortable for you or you can tel which IM client you can chat live

[FirasAtaya]

thanks @fnc12 that would you be perfect, I have just sent you an email

[heilkn]

Good evening!

column_t has a lot of template parameters which results in many template instantiations. You could try replacing class O and class T by the member_pointer_t.
template <class T> struct deduce_types { }; template <class T, class U> struct deduce_types<T U::*> { using object_t = U; using field_t = T; }; auto p = &User::name; using field_type = deduce_types<decltype(p)>::field_t; using object_type = deduce_types<decltype(p)>::object_t ;
It could result in less instantiations. Maybe, you can even get away having only the getter and setter parameter.

[fnc12]

It is a good idea. Also there are getter_traits and setter_traits in sqltie_orm already. I think I can use them to avoid extra template arguments. Thanks

[fnc12]

v1.4 must solve this issue. Also the issue is solved personally with @FirasAtaya so I am closing this one