Protobuf: error when I reload dynamic library which links against shared protobuf lib

[wawade]

Hi,

I have a plugin architecture which will load shared libraries at run time. The shared libraries are linked against protobuf shared library (3.5.0). When I do load and unload, everything is fine, but when I do load again, I get the following error message:

[libprotobuf ERROR /home/ling/docker_dev/src/Packages/protobuf/src/google/protobuf/descriptor_database.cc:58] File already exists in database: google/protobuf/wrappers.proto
[libprotobuf FATAL /home/ling/docker_dev/src/Packages/protobuf/src/google/protobuf/descriptor.cc:1394] CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of 'google::protobuf::FatalException'
what(): CHECK failed: generated_database_->Add(encoded_file_descriptor, size):
Backend got SIGABRT !

My understanding is that some global variables are initialized during the first load. And when I unload my shared libraries, the protobuf-generated global variables are not cleaned up and they are initialized the second time when I call load again. Then something goes wrong... Am I correct? I have checked ShutdownProtobufLibrary, but it doesn't work for me and I only want to clean the protobuf data generated in my shared library, not overall. And I do not want to use lite version.

Do you have an idea how to fix this? I am looking forward to your reply.

Best, Siqi

[achilsh]

how to fix this issue? I has this problem but not resolve. if you has an idea, please notify me.

[achilsh]

have any solution for this problem? or other pb version can do it?

[wawade]

@achilsh I also want to know.

[wawade]

1489 #1941 #3782

Hi, @xfxyjwf ,
in all the similar problems I found, the solution is to use a separated shared proto or to use protobuf lite. I want to know if there are other solutions, e.g. when I load the same proto file again, just ignore it, or I can unload (remove a proto file). Do you have a suggestion or idea how to implement such a function within protobuf source? I would appreciate it.

[xfxyjwf]

@wawade Protobuf C++ is not designed/implemented to be a re-loadable module so it's likely very hard to make it work. I don't have any better suggestions other than not unloading protobufs...

[neundorf]

I would also like to have that functionality.

Looking at the code, it seems to be possible ?
I mean, instead of erroring out if the proto file already exists in the database, how about checking whether the new proto file is exactly the same as the one which is already in the database (in InternalAddGeneratedFile() and InternalRegisterGeneratedFile()), e.g. by comparing bytewise whether the content of the new descriptor is identical to the content of the already existing descriptor, and if so, accept it, and instead of adding it, increase a refererence counter ?

[xfxyjwf]

@neundorf I think that only fixes the surface of the problem. Apparently after unloading, some descriptor data is left behind in the global database. They are likely unusable because the code is unloaded and could crash the program if triggered. For example, there is a global factory mapping descriptor to the generated message object:
https://github.com/google/protobuf/blob/master/src/google/protobuf/message.h#L1071

I have no idea what will be returned after unloading the generated code.

Simply avoiding adding the descriptor twice doesn't work because the above factory mapping must be rebuilt to point to the new generated message type at a different memory location. Likewise, other global registries like the extension registry must be rebuilt as well. But all of those global structures lack the ability to be mutated after construction and would require major refactoring to do so.

You can probably avoid an immediate crash doing what you proposed at reloading time, but the inconsistency of protobuf's global state can cause problems later on when the protos are actually used (in particular, the reflection/descriptor functionality is likely very broken at that point).

[neundorf]

I'll dig into the code a bit.

[neundorf]

It seems I got around these problems by using LITE_RUNTIME.
Then I don't get the "file already in database" error and it seems everything is working... ?