Or-tools: Fatal protobuf exception for sat.CpSolver

[ppallesen]

I have a cpsat.solver model which fails to solve by given this error for a given instance on a server, the model is develop in python and works on my local machine. Both server and local machines runs using a pipenv enviroment. Both server and local machine is windows 10 any idea for such an error can occur? Thanks for your help.

[libprotobuf FATAL /Users/corentinl/work/master/dependencies/sources/protobuf-3.6.1/src/google/protobuf/message_lite.cc:71] CHECK failed: (bytes_produced_by_serialization) == (byte_size_before_serialization): Byte size calculation and serialization were inconsistent.  This may indicate a bug in protocol buffers or it may be caused by concurrent modification of operations_research.sat.CpSolverResponse.
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: (bytes_produced_by_serialization) == (byte_size_before_serialization): Byte size calculation and serialization were inconsistent.  This may indicate a bug in protocol buffers or it may be caused by concurrent modification of operations_research.sat.CpSolverResponse.

[Mizux]

our ortools python package is based on protobuf 3.6.1. Since it is a native library iguess you need to verify that protobuf on server is also the version 3.6.1.
note: Protobuf just release the 3.7.0 https://pypi.org/project/protobuf/#history

python -m pip show ortools
python -m pip show protobuf

[Mizux]

I'll try to generate a version with protobuf 3.7.0 soon

[ppallesen]

The pipenv environment is specificed to ortools 6.9 and the protobuf is version 3.6.1. But the problem is that this error is fatal and leads to an entire application break down. So do you have any idea how I with Python googleortools can generate generate: "concurrent modification of operations_research.sat.CpSolverRespons".
This discussion https://groups.google.com/forum/#!msg/protobuf/8urLQSXwzOI/jh1ot0M8gSIJ points at potential courses. Where one is that the use of uninitialized booleans can lead to memory corruption. So do you think this can be posible that google ORtools generates these?

[Mizux]

up to v6.10 we have proto files with booleans:
e.g.
https://github.com/google/or-tools/blob/fbd78928deecbcf13b7da3286f2c60eb568dc62d/ortools/constraint_solver/routing_parameters.proto#L213
they have been replaced by an OptionalBoolean object on the last v7.0 release...
https://github.com/google/or-tools/blob/e7a9bbfae8bc57de414814553f82a9d01722d295/ortools/constraint_solver/routing_parameters.proto#L292

Don't know if it may fix your issue....

EDIT: Sat CpModel.proto still use few booleans

[ppallesen]

Potentially. I have difficulty recreating the problem. I think it is very difficult to develop a fix with the current knowledge. I suggest closing the issue until another person encounters this problem. Because it might be an error with the google protocol buffer or a C-compiler vs hardware problem as stated by Niklas Hambüchen in https://groups.google.com/forum/#!msg/protobuf/8urLQSXwzOI/jh1ot0M8gSIJ . So I'm not sure it is a googleORtools bug.

[Mizux]

I agree with you, so i'll close this bug. But feel free to re-open it if you want.

note: In the link you pass there are saying to use -march=haswell -mno-lzcnt instead of -march=haswell BUT since on windows we are compiling/providing OR-Tools using VS 2017 compiler, don't know if it's still relevant and/how we can pass this flag

note2: AFAIK on the Makefile based-build (used to generate archives) we are NOT passing any -march options...

BTW thanks a lot for the investigation !

[koirikivi]

I encountered this issue with macOS High Sierra, Python 3.7.3, ortools 6.10.6025 and protobuf 3.7.0.
Downgrading protobuf to 3.6.1 fixed the issue. Thanks!

[Mizux]

@koirikivi please note last release v7.0.6546 have been generated using Protobuf 3.7.1 IIRC
EDIT: false, it was generated using Protobuf 3.6.1 see comment below

[andy-kimball]

I'm having the same issue running this simple code:

def SolveTest():
    model = cp_model.CpModel()

    x = model.NewIntVar(0, 1000, 'x')
    y = model.NewIntVar(0, 1000, 'y')
    model.Add(x + y == 1000)

    solver = cp_model.CpSolver()
    status = solver.Solve(model)

[libprotobuf FATAL /Users/corentinl/work/master/dependencies/sources/protobuf-3.6.1/src/google/protobuf/message_lite.cc:71] CHECK failed: (bytes_produced_by_serialization) == (byte_size_before_serialization): Byte size calculation and serialization were inconsistent.  This may indicate a bug in protocol buffers or it may be caused by concurrent modification of operations_research.sat.CpSolverResponse.
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: (bytes_produced_by_serialization) == (byte_size_before_serialization): Byte size calculation and serialization were inconsistent.  This may indicate a bug in protocol buffers or it may be caused by concurrent modification of operations_research.sat.CpSolverResponse.
Abort trap: 6



md5-1b0dc91c82c50213ea252d746ff8d0b8



~/python/constraints python -m pip show ortools
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Name: ortools
Version: 7.0.6546
Summary: Google OR-Tools python libraries and modules
Home-page: https://developers.google.com/optimization/
Author: Google Inc
Author-email: [email protected]
License: Apache 2.0
Location: /Users/andyk/Library/Python/2.7/lib/python/site-packages
Requires: protobuf, six
Required-by:
~/python/constraints python -m pip show protobuf
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Name: protobuf
Version: 3.7.1
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: None
Author-email: None
License: 3-Clause BSD License
Location: /Users/andyk/Library/Python/2.7/lib/python/site-packages
Requires: setuptools, six
Required-by: ortools

Downgrading to protobuf 3.6.1 fixed the issue as well, but thought I'd comment given @Mizux's comment above that v7.0.6546 was generated with protobuf 3.7.1. Something's wrong there.

[Mizux]

my bad from stable (v7.0):
https://github.com/google/or-tools/blob/39f44709bba203f5ff3bc18fab8098739f189a6d/makefiles/Makefile.third_party.unix.mk#L24

but master:
https://github.com/google/or-tools/blob/3bc57f1aec91a31cb9c34c7a7a6d7a69986225d6/makefiles/Makefile.third_party.unix.mk#L24

Hope to release the v7.1 soon.....

EDIT: stable now point to v7.1 (Released in May 2019)

[Mizux]

@ppallesen why did you reopen this issue ?

note: Protobuf 3.8.0 is still not released (only in preview/RC currently)
see: https://pypi.org/project/protobuf/#history

[ppallesen]

Because of the original argument for closing it was that I could not replicate it. So I reopened it to ensure that it was either closed with a proper solution or stayed reopened until it was proper solution was in place. Sorry for not commenting on this. But feel free to close it again.

[Mizux]

Protobuf mismath between pypi protobuf package and ortools package.
Should be fixed now (v7.1/May 2019) until the next Protobuf release....

For the record:
v6.10:
https://github.com/google/or-tools/blob/554cbccaa95b4c11eced13f145de1bf468e1f919/makefiles/Makefile.third_party.unix.mk#L23

v7.0-beta
https://github.com/google/or-tools/blob/769d9d3ff4773869c3420029d0109c0f0e3936b1/makefiles/Makefile.third_party.unix.mk#L24

v7.0
https://github.com/google/or-tools/blob/39f44709bba203f5ff3bc18fab8098739f189a6d/makefiles/Makefile.third_party.unix.mk#L24

v7.1
https://github.com/google/or-tools/blob/f3fd201e68cf75b7720ff5c3cadc599a1d02b54b/makefiles/Makefile.third_party.unix.mk#L24