Drake: Ensure that Scalar's used within Eigen::Matrix<> do not cause undefined behavior when the matrix is resized

[EricCousineau-TRI]

@jwnimmer-tri, @soonho-tri, and I did some digging, and realized that Eigen::Matrix<> will not play well with std::string (or any non-trivially copyable type) due to how Eigen's reallocation mechanisms work (by moving bytes, without calling a move constructor).
For std::string, the issue arise due to it's Short String Optimization. Per Jeremy's words:

For short strings, the data pointer points back into an offset of the this pointer, and there is no heap allocation.
But when you _reallocate_ where this is, the self-pointer is no longer valid.

Takeaway

We need to ensure that we use the correct code paths when an Eigen::Matrix<> needs to be resized when the Scalar type may not be ~Trivially Copyable~ memcpy-movable. The options are:

1. Ensure that the Scalar type is indeed trivially copyable. In this case, could mean:
   
   
   
   • Making any std::string (or other things that do weird pointer arithmetic) members either (a) pre-allocate via resize to avoid SSO, (b) change it to a std::string* and allocate it on the heap s.t. the pointer is trivially copyable, or (c) use a primitive type, such as char[N] or std::array<char, N> with a maximum size.
   
   
   • Jeremy mentioned that for this option, there should be a unittest that explicitly checks for compatibility with a memcpy primitive move operation.
   
   
2. Update Eigen to have an optional flag to use copy/move constructors with placement new / delete, to be robust to stuff like this. This would hurt efforts such as #5785, but would definitely be a good thing to contribute towards Eigen.
3. Try to ensure that conservativeResize calls the copy constructor when reallocating. In most cases, it may use placement new and delete. However, (I think) in others, it may allocate a temporary and do a block assignment (with implicit construction).
   
   
   
   • This enables non-trivial types. This seems to be triggered (at least in a test case) with conservativeResize(int, int). On the other hand, conservativeResize(int) seems to fail.
   
   

Notes

This does not seem to affect AutoDiff... stuff, since it uses Eigen containers which do not seem to do any SSO-like pointer ~hackery~ magic.

Repro Steps

Thus far, valgrind has only caught memory leaks on clang version 3.9.1-svn288847-1~exp1 (branches/release_39), on Ubuntu 16.04 LTS. Other compilers / platforms seem to show no issue.

Directory: eigen_resize_dtor
Branch: issue/eigen_override
Files:

• conservative_resize_issue.cc - Shows AppendToVector being implemented with conservativeResize(int) and conservativeResize(int, int).
• output.txt - The output from executing ./repro.sh.

Cases: (and SHA)

• Memory leak with short std::string scalars: b1085a0
• No leak with long std::string scalars: 2ffbfe8
• No leak with VectorXd scalars: fa68f30

[EricCousineau-TRI]

One additional note: I started investigating this because I was encountering hard crashes when using conservativeResize(int, int) on my machine.

[EricCousineau-TRI]

Per f2f with @soonho-tri and @jwnimmer-tri, the main thing to distill is that an object's representation must be invariant of its location in memory, hence the above edit from "Trivially Copyable" to "memcpy Movable".

[jwnimmer-tri]

FYI the issue title "Leak memory" is a bit misleading. "Crash and burn with extreme UB prejudice" is more like it.

[EricCousineau-TRI]

Tentatively closing this issue per @soonho-tri's PR #5983.

We can reopen or reference this issue if we encounter any other scalar types in the future.

[EricCousineau-TRI]

Posted on the Eigen forum to see what other solutions may be available. There is Eigen::internal::smart_copy that either does memcpy or std::copy based on Eigen::NumTraits<T>::RequireInitialization, so I'm checking to see if there is something like RequireMoveConstructor or what not:
Correct Method to use Non-Memcpy-Movable Types as Scalars?