Cudf: [FEA] gpuCI should run tests under cuda-memcheck

[jrhemstad]

Is your feature request related to a problem? Please describe.
Memory errors can occur in unit tests that aren't reported as failed tests.

Describe the solution you'd like
In addition to running our tests as they are now, we should also run our Python and C++ unit tests under cuda-memcheck and valgrind in order to catch any uncaught memory errors.

[felipeblazing]

Yes please for the love of everything that is universe

[jrhemstad]

Revisiting this issue, given that today our CI takes quite a while to run, I think running under cuda-memcheck is more of a nightly kind of thing.

[jlowe]

After helping to chase down #5792 I would love to see this be run on a regular basis. Running this nightly or even only weekly seems like a good compromise.

[mnicely]

@awthomp We should consider this as well for cuSignal

[harrism]

@jlowe we would have to disable pool allocation in tests to make cuda-memcheck work for leak checking.

[jlowe]

we would have to disable pool allocation in tests to make cuda-memcheck work for leak checking.

Sure, but I don't see that as a blocker. Should be straightforward to update the unit tests to use the pool allocator based on a cmake flag. The nightly/weekly memcheck job can build with the cmake flag set appropriately. Devs may also appreciate this flag if they want to run one or more unit tests under memcheck manually as part of their own testing before submitting a PR.

[jrhemstad]

Should be straightforward to update the unit tests to use the pool allocator based on a cmake flag.

It doesn't even need to be a cmake flag. It's already possible to change the allocator via a command line argument to any test binary.

https://github.com/rapidsai/cudf/blob/9e5196da813568450da1e4549b1baab2d54753dd/cpp/tests/utilities/base_fixture.hpp#L250-L268

[harrism]

Nice. BTW, I wasn't saying it's a showstopper. Just pointing out a requirement.