Xgboost: Model reproduciblility with histogram tree method.

[trivialfis]

A link to the original issue: https://github.com/dask/dask-xgboost/issues/37

TODOS:

• [x] CPU/GPU deterministic scattered add for building histogram.
• [x] Verify allreduce is deterministic.
• [x] Verify cub BlockSum is deterministic.
• [ ] Distributed environment.

Related:
https://github.com/dmlc/xgboost/issues/4204
https://github.com/dmlc/xgboost/issues/3921
https://github.com/dmlc/xgboost/issues/3707

[trivialfis]

It would be nice to test for hist and gpu_hist too, as these two are most used in production env.

[mmccarty]

Thank you @trivialfis Very interested to find out what's going on here.

[trivialfis]

Single node GPU hist for regression and classification is now deterministic.

[trivialfis]

Remaining issues are dask partitioning functions and GPU ranking. Ranking is tracked in https://github.com/dmlc/xgboost/issues/5561 . Dask partitioning still needs some more investigation.

[trivialfis]

The histogram method inside xgboost is bit to bit reproducible now. Remaining question is in dask data partitioning.