Cuml: [FEA] Speed up RF -> FIL conversion for inference

Created on 11 Jun 2020  ·  8Comments  ·  Source: rapidsai/cuml

In today's nightly (cuml commit f1f1c7f6a), the predict method of random forest classifier takes quite a bit of time the first time it's called on 1M rows binary classification, but is much faster the second time. Perhaps this could be related to #1922 ? 

import cupy as cp
from sklearn.datasets import make_classification
from cuml.ensemble import RandomForestClassifier as gpu_rf
​
X, y = make_classification(
    n_samples=1000000,
    n_features=20,
    n_informative=18,
    n_classes=2,
    random_state=0,
)
​
n_trees = 300
​
X = X.astype("float32")
y = y.astype("int32")
​
gX, gy = cp.asarray(X), cp.asarray(y)
​
clf1 = gpu_rf(n_estimators=n_trees)
clf1.fit(gX, gy)
​
%time clf1.predict(gX)
%time clf1.predict(gX)
CPU times: user 11.4 s, sys: 3.08 s, total: 14.5 s
Wall time: 13.6 s
CPU times: user 2.14 s, sys: 397 ms, total: 2.54 s
Wall time: 2.53 s
array([1., 0., 1., ..., 0., 1., 1.], dtype=float32)

After this, I added a print statement in the predict method to see if it's using the GPU path, which it appears to be.

https://github.com/rapidsai/cuml/blob/4b3213d9dac68c9dce4600f82305c2989ce67c2d/python/cuml/ensemble/randomforestclassifier.pyx#L869-L871

import cupy as cp
from sklearn.datasets import make_classification
from cuml.ensemble import RandomForestClassifier as gpu_rf
X, y = make_classification(
    n_samples=1000000,
    n_features=20,
    n_informative=18,
    n_classes=2,
    random_state=0,
)
n_trees = 300
X = X.astype("float32")
y = y.astype("int32")
gX, gy = cp.asarray(X), cp.asarray(y)
clf1 = gpu_rf(n_estimators=n_trees)
clf1.fit(gX, gy, convert_dtype=False)
%time clf1.predict(gX)
%time clf1.predict(gX)
using GPU
CPU times: user 11.4 s, sys: 1.34 s, total: 12.7 s
Wall time: 11.7 s
using GPU
CPU times: user 1.71 s, sys: 259 ms, total: 1.97 s
Wall time: 1.97 s

clf1 = gpu_rf(n_estimators=n_trees)
clf1.fit(gX, gy, convert_dtype=True)
%time clf1.predict(gX)
%time clf1.predict(gX)
using GPU
CPU times: user 11.2 s, sys: 1.21 s, total: 12.4 s
Wall time: 11.4 s
using GPU
CPU times: user 1.64 s, sys: 312 ms, total: 1.95 s
Wall time: 1.95 s
feature request
Source
picture beckernick

Most helpful comment

@hcho3 looks like the serialization changes made a meaningful improvement. The below example is from the 2020-07-01 nightly as of 3 PM EDT. Looks to be about 4 seconds shaved off, or about 1/3 of the time.

import cupy as cp
from sklearn.datasets import make_classification
from cuml.ensemble import RandomForestClassifier as gpu_rf
​
X, y = make_classification(
    n_samples=1000000,
    n_features=20,
    n_informative=18,
    n_classes=2,
    random_state=0,
)
​
n_trees = 300
​
X = X.astype("float32")
y = y.astype("int32")
​
gX, gy = cp.asarray(X), cp.asarray(y)
​
clf1 = gpu_rf(n_estimators=n_trees)
clf1.fit(gX, gy)
​
%time clf1.predict(gX)
%time clf1.predict(gX)
CPU times: user 7.57 s, sys: 1.21 s, total: 8.77 s
Wall time: 7.87 s
CPU times: user 878 ms, sys: 345 ms, total: 1.22 s
Wall time: 1.22 s
array([1, 0, 1, ..., 0, 1, 1], dtype=int32)

The conversion slowdown appears strongly related to the number of features. While not surprising, it's interesting to see it play out. I wonder if there is an inflection point.

import cupy as cp
from sklearn.datasets import make_classification
from cuml.ensemble import RandomForestClassifier as gpu_rf
​
​
n_trees = 300
​
for nfeat in [5, 10, 15, 20]:
    X, y = make_classification(
        n_samples=1000000,
        n_features=nfeat,
        n_informative=nfeat-2,
        n_classes=2,
        random_state=0,
    )
​
    X = X.astype("float32")
    y = y.astype("int32")
​
    gX, gy = cp.asarray(X), cp.asarray(y)
​
    clf1 = gpu_rf(n_estimators=n_trees)
    clf1.fit(gX, gy)
    print(f"{nfeat} Features")
    %time clf1.predict(gX)
    print()
5 Features
CPU times: user 1.33 s, sys: 35.9 ms, total: 1.36 s
Wall time: 404 ms

10 Features
CPU times: user 5.45 s, sys: 687 ms, total: 6.14 s
Wall time: 5.24 s

15 Features
CPU times: user 6.19 s, sys: 785 ms, total: 6.97 s
Wall time: 6.43 s

20 Features
CPU times: user 7.23 s, sys: 570 ms, total: 7.8 s
Wall time: 6.88 s

Paying the one-time cost is probably more impactful in a cross-validation workflow, in which potentially many unique models call predict over the lifecycle. We'd end up with a linear lower bound on total time of num_models x RF/FIL conversion time. With that said, if it's still faster than other approaches, we're still coming out ahead.

picture beckernick on 2 Jul 2020
🎉2 👍2

All 8 comments

The time for the first prediction includes conversion to FIL format. The FIL-converted tree is cached after the first call. If you have a tiny dataset and don't want to pay the setup cost, you can use the CPU-based predict call, which is much lower throughput but lower latency.

JohnZed picture JohnZed on 11 Jun 2020

I'm going to rephrase this as a feature request to speed up FIL translation and leave it in the FEA queue.

JohnZed picture JohnZed on 12 Jun 2020

Thanks @JohnZed ! That sounds great

beckernick picture beckernick on 12 Jun 2020

Note that #2263 was merged yesterday, which speeds up serialization of RF objects. We should run the benchmark again to obtain the new measurement for RF->FIL conversion.

hcho3 picture hcho3 on 19 Jun 2020

@hcho3 looks like the serialization changes made a meaningful improvement. The below example is from the 2020-07-01 nightly as of 3 PM EDT. Looks to be about 4 seconds shaved off, or about 1/3 of the time.

import cupy as cp
from sklearn.datasets import make_classification
from cuml.ensemble import RandomForestClassifier as gpu_rf
​
X, y = make_classification(
    n_samples=1000000,
    n_features=20,
    n_informative=18,
    n_classes=2,
    random_state=0,
)
​
n_trees = 300
​
X = X.astype("float32")
y = y.astype("int32")
​
gX, gy = cp.asarray(X), cp.asarray(y)
​
clf1 = gpu_rf(n_estimators=n_trees)
clf1.fit(gX, gy)
​
%time clf1.predict(gX)
%time clf1.predict(gX)
CPU times: user 7.57 s, sys: 1.21 s, total: 8.77 s
Wall time: 7.87 s
CPU times: user 878 ms, sys: 345 ms, total: 1.22 s
Wall time: 1.22 s
array([1, 0, 1, ..., 0, 1, 1], dtype=int32)

The conversion slowdown appears strongly related to the number of features. While not surprising, it's interesting to see it play out. I wonder if there is an inflection point.

import cupy as cp
from sklearn.datasets import make_classification
from cuml.ensemble import RandomForestClassifier as gpu_rf
​
​
n_trees = 300
​
for nfeat in [5, 10, 15, 20]:
    X, y = make_classification(
        n_samples=1000000,
        n_features=nfeat,
        n_informative=nfeat-2,
        n_classes=2,
        random_state=0,
    )
​
    X = X.astype("float32")
    y = y.astype("int32")
​
    gX, gy = cp.asarray(X), cp.asarray(y)
​
    clf1 = gpu_rf(n_estimators=n_trees)
    clf1.fit(gX, gy)
    print(f"{nfeat} Features")
    %time clf1.predict(gX)
    print()
5 Features
CPU times: user 1.33 s, sys: 35.9 ms, total: 1.36 s
Wall time: 404 ms

10 Features
CPU times: user 5.45 s, sys: 687 ms, total: 6.14 s
Wall time: 5.24 s

15 Features
CPU times: user 6.19 s, sys: 785 ms, total: 6.97 s
Wall time: 6.43 s

20 Features
CPU times: user 7.23 s, sys: 570 ms, total: 7.8 s
Wall time: 6.88 s

Paying the one-time cost is probably more impactful in a cross-validation workflow, in which potentially many unique models call predict over the lifecycle. We'd end up with a linear lower bound on total time of num_models x RF/FIL conversion time. With that said, if it's still faster than other approaches, we're still coming out ahead.

beckernick picture beckernick on 2 Jul 2020
🎉2 👍2

Update: Pickle protocol 5 speeds up RF -> FIL conversion further. It uses a technique called "out-of-band serialization" to speed up conversion between NumPy arrays and bytes.

Benchmark setup

  • Using this script by @Salonijain27 and @hcho3. See the link for instructions. To use a more recent version of cuML, it suffices to copy over make_dataset.py and rf_benchmark.py to newer cuML source directory.
  • AWS EC2 instance g4dn.12xlarge, T4 GPU (16 GB GDDR6) X 4
  • Benchmark setting: n_gpus=2, n_gb=2, n_features=20, depth=25, n_estimators=10. This leads to a forest consisting of 10 depth-25 trees, and we run through 25 million data rows.

Benchmark Results

| | End-to-end time for prediction (sec) |
|---|---|
| Before #2263 | 269.0 sec |
| Most recent commit (489a7d80908271245152d7d0be7a32f4faf68928) | 108.4 sec |
| Most recent commit (489a7d80908271245152d7d0be7a32f4faf68928) + Pickle5 | 73.4 sec |

As noted in #2263, most of the run time is consumed by RF->FIL conversion.

How to opt into Pickle 5

There are two options:

  1. Upgrade to Python 3.8. In this case, the built-in pickle module will use Pickle protocol 5. OR
  2. Install latest versions of cloudpickle, pickle5, distributed by running the following commands:
conda install -c rapidsai -c nvidia -c rapidsai-nightly -c conda-forge cloudpickle pickle5
# Install development version of Dask and Distributed
conda remove --force distributed dask
git clone https://github.com/dask/dask.git
cd dask
python -m pip install .
cd ..
git clone https://github.com/dask/distributed.git
cd distributed
python setup.py install

Special thanks to @jakirkham who brought Pickle 5 to Dask.

hcho3 picture hcho3 on 15 Jul 2020
🎉2

As a note, the Distributed change ( https://github.com/dask/distributed/pull/3849 ) will be part of the 2.21.0 release.

jakirkham picture jakirkham on 15 Jul 2020
🚀1

Awesome benchmark and summary @hcho3 . Do you have a sense of how 73 seconds for prediction compares to sklearn's random forest on the same data?

beckernick picture beckernick on 27 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

teju85 picture teju85  ·  3Comments
aerdem4 picture aerdem4  ·  3Comments
gumdropsteve picture gumdropsteve  ·  3Comments
teju85 picture teju85  ·  7Comments
MurrayData picture MurrayData  ·  6Comments