cuml KMeans randomly terminates with 'thrust::system::system_error' on larger datasets/no of clusters

[MurrayData]

I pulled the new Rapids Docker container particularly to re-run a KMeans exercise on Twitter location data that I've previously run successfully in both TensorFlow and Scikit-Learn.

from cuml import KMeans as km
import cudf

names = ['0','1']
dtypes = ['float64','float64']
filename = "/data/twitter/cluster_points.csv"

clustering_cuml = km(n_clusters=100)
clustering_cuml.fit(gdf)

The gdf looks like this:

                    0                  1
 0          392159.91           223933.2
 1          434359.54          278703.86
 2  436988.1599999999          335566.98
 3 386173.63999999996 349452.80000000005
 4          275936.06  674298.0899999999
 5          432248.25          444924.63
 6 458423.64999999997  304714.0300000001
 7          591923.55          120227.19
 8          532864.35          182221.79
 9 336145.64999999997          390272.31
[10023370 more rows]

And in the Docker container it hangs. I noticed 2 things by monitoring the system in another window:

1) The GPU is not utilised, it remains at 0% in nvidia-smi
2) A single core of the CPU runs at 100% for quite some time and then the process stops but the notebook doesn't detect this.
3) The GPU RAM is freed upon termination, including the dataframe.

The log shows:

terminate called after throwing an instance of 'thrust::system::system_error'
  what():  trivial_device_copy D->H failed: invalid argument

I tried then tried using a smaller sample of just 10,000 records and the same thing happened.

I reduced the number of clusters to 10 and it worked fine. I then steadily increased the number of records back up to 10 million and found it would intermittently hang on some runs above 12 clusters, but then work other times. I then increased the number of clusters and found I could never get it to work beyond 20 clusters.

It's a difficult error to reproduce as it appears random although happens consistently on larger datasets. I always shutting down all kernels and restarting each time for a clean environment.

I've also tried it on different computers with different GPUs: GV100, Titan V and Titan Xp and experienced the same issue.

I also tried it outside Docker and the same thing happened.

[MurrayData]

Transferred from https://github.com/rapidsai/cudf/issues/459

[datametrician]

@dantegd would you or Devavret check this out

[raghavmi]

Pat Walters from Relay Therapeutics ran into a similar issue.

I was able to reproduce it on the latest Docker image.

Pat has the code & data to reproduce it in this repo:
https://github.com/PatWalters/gpu_kmeans

[PatWalters]

This seems to work well until the number of clusters gets large. I have a code example here
https://github.com/PatWalters/gpu_kmeans

In [2]: df = pd.read_csv("fp.csv")
In [3]: df.shape
Out[3]: (49999, 128)
In [4]: df.info(memory_usage='deep')

RangeIndex: 49999 entries, 0 to 49998
Columns: 128 entries, B000 to B127
dtypes: float64(128)
memory usage: 48.8 MB

Not, very big, but when I run, I get this
Clusters Runtime
1000 50.80
2000 50.13
3000 50.34
4000 50.39
5000 52.89
6000 52.92
7000 52.97
8000 52.02
Segmentation fault (core dumped)

Note that everything blows up at 9K clusters.

[dantegd]

Thanks for the reproducible code @PatWalters and @raghavmi ! We are looking into it, I think I've isolated the cause of the bug, if so a patch should be released this week.

[dantegd]

@MurrayData This should be solved by PR #96 but if you still see any issues (particularly after the release of 0.5) please keep the issues coming! Thanks!