cuml dbscan terminating on large datasets 'invalid configuration argument'

[MurrayData]

I have an issue with DBSCAN terminating on large datasets. I'm running the latest NGC Rapids Docker container. I've seen the comments in #31.

[I 14:36:16.181 LabApp] Kernel started: d712459f-cf0a-4c0e-a6ec-ecc73b9e91f1
[I 14:36:16.667 LabApp] Adapting to protocol v5.1 for kernel d712459f-cf0a-4c0e-a6ec-ecc73b9e91f1
[I 14:37:01.848 LabApp] Saving file at /cuml/dbscan_twitter.ipynb
terminate called after throwing an instance of 'std::runtime_error'
  what():  Exception occured! file=/rapids/cuml/cuML/src/dbscan/vertexdeg/algo5.h line=141: FAIL: call='res.result'. Reason:invalid configuration argument
[I 14:38:40.180 LabApp] KernelRestarter: restarting kernel (1/5), keep random ports
kernel d712459f-cf0a-4c0e-a6ec-ecc73b9e91f1 restarted

I'm using the same Twitter derived point dataset as in #53 and sampling it down until it works. The source data has 10.5 million points. The crashes happen above 5 million rows x 2 columns, row counts below that work fine. Minimum sample size set to 1000 eps to none.

In addition to using the Twitter file, I've also tested it with randomly generated data.

                    0                  1
 0 478401.76952889207  542950.5525448014
 1  454622.9484872194  463117.5441902199
 2 340651.60100943915   568573.833436874
 3  60462.91779449762  248186.3741022621
 4 290905.83582845033  564827.5589121555
 5  305875.7357089389  187773.6372960709
 6 122647.20323430178 444709.74442503956
 7 11336.977964928763 382183.34051422554
 8 22657.665326672173  527002.8538174401
 9 106190.10338763308 428359.52118789754
[9999990 more rows]

[dantegd]

Thanks @MurrayData ! There are 2 open PRs (#44 and #47) that are doing a lot of improvements for dbscan, so we're definitely working actively on solving this soon.

[cjnolet]

@MurrayData, what version of CUDA are you running?

[cjnolet]

@teju85 I wonder if this is being caused by the way Cutlass is using TilingStrategy and calculating the resources for the kernels. I was able to get past the error by modifying dbscan.cu to use the tall tiling strategy (503) but now the kernel appears to run indefinitely.

Here's the script I'm using to reproduce this:

import numpy as np
import pandas as pd
from sklearn.cluster import DBSCAN as skDBSCAN
from cuml import DBSCAN as cumlDBSCAN
import cudf
import os

import gzip
def load_data(nrows, ncols, cached = 'data/mortgage.npy.gz'):
    if os.path.exists(cached):
        print('use mortgage data')
        with gzip.open(cached) as f:
            X = np.load(f)
        X = X[np.random.randint(0,X.shape[0]-1,nrows),:ncols]
    else:
        print('use random data')
        X = np.random.rand(nrows,ncols)
    df = pd.DataFrame({'fea%d'%i:X[:,i] for i in range(X.shape[1])})
    return df

nrows = 10000000
ncols = 2

X = load_data(nrows,ncols)
print('data',X.shape)

eps = 3
min_samples = 2

X = cudf.DataFrame.from_pandas(X)

clustering_cuml = cumlDBSCAN(eps = eps, min_samples = min_samples)
clustering_cuml.fit(X)

[MurrayData]

@cjnolet I'm using 10.0 with the corresponding Docker container : rapidsai/rapidsai:cuda10.0_ubuntu16.04

Apologies for delay in replying, I was away for most of the weekend.

[teju85]

@cjnolet The tiling strategy only means that we are still using my fork of cutlass, right? I was under the impression that we were going to use the latest version of cutlass and deprecate my fork of cutlass used here. And this was the plan with PR's #44 and #47, right? Or did I get anything wrong here?

[dantegd]

@teju85 yes, that is the plan in the very near future (i.e. version 0.5), with #47 being merged pretty soon, and #44 should follow not much thereafter, though I believe there is still a scalability issue in it and Corey is working on plenty of stuff. @cjnolet feel free to correct me :)

@MurrayData we’ll keep you posted (though if I had to bet, I’d put money on you already saw there is a branch called branch-0.5) in case the refactored dbscan is merged to 0.5 before we squash the bug in the current version.

[dantegd]

@cjnolet do you know what is the status on this bug in the refactored dbscan of 0.5?

[cjnolet]

@teju85, have you been able to find a culprit for this issue yet?

I think we should push this to 0.6 and focus on #80.

[dantegd]

@cjnolet based on the bugs themselves I don’t see any reason why 80 would be higher priority since this is a full blown crash. We can revisit if by the end of the week there is no progress to a solution due to lack of cycles, but i don’t think we should before three.

[cjnolet]

80 is a problem because users are getting incorrect results- in some cases no clusters are found whatsoever when there should be several. This is happening at all scales.

This ticket is related to the algorithm's ability to scale to millions of training examples.

Ideally we would all have the cycles to knock both of these tickets out, however, if I have to choose between the two, I'd personally prioritize correctness over scale.

[dantegd]

@cjnolet @teju85 I was wondering if you know the status of this bug currently?

[teju85]

@dantegd Unfortunately, with so many other things happening, this has taken a backseat. May be sometime next week I'll try to see if I can make some progress on these dbscan-related bugs.