Recommenders: Python chrono splitter takes too long

[miguelgfierro]

What is affected by this bug?

The python chrono splitter takes around 5h to split movielens 10M

In which platform does it happen?

• Azure Data Science Virtual Machine.

Expected behavior (i.e. solution)

Try to reduce that time

[WessZumino]

I noticed the same problem with the stratified_splitter. Maybe a solution is to perform the split using numpy in the middle, i.e. : pd.dataframe --> np.array --> split --> new pd.dataframes ?

[miguelgfierro]

yeah that make sense

[yueguoguo]

I will take a look.

[loomlike]

Quick looked the splitters code. The main bottleneck seems dataframe concat. If we do split for each 'user', the codes do split & concat len(users) times, and this is really bad if we have large number of users.

One simple way around I think is, instead of split for each user (or item, depending on filter_by), add a dummy indicator column and set index values of each split and filter by at the end to generate split dataframes. Adding split-index will be easy and fast since we already sort and group by users (or items).

Btw, seems we sort twice:

# Sort data by timestamp.
    data = data.sort_values(
        by=[split_by_column, col_timestamp], axis=0, ascending=False
    )

and

df_grouped = data.sort_values(col_timestamp).groupby(split_by_column)

[loomlike]

A very rough sketch of the algo will be something like:

# initialize split-index array and original data's index
# At the end, split will look like [0, 0, 1, 0, 0, 0, 1, ...] where 0 will be training, 1 is testing. e.g.
# also works for multiple split as we use enumerate(ratio) in the loop below.
split = np.zeros(len(data), dtype=np.int8)
key = np.zeros(len(data), dtype=np.int32)  # Keep original data's index to apply split index to data

# s is start, e is end of each split. We will assign split-index based on this.
s = 0
for name, group in df_grouped:
    for i, r in enumerate(ratio):
        l = int(len(group) * r)  # number of samples for this portion
        e = s+l
        np.put(split, range(s, e), i)  # assign split index
        np.put(key, range(s, e), group.index.values)  # remember original data index 
        s = e
...
# Assign split index to original data dataframe
data.loc[key, 'split'] = split

# Filter by split index to split data
...

[WessZumino]

If you guys want, I could merge the numpy splitter into the stratified python and adapt the script for the chrono splitter as well.