Cudf: [BUG] Incorrect mask being copied after `slice` followed by `copy_bitmask`

[galipremsagar]

Describe the bug
In cudf, after a libcudf slice operation is performed on a column and a python Coloumn object is constructed, the mask is obtained by performing a call to libcudf copy_bitmask here: https://github.com/rapidsai/cudf/blob/branch-0.15/cpp/src/bitmask/null_mask.cu#L636-#L646. This sequence of operations seems to be returning incorrect bitmask which is resulting in incorrect null values obtained while constructing a pyarrow array.

Steps/Code to reproduce bug

import cudf
sr = cudf.Series(['3'] * 34 + [None, None, None], dtype='str')
sliced = sr[19:]

print(sr)
Out[5]: 
0        3
1        3
2        3
3        3
4        3
5        3
6        3
7        3
8        3
9        3
10       3
11       3
12       3
13       3
14       3
15       3
16       3
17       3
18       3
19       3
20       3
21       3
22       3
23       3
24       3
25       3
26       3
27       3
28       3
29       3
30       3
31       3
32       3
33       3
34    None
35    None
36    None
dtype: object

print(sliced)
Out[6]: 
19       3
20       3
21       3
22       3
23       3
24       3
25       3
26       3
27       3
28       3
29       3
30       3
31       3
32    None
33    None
34    None
35    None
36    None
dtype: object

print(sliced._column.mask.to_host_array())
Out[7]: 
array([255,  31,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0],
      dtype=uint8)

print(sliced._column.base_mask.to_host_array())
Out[15]: 
array([255, 255, 255, 255,   3,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,
         0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0],
      dtype=uint8)

print(sliced._column.offset)
Out[16]: 19

Expected behavior
The expected behavior is to have only 3 null values instead of 6.

Environment overview (please complete the following information)

• Environment location: Bare-metal
• Method of cuDF install: from source(branch-0.15)

[galipremsagar]

Note: This happens with all column types, not specific to str column type. Above provided is just an example.

[jrhemstad]

Can you reproduce with a C++ test? There's a lot of Python machinery in the way that I'd like to rule out as an issue first.

[galipremsagar]

So while digging through I could not reproduce this in a C++ test. In a c++ for slice it seems to work fine:

TEST_F(SliceTableCornerCases, MiscOffset)
{
  cudf::test::fixed_width_column_wrapper<int32_t> col2{{3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3}
  , {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0}};
  cudf::test::fixed_width_column_wrapper<int32_t> col3{{3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3}, {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0}};
  std::vector<cudf::size_type> indices{19, 38};
  std::vector<cudf::column_view> result = cudf::slice(col2, indices);
  cudf::test::expect_columns_equal(col3, result[0]);
}

[kkraus14]

In a c++ for slice it seems to work fine

Sounds like we should double check the column_view being created by Python to make sure we're not doing something like passing both an offset and messing with a pointer or copy constructing a mask somewhere.

[rgsl888prabhu]

This test case reproduces the issue in gtest

TEST_F(SliceTableCornerCases, MiscOffset)
{
  cudf::test::fixed_width_column_wrapper<int32_t> col2{{3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3}
  , {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0}};
  cudf::test::fixed_width_column_wrapper<int32_t> col3{{3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3}, {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0}};
  std::vector<cudf::size_type> indices{19, 38};
  std::vector<cudf::column_view> result = cudf::slice(col2, indices);
  cudf::column result_column(result[0]);
  cudf::test::print(result_column);

  cudf::test::expect_columns_equal(col3, result_column);
}

I am just converting view to column to trigger copy_bitmask.

[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from SliceTableCornerCases
[ RUN      ] SliceTableCornerCases.MiscOffset
3,3,3,3,3,3,3,3,3,3,3,3,3,NULL,NULL,NULL,NULL,NULL,NULL
/home/rgsl888/Projects/backup/cudf/cpp/tests/utilities/column_utilities.cu:175: Failure
      Expected: differences.size()
      Which is: 3
To be equal to: size_t{0}
      Which is: 0
first difference: lhs[13] = 3, rhs[13] = NULL
[  FAILED  ] SliceTableCornerCases.MiscOffset (1 ms)
[----------] 1 test from SliceTableCornerCases (1 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (2 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] SliceTableCornerCases.MiscOffset

 1 FAILED TEST