Julia: Random behavior when converting uninitialized Array{Bool} entries

[nalimilan]

This issue has been reported against DataArrays (https://github.com/JuliaStats/DataArrays.jl/issues/233). A short reproducer without using any package can be found in this gist: running the last line multiple times gives different results for several entries which should be defined.

Basically, it appears that leaving some entries in a BitArray undefined iterating over a BitArray can lead to an unexpected behaviour when using the BitArray entries in a condition to decide whether to initialize another array's entries or not. Initializing the array fixes the randomness, as does changing the na::BitArray{N} field to an Array{Bool,N}.

This happens on both 0.5 and master, but not on 0.4.7.

Cc: @carlobaldassi

[yuyichao]

similar never initialize the array.

[nalimilan]

Please read this more carefully. All array entries are manually initialized later except for the first one, and yet they are random. (Or I'm missing something obvious...).

[nalimilan]

That said, description wasn't completely accurate, I've tried to improve it.

[yuyichao]

AFAICT res is not initialized.

[nalimilan]

Well, the loop initializes all entries for which na is false, i.e. all entries except for the first one, right?

[yuyichao]

Though the issue does not seem to be related to the fill!, uncommenting it does not fix the issue. It seems to come from the & or !.

[nalimilan]

Funny. I've tried again, and uncommenting that line definitely fixes the randomness for me, even across 1M runs. Though it could still be another random result...

[carlobaldassi]

I tracked down the issue to the code which converts from Vector{Bool} to BitArray, where the random behaviour occurs. Specifically, in the conversion from Bool to UInt64, when the Bool in question is read from an uninizialized array. Here is a gist with a much simplified code which exposes the bug (see the example run at the bottom of the gist.)

Running it repeatedly seems to suggest the following cause: when the first element of the res array is read, since it was never initialized, random data is returned, in this line:

c |= (UInt64(C[ind]) << j) # the result is random here when ind==1

Changing UInt64(C[ind]) to C[ind]?UInt64(1):UInt64(0) seems to fix this issue, for some reason. However, note that even things like @show ind,C[1] or print("C[$ind]=",C[ind]) result in random behaviour, i.e. they often have different results from @show C[1] or @show C.
I guess this is a problem with the compiler being "too smart" (resulting in undefined behaviour).

I could submit the above fix as a bandaid, but it's probably better if some more knowledgeable people (@JeffBezanson, @vtjnash, ...) could fix the root cause.

[mbauman]

More simply:

f(A, i) = UInt(A[i])
A = Array{Bool}(5)

julia> f(A, 1)
0x0000000000000050

julia> UInt(A[1])
0x0000000000000000

[carlobaldassi]

Nice find @mbauman. I also found that the bug is not specific to UInts, almost any type will do: I tried Int8/16/32/64/128, UInt8/16/32/64/128, Float16/32/64 and Char, and the only type for which it's not triggered is Float16. See the updated gist.

[nalimilan]

Any news? For some reason, #21401 makes this failure visible (though it's not yet clear whether we want to merge that PR).

[simonster]

Perhaps this is an unintended consequence of https://github.com/JuliaLang/julia/pull/17225?

[nalimilan]

Can we do something about this? It completely breaks some use cases with isna on DataArrays (https://github.com/JuliaStats/DataArrays.jl/issues/233).

[nalimilan]

@carlobaldassi Could you submit the bandaid you described above as a PR? That would at least fix (one of) the most visible consequence of this bug for 0.6.