Flux.jl: Float32 weights and biases causing SegFaults.

[ayush1999]

Probably the reason behind a few Metalhead models giving segfaults in Julia1.0 is that Conv crashes abruptly when the data type isn't Float64.

>>> m = Conv(rand(Float32, 3,3,3,64), rand(Float32, 64,), stride=(2,2), pad=(0,0), dilation=(1,1))
Conv((3, 3), 3=>64)
>>> m(rand(Float32, 224, 224, 3, 1))
signal (11): Segmentation fault
in expression starting at no file:0
in expression starting at no filein expression starting at no file:0
in expression starting at no filein expression starting at no file:0
Segmentation fault (core dumped)

[MikeInnes]

Does running with bounds checks give a clearer error?

[avik-pal]

Just checked it doesn't give any additional error message with --check-bounds=yes.

[avik-pal]

Directly using NNlib conv function gives julia: malloc.c:2852: mremap_chunk: Assertion '((size + offset) & (GLRO (dl_pagesize) - 1)) == 0' failed.

[ayush1999]

The error is probably coming from https://github.com/FluxML/NNlib.jl/blob/master/src/impl/conv.jl#L273 . Has the nature of @inbounds changed in these updates?

[ayush1999]

@avik-pal Directly using NNlib.conv gives me the same error as before.

julia> NNlib.conv(rand(Float32, 224,224,3,1), rand(Float32, 3,3,3,64), stride=(1,1), pad=(0,0), dilation=(1,1)) |> size

signal (11): Segmentation fault

signal (6): Aborted
in expression starting at no file:0
in expression starting at no filein expression starting at no file:0
in expression starting at no filein expression starting at no file:0
in expression starting at no file:0
Segmentation fault (core dumped)

Also, removing @inbounds from https://github.com/FluxML/NNlib.jl/blob/master/src/impl/conv.jl#L273 seems to work just fine.

[avik-pal]

@ayush1999 I got a simple segfault the first time I ran conv. I ran the same thing again and I got this error.

[MikeInnes]

It could be due to memory being freed during the ccall. You could try throwing a GC.@preserve in.

[vchuravy]

Looks like gemm!('N','N',M,N,K,alpha,pointer(x2),pointer(w),T(0),pointer(y,yidx)) is probably GC unsafe if pointer returns a Ptr since the GC is free to then delete x2. As Mike note GC@preserve should help or you can use Ref

[ayush1999]

I'm trying to use GC.@preserve wherever pointer is used, but the results are pretty ambiguous. Sometimes conv runs without any error, while on other occasions there is a segfault.

[MikeInnes]

I played around with this; NNlib.conv(rand(Float32,224,224,3,1), rand(Float32,3,3,3,64)) is enough to reproduce it. It's not deterministic but will usually fail on the second run. It's also odd that this doesn't fail for Float64 since all the code is the same.

I played around with making the gemm! a bit more consistent and adding some GC preserves here, but to no effect.

[MikeInnes]

Just checked with GC.enable(false) and it's not a GC issue after all. Something is probably wrong with the bounds we are giving to gemm!; did something change between 0.6 and 0.7?

The best fix is probably to remove NNlib's gemm! entirely and rely on Base's one.

[ayush1999]

Closing as fixed in the latest commit to NNlib (https://github.com/FluxML/NNlib.jl/pull/64)