Flux.jl: Calling a gpu model with a cpu array crashes julia

[KristofferC]

julia> using Flux; using CuArrays                                                

julia> m = Chain(Dense(10, 10), softmax) |> gpu                                  
Chain(Dense(10, 10), NNlib.softmax)                                              

julia> m(rand(10))                                                               

# This is a crash to the shell without any error                                                   
C:\Users\Kristoffer\MachineLearning    
λ                                                                                

[MikeInnes]

At a guess, due to calling BLAS.gemm! or something with a GPU pointer and segfaulting. It's not immediately obvious what to do about that, however.

At one point we had pointer disabled so that kind of thing wouldn't happen, but I have no idea if GPUArrays added it back in or what.

[maleadt]

Probably also the cause of https://github.com/JuliaGPU/CuArrays.jl/issues/264?

At one point we had pointer disabled so that kind of thing wouldn't happen

We still do

julia> a = cu([1])
1-element CuArray{Float32,1}:
 1.0

julia> pointer(a)
ERROR: conversion to pointer not defined for CuArray{Float32,1}
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] unsafe_convert(::Type{Ptr{Float32}}, ::CuArray{Float32,1}) at ./pointer.jl:67
 [3] pointer(::CuArray{Float32,1}) at ./abstractarray.jl:880
 [4] top-level scope at none:0

[Keno]

That's no the lowering that ccall uses though:

julia> Base.unsafe_convert(Ptr{Float32}, Base.cconvert(Ptr{Float32}, CuArray{Float32}(undef, 2, 2)))
Ptr{Float32} @0x00007f1da7584800

That's the underlying cause of the crash. One of those two calls should probably be an error.

[maleadt]

On https://github.com/JuliaGPU/CUDAdrv.jl/pull/125 + https://github.com/JuliaGPU/CUDAnative.jl/pull/337 + https://github.com/JuliaGPU/CuArrays.jl/pull/271

julia> m(rand(10))
ERROR: cannot take the CPU address of a GPU array
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] cconvert(::Type{Ptr{Float32}}, ::CuArray{Float32,2}) at /home/tbesard/Julia/CuArrays/src/array.jl:129
 [3] gemv!(::Char, ::Float32, ::CuArray{Float32,2}, ::Array{Float32,1}, ::Float32, ::Array{Float32,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/blas.jl:577
 [4] gemv!(::Array{Float32,1}, ::Char, ::CuArray{Float32,2}, ::Array{Float32,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:360
 [5] mul! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:64 [inlined]
 [6] * at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:46 [inlined]
 [7] _forward at /home/tbesard/Julia/Flux/src/tracker/lib/array.jl:361 [inlined]
 [8] #track#1 at /home/tbesard/Julia/Flux/src/tracker/Tracker.jl:51 [inlined]
 [9] track at /home/tbesard/Julia/Flux/src/tracker/Tracker.jl:51 [inlined]
 [10] * at /home/tbesard/Julia/Flux/src/tracker/lib/array.jl:353 [inlined]
 [11] Dense at /home/tbesard/Julia/Flux/src/layers/basic.jl:82 [inlined]
 [12] Dense at /home/tbesard/Julia/Flux/src/layers/basic.jl:122 [inlined]
 [13] (::Dense{typeof(identity),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}})(::Array{Float64,1}) at /home/tbesard/Julia/Flux/src/layers/basic.jl:125
 [14] applychain(::Tuple{Dense{typeof(identity),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}},typeof(softmax)}, ::Array{Float64,1}) at /home/tbesard/Julia/Flux/src/layers/basic.jl:31
 [15] (::Chain{Tuple{Dense{typeof(identity),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}},typeof(softmax)}})(::Array{Float64,1}) at /home/tbesard/Julia/Flux/src/layers/basic.jl:33
 [16] top-level scope at none:0

[maleadt]

Another example:

using Flux
using CuArrays

struct IdentitySkip
   inner
end

(m::IdentitySkip)(x) = m.inner(x) .+ x

import Flux: @treelike
#@treelike IdentitySkip

m = Chain(
  IdentitySkip(Dense(100, 100, relu))
  ) |> gpu

x = rand(Float32, 100) |> gpu
m(x) # crash!

julia> m(x) # crash!
ERROR: ArgumentError: cannot take the CPU address of a CuArray{Float32,1}
Stacktrace:
 [1] cconvert(::Type{Ptr{Float32}}, ::CuArray{Float32,1}) at /home/tbesard/Julia/CuArrays/src/array.jl:131
 [2] gemv!(::Char, ::Float32, ::Array{Float32,2}, ::CuArray{Float32,1}, ::Float32, ::CuArray{Float32,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/blas.jl:577
 [3] gemv!(::CuArray{Float32,1}, ::Char, ::Array{Float32,2}, ::CuArray{Float32,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:360
 [4] mul! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:64 [inlined]
 [5] * at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/matmul.jl:46 [inlined]
 [6] _forward at /home/tbesard/Julia/Flux/src/tracker/lib/array.jl:361 [inlined]
 [7] #track#1 at /home/tbesard/Julia/Flux/src/tracker/Tracker.jl:51 [inlined]
 [8] track at /home/tbesard/Julia/Flux/src/tracker/Tracker.jl:51 [inlined]
 [9] * at /home/tbesard/Julia/Flux/src/tracker/lib/array.jl:353 [inlined]
 [10] Dense at /home/tbesard/Julia/Flux/src/layers/basic.jl:82 [inlined]
 [11] (::Dense{typeof(relu),TrackedArray{…,Array{Float32,2}},TrackedArray{…,Array{Float32,1}}})(::CuArray{Float32,1}) at /home/tbesard/Julia/Flux/src/layers/basic.jl:122
 [12] (::IdentitySkip)(::CuArray{Float32,1}) at ./REPL[4]:1
 [13] applychain at /home/tbesard/Julia/Flux/src/layers/basic.jl:31 [inlined]
 [14] (::Chain{Tuple{IdentitySkip}})(::CuArray{Float32,1}) at /home/tbesard/Julia/Flux/src/layers/basic.jl:33
 [15] top-level scope at none:0

From https://github.com/JuliaGPU/CuArrays.jl/issues/264.

[KristofferC]

Seems fixed

[amckay1]

I ran into the same problem using maleadt's example code with Flux v0.8.3, CuArrays v1.1.0, and Julia v1.2.0.

ArgumentError: cannot take the CPU address of a CuArray{Float32,1}

So far, tested on a Tesla K40m, a Tesla V100, and a Tesla P100, CUDA v10.0.130 and CuDNN v7.4.1.5. All of this is on two fo the university's clusters, so going to try to rule out some weird environmental setup, but I hesitate to think this is fully closed. Just tried on Flux#master and CuArrays#master, same error popped up. Maybe something with the Tesla architecture?

[maleadt]

No, the hard crash is fixed. You can always file an issue for the remaining problem (the fact that some part of Flux tries to convert a GPU array to a CPU pointer).

[amckay1]

Got it, thanks for the clarification! Sounds like it might be a better use of time for me to work around the issue on my side until Flux transitions to the new structure Mike described in his juliacon talk. Thanks again!