Flux.jl: Flux fails its own tests when run with CuArrays
The Flux version is v0.8.3 and CuArrays is v1.2.1.
Here is the output from ] test Flux
[ Info: Testing Basics
[ Info: Testing Layers
0.044721193126701045
[ Info: Running Gradient Checks
[ Info: Testing GPU Support
[ Info: Testing Flux/CUDNN
┌ Warning: `cuzeros` is deprecated, use `CuArrays.zeros` instead.
│ caller = #call#2(::Int64, ::Type{Flux.CUDA.RNNDesc{Float32}}, ::Int64, ::Int64, ::Int64) at curnn.jl:67
â”” @ Flux.CUDA ~/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:67
┌ Warning: `cuones` is deprecated, use `CuArrays.ones` instead.
│ caller = hBatch(::CuArray{Float32,2}, ::CuArray{Float32,1}) at curnn.jl:134
â”” @ Flux.CUDA ~/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:134
┌ Warning: `cuones` is deprecated, use `CuArrays.ones` instead.
│ caller = hBatch(::CuArray{Float32,2}, ::CuArray{Float32,2}) at curnn.jl:135
â”” @ Flux.CUDA ~/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:135
┌ Warning: `cuones` is deprecated, use `CuArrays.ones` instead.
│ caller = hBatch(::TrackedArray{…,CuArray{Float32,2}}, ::CuArray{Float32,2}) at curnn.jl:135
â”” @ Flux.CUDA ~/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:135
┌ Warning: `cuones` is deprecated, use `CuArrays.ones` instead.
│ caller = hBatch(::TrackedArray{…,CuArray{Float32,2}}, ::CuArray{Float32,1}) at curnn.jl:134
â”” @ Flux.CUDA ~/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:134
batch_size = 1: Error During Test at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:7
Got exception outside of a @test
CUDNNError(code 3, CUDNN_STATUS_BAD_PARAM)
Stacktrace:
[1] macro expansion at /home/azamat/.julia/packages/CuArrays/wXQp8/src/dnn/error.jl:19 [inlined]
[2] cudnnRNNBackwardData(::Flux.CUDA.RNNDesc{Float32}, ::Int64, ::Array{CuArrays.CUDNN.TensorDesc,1}, ::CuArray{Float32,1}, ::Array{CuArrays.CUDNN.TensorDesc,1}, ::CuArray{Float32,1}, ::CuArrays.CUDNN.TensorDesc, ::CuArray{Float32,1}, ::Ptr{Nothing}, ::CUDAdrv.CuPtr{Nothing}, ::CuArrays.CUDNN.FilterDesc, ::CuArray{Float32,1}, ::CuArrays.CUDNN.TensorDesc, ::CuArray{Float32,1}, ::Ptr{Nothing}, ::CUDAdrv.CuPtr{Nothing}, ::Array{CuArrays.CUDNN.TensorDesc,1}, ::CuArray{Float32,1}, ::CuArrays.CUDNN.TensorDesc, ::CuArray{Float32,1}, ::Ptr{Nothing}, ::CUDAdrv.CuPtr{Nothing}, ::CuArray{UInt8,1}, ::CuArray{UInt8,1}) at /home/azamat/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:171
[3] backwardData(::Flux.CUDA.RNNDesc{Float32}, ::CuArray{Float32,1}, ::CuArray{Float32,1}, ::CuArray{Float32,1}, ::Nothing, ::CuArray{Float32,1}, ::Nothing, ::CuArray{UInt8,1}) at /home/azamat/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:188
[4] backwardData(::Flux.CUDA.RNNDesc{Float32}, ::CuArray{Float32,1}, ::CuArray{Float32,1}, ::CuArray{Float32,1}, ::CuArray{Float32,1}, ::CuArray{UInt8,1}) at /home/azamat/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:196
[5] (::getfield(Flux.CUDA, Symbol("##8#9")){Flux.GRUCell{TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}},TrackedArray{…,CuArray{Float32,1}},TrackedArray{…,CuArray{Float32,1}},CuArray{UInt8,1},Tuple{CuArray{Float32,1},CuArray{Float32,1}}})(::Tuple{CuArray{Float32,1},CuArray{Float32,1}}) at /home/azamat/.julia/packages/Flux/qXNjB/src/cuda/curnn.jl:307
[6] back_(::Tracker.Call{getfield(Flux.CUDA, Symbol("##8#9")){Flux.GRUCell{TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}},TrackedArray{…,CuArray{Float32,1}},TrackedArray{…,CuArray{Float32,1}},CuArray{UInt8,1},Tuple{CuArray{Float32,1},CuArray{Float32,1}}},Tuple{Tracker.Tracked{CuArray{Float32,1}},Tracker.Tracked{CuArray{Float32,1}},Tracker.Tracked{CuArray{Float32,2}},Tracker.Tracked{CuArray{Float32,2}},Tracker.Tracked{CuArray{Float32,1}}}}, ::Tuple{CuArray{Float32,1},CuArray{Float32,1}}, ::Bool) at /home/azamat/.julia/packages/Tracker/SAr25/src/back.jl:35
[7] back(::Tracker.Tracked{Tuple{CuArray{Float32,1},CuArray{Float32,1}}}, ::Tuple{CuArray{Float32,1},Int64}, ::Bool) at /home/azamat/.julia/packages/Tracker/SAr25/src/back.jl:58
[8] (::getfield(Tracker, Symbol("##13#14")){Bool})(::Tracker.Tracked{Tuple{CuArray{Float32,1},CuArray{Float32,1}}}, ::Tuple{CuArray{Float32,1},Int64}) at /home/azamat/.julia/packages/Tracker/SAr25/src/back.jl:38
[9] foreach(::Function, ::Tuple{Tracker.Tracked{Tuple{CuArray{Float32,1},CuArray{Float32,1}}},Nothing}, ::Tuple{Tuple{CuArray{Float32,1},Int64},Nothing}) at ./abstractarray.jl:1921
[10] back_(::Tracker.Call{getfield(Tracker, Symbol("##361#363")){Tracker.TrackedTuple{Tuple{CuArray{Float32,1},CuArray{Float32,1}}},Int64},Tuple{Tracker.Tracked{Tuple{CuArray{Float32,1},CuArray{Float32,1}}},Nothing}}, ::CuArray{Float32,1}, ::Bool) at /home/azamat/.julia/packages/Tracker/SAr25/src/back.jl:38
[11] back(::Tracker.Tracked{CuArray{Float32,1}}, ::CuArray{Float32,1}, ::Bool) at /home/azamat/.julia/packages/Tracker/SAr25/src/back.jl:58
[12] back!(::TrackedArray{…,CuArray{Float32,1}}, ::CuArray{Float32,1}) at /home/azamat/.julia/packages/Tracker/SAr25/src/back.jl:77
[13] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:23
[14] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[15] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:7
[16] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[17] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
[18] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
[19] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
[20] include at ./boot.jl:328 [inlined]
[21] include_relative(::Module, ::String) at ./loading.jl:1094
[22] include(::Module, ::String) at ./Base.jl:31
[23] include(::String) at ./client.jl:431
[24] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/cuda.jl:45
[25] include at ./boot.jl:328 [inlined]
[26] include_relative(::Module, ::String) at ./loading.jl:1094
[27] include(::Module, ::String) at ./Base.jl:31
[28] include(::String) at ./client.jl:431
[29] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/runtests.jl:30
[30] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
[31] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/runtests.jl:11
[32] include at ./boot.jl:328 [inlined]
[33] include_relative(::Module, ::String) at ./loading.jl:1094
[34] include(::Module, ::String) at ./Base.jl:31
[35] include(::String) at ./client.jl:431
[36] top-level scope at none:5
[37] eval(::Module, ::Any) at ./boot.jl:330
[38] exec_options(::Base.JLOptions) at ./client.jl:271
[39] _start() at ./client.jl:464
batch_size = 5: Test Failed at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:26
Expression: rnn.cell.Wi.grad ≈ collect(curnn.cell.Wi.grad)
Evaluated: Float32[0.026422147 0.031162314 … 0.04015291 0.048164792; -0.00580488 -0.006842452 … -0.0006331135 -0.0051276702; … ; -0.48899454 -0.3973075 … -0.5430576 -0.59323865; -1.5976363 -1.7279379 … -1.2472157 -2.0624979] ≈ Float32[0.0163969 0.020081667 … 0.035748813 0.03631903; -0.0038312192 -0.004661015 … 0.00023391607 -0.002795606; … ; -0.6556128 -0.5814663 … -0.616253 -0.79011357; -1.3221657 -1.4234674 … -1.1262015 -1.7370036]
Stacktrace:
[1] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:26
[2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[3] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:7
[4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[5] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
[6] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
[7] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
batch_size = 5: Test Failed at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:27
Expression: rnn.cell.Wh.grad ≈ collect(curnn.cell.Wh.grad)
Evaluated: Float32[-0.015565421 0.0032692542 … -0.02177089 -0.0079453755; 0.0033148595 -0.00096146535 … 0.002088631 -0.00068250747; … ; 0.060095496 -0.037989482 … 0.0877803 0.05621599; 0.13518983 -0.049921982 … 0.1269001 0.046479825] ≈ Float32[-0.0102084465 0.0006350695 … -0.017785752 -0.0077674175; 0.0022602363 -0.00044287564 … 0.0013040807 -0.00071754144; … ; 0.08005668 -0.047804993 … 0.10262971 0.056879077; 0.12019784 -0.042549968 … 0.11574733 0.0459818]
Stacktrace:
[1] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:27
[2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[3] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:7
[4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[5] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
[6] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
[7] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
batch_size = 5: Test Failed at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:28
Expression: rnn.cell.b.grad ≈ collect(curnn.cell.b.grad)
Evaluated: Float32[0.043129362, -0.0052406434, 0.0032806913, 0.03839735, -0.027703835, -0.1904509, 0.3789641, 0.11353702, -1.1046532, -0.10988483, -0.3808116, -0.806893, -0.59165734, -0.8405457, -2.1673152] ≈ Float32[0.030957595, -0.002844398, 0.007849413, 0.050801173, -0.025155623, -0.11647805, 0.19690844, 0.15651213, -1.3034244, -0.10899977, -0.033872187, -0.53707606, -0.8408148, -1.0428388, -1.8328632]
Stacktrace:
[1] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:28
[2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[3] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:7
[4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[5] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
[6] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
[7] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
batch_size = 5: Test Failed at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:29
Expression: rnn.cell.h.grad ≈ collect(curnn.cell.h.grad)
Evaluated: Float32[-0.012821687, -0.7067668, -0.35861096, -1.4159613, -0.54712236] ≈ Float32[0.066714235, -0.12755066, -0.45448342, -1.797685, -0.41404387]
Stacktrace:
[1] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:29
[2] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[3] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:7
[4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1186
[5] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
[6] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.2/Test/src/Test.jl:1113
[7] top-level scope at /home/azamat/.julia/packages/Flux/qXNjB/test/cuda/curnn.jl:4
Test Summary: | Pass Fail Error Total
Flux | 254 4 1 259
Throttle | 11 11
Jacobian | 1 1
Initialization | 12 12
Params | 2 2
Basic Stacking | 1 1
Precision | 6 6
Stacking | 3 3
onecold | 4 4
Optimise | 11 11
Optimiser | 3 3
Training Loop | 2 2
basic | 25 25
Dropout | 8 8
BatchNorm | 14 14
InstanceNorm | 16 16
GroupNorm | 16 16
losses | 30 30
Pooling | 2 2
CNN | 1 1
asymmetric padding | 7 7
Depthwise Conv | 4 4
ConvTranspose | 1 1
Tracker | 4 4
CuArrays | 8 8
CUDNN BatchNorm | 10 10
RNN | 40 4 1 45
R = RNN | 16 16
R = GRU | 6 4 1 11
batch_size = 1 | 2 1 3
batch_size = 5 | 4 4 8
R = LSTM | 18 18
ERROR: LoadError: Some tests did not pass: 254 passed, 4 failed, 1 errored, 0 broken.
in expression starting at /home/azamat/.julia/packages/Flux/qXNjB/test/runtests.jl:9
ERROR: Package Flux errored during testing
AzamatB
All 11 comments
These don't fail for me in the master branch. Try ] add Flux#master if you don't want to deal with them.
janEbert
on 28 Aug 2019
This is a known issue; they unfortunately fail intermittently so it's not reproducible and difficult to track down. The plan is to get the core API wrappers into CuArrays and develop them there.
MikeInnes
on 28 Aug 2019
These tests always fail for me. Both on my personal PC with Nvidia 1080ti, and enterprise grade Nvidia servers.
freddycct
on 28 Aug 2019
267
tanhevg
on 5 Sep 2019
From slack, I and @tanhevg are suspecting GC is the reason behind of the error. However, I couldn't reproduce the error when running cudnn test alone (I have no idea how to use Debugger.jl within test). I could only reproduce the error with whole tests by ] test.
For who digs this issue, here are what we found
- I found a similar issue from torch
- Someone can use
CUARRAYS_MEMORY_LIMITto limit GPU memory usage - @maleadt showed us where breakpoint should be. (https://github.com/JuliaGPU/CuArrays.jl/blob/master/src/dnn/error.jl)
diff --git a/src/dnn/error.jl b/src/dnn/error.jl
index c8c8d00..35ee676 100644
--- a/src/dnn/error.jl
+++ b/src/dnn/error.jl
@@ -15,7 +15,9 @@ macro check(dnn_func)
quote
local err::cudnnStatus_t
err = $(esc(dnn_func))
- if err != CUDNN_STATUS_SUCCESS
+ if err == CUDNN_STATUS_BAD_PARAM
+ @breakpoint_here
+ elseif err != CUDNN_STATUS_SUCCESS
throw(CUDNNError(err))
end
err
appleparan
on 10 Sep 2019
Pointing at the GC is a good idea; the randomness of it suggests something like GC, and I just realised we're not doing any of the GC.@preserves that we should be using to prevent GC from collecting arrays (and possibly also things like tensor descriptors). I would normally expect a segfault here but I guess use-after-free is different with GPU memory.
Unfortunately, when I tried disabling the GC around the ccalls in this patch it made no difference. I'm not certain that this rules out GC entirely, but I don't know how else it could be causing something like this.
MikeInnes
on 10 Sep 2019
I'm not certain that this rules out GC entirely, but I don't know how else it could be causing something like this.
There's manual calls to GC.gc() in the allocator, and I'm not sure how those play together with GC.enable().
maleadt
on 10 Sep 2019
I can reproduce the error again by using BenchmarkTools.jl
Setting gcsample and gctrial as true then this error happens again. It seems definitely GC problem.
Check BenchmarkTools docs
I just copied test/cuda/curnn.jl to test/cuda/curnn_debug.jl, removed @testset, wrap test as function run_test(), then use BenchmarkTools as following
julia> include("test/cuda/curnn_debug.jl")
run_test (generic function with 1 method)
julia> b = @benchmarkable run_test() gctrial=true gcsample=true
Benchmark(evals=1, seconds=5.0, samples=10000)
julia> @enter run(b)
In #run#39(kwargs, , b, p) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:46
>46 Base.run(b::Benchmark, p::Parameters = b.params; kwargs...) = run_result(b, p; kwargs...)[1]
About to run: (NamedTuple)()
1|debug> c
ERROR: CUDNNError(code 8, CUDNN_STATUS_EXECUTION_FAILED)
Stacktrace:
[1] #DropoutDesc#12(::Int64, ::DataType, ::Int64) at /home/appleparan/.julia/packages/CuArrays/vmYgE/src/dnn/error.jl:19
[2] (::DataType)(::Int64) at /home/appleparan/src/Flux.jl/src/cuda/cudnn.jl:19
[3] #call#2(::Int64, ::DataType, ::Int64, ::Int64, ::Int64) at /home/appleparan/src/Flux.jl/src/cuda/curnn.jl:63
[4] (::DataType)(::Int64, ::Int64, ::Int64) at /home/appleparan/src/Flux.jl/src/cuda/curnn.jl:60
[5] (::UnionAll)(::Flux.RNNCell{typeof(tanh),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}) at /home/appleparan/src/Flux.jl/src/cuda/curnn.jl:262
[6] desc(::Flux.RNNCell{typeof(tanh),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}) at /home/appleparan/src/Flux.jl/src/cuda/curnn.jl:269
[7] _forward(::Flux.RNNCell{typeof(tanh),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}, ::TrackedArray{…,CuArray{Float32,1}}, ::TrackedArray{…,CuArray{Float32,1}}, ::TrackedArray{…,CuArray{Float32,2}}, ::TrackedArray{…,CuArray{Float32,2}}, ::TrackedArray{…,CuArray{Float32,1}}) at /home/appleparan/src/Flux.jl/src/cuda/curnn.jl:305
[8] #track#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Tracker.track), ::Flux.RNNCell{typeof(tanh),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}, ::Tuple{TrackedArray{…,CuArray{Float32,1}},TrackedArray{…,CuArray{Float32,1}},TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}) at /home/appleparan/.julia/packages/Tracker/SAr25/src/Tracker.jl:51
[9] track(::Flux.RNNCell{typeof(tanh),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}, ::Tuple{TrackedArray{…,CuArray{Float32,1}},TrackedArray{…,CuArray{Float32,1}},TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}) at /home/appleparan/.julia/packages/Tracker/SAr25/src/Tracker.jl:51
[10] (::Flux.RNNCell{typeof(tanh),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}})(::TrackedArray{…,CuArray{Float32,1}}, ::TrackedArray{…,CuArray{Float32,1}}) at /home/appleparan/src/Flux.jl/src/cuda/curnn.jl:280
[11] (::Flux.Recur{Flux.RNNCell{typeof(tanh),TrackedArray{…,CuArray{Float32,2}},TrackedArray{…,CuArray{Float32,1}}}})(::Tuple{TrackedArray{…,CuArray{Float32,1}}}) at /home/appleparan/src/Flux.jl/src/layers/recurrent.jl:36
[12] run_test() at /home/appleparan/src/Flux.jl/test/cuda/curnn_debug.jl:17
[13] ##core#767() at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:297
[14] ##sample#768(::BenchmarkTools.Parameters) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:303
[15] #_run#7(::Bool, ::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(BenchmarkTools._run), ::BenchmarkTools.Benchmark{Symbol("##benchmark#766")}, ::BenchmarkTools.Parameters) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:331
[16] _run(::BenchmarkTools.Benchmark{Symbol("##benchmark#766")}, ::BenchmarkTools.Parameters) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:325
[17] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Base.invokelatest), ::typeof(BenchmarkTools._run), ::Tuple{BenchmarkTools.Benchmark{Symbol("##benchmark#766")},BenchmarkTools.Parameters}) at essentials.jl:742
[18] invokelatest(::typeof(BenchmarkTools._run), ::Tuple{BenchmarkTools.Benchmark{Symbol("##benchmark#766")},BenchmarkTools.Parameters}) at essentials.jl:741
[19] #run_result#37(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(BenchmarkTools.run_result), ::BenchmarkTools.Benchmark{Symbol("##benchmark#766")}, ::BenchmarkTools.Parameters) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:32
[20] run_result(::BenchmarkTools.Benchmark{Symbol("##benchmark#766")}, ::BenchmarkTools.Parameters) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:32
[21] #run#39(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(run), ::BenchmarkTools.Benchmark{Symbol("##benchmark#766")}, ::BenchmarkTools.Parameters) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:46
[22] run(::BenchmarkTools.Benchmark{Symbol("##benchmark#766")}, ::BenchmarkTools.Parameters) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:46
[23] run(::BenchmarkTools.Benchmark{Symbol("##benchmark#766")}) at /home/appleparan/.julia/packages/BenchmarkTools/7aqwe/src/execution.jl:46
Below is my info
(Flux) pkg> st
Project Flux v0.9.0
Status `~/src/Flux.jl/Project.toml`
[1520ce14] AbstractTrees v0.2.1
[79e6a3ab] Adapt v1.0.0
[6e4b80f9] + BenchmarkTools v0.4.3
[3895d2a7] CUDAapi v1.1.0
[944b1d66] CodecZlib v0.6.0
[5ae59095] Colors v0.9.6
[3a865a2d] ↑ CuArrays v1.2.1 ⇒ v1.3.0 #master (https://github.com/JuliaGPU/CuArrays.jl.git)
[31a5f54b] + Debugger v0.6.1
[5903a43b] + Infiltrator v0.1.0
[e5e0dc1b] Juno v0.7.2
[1914dd2f] MacroTools v0.5.1
[872c559c] NNlib v0.6.0
[189a3867] Reexport v0.2.0
[2913bbd2] StatsBase v0.32.0
[9f7883ad] Tracker v0.2.3
[a5390f91] ZipFile v0.8.3
[8bb1440f] DelimitedFiles
[37e2e46d] LinearAlgebra
[44cfe95a] Pkg
[de0858da] Printf
[9a3f8284] Random
[ea8e919c] SHA
[10745b16] Statistics
Status `~/src/Flux.jl/Manifest.toml`
[6e4b80f9] + BenchmarkTools v0.4.3
[3a865a2d] ↑ CuArrays v1.2.1 ⇒ v1.3.0 #master (https://github.com/JuliaGPU/CuArrays.jl.git)
[31a5f54b] + Debugger v0.6.1
[5903a43b] + Infiltrator v0.1.0
julia> versioninfo()
Julia Version 1.1.1
Commit 55e36cc308 (2019-05-16 04:10 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
JULIA_PATH = /usr/local/julia
JULIA_BINDIR = /usr/local/julia/bin
There's manual calls to
GC.gc()in the allocator, and I'm not sure how those play together withGC.enable().
AFAICT there shouldn't be any GPU memory allocations in the ccalls that we're wrapping nogc around. Probably worth some logging to check that though.
MikeInnes
on 11 Sep 2019
This is fixed now.
maleadt
on 28 Sep 2019
🎉7
❤4
Might be worth a quick summary of why this happened, in case it might help the next poor soul who has to write CUDNN wrappers:
CUDNN RNNs require a workspace parameter which (presumably) provides a scratch space for internal calculations. We re-use the workspace between RNN calls, but during the forward pass we call CUDNN to make sure the shared workspace is at least as large as CUDNN requires:
We didn't bother to do this again on the backwards pass, instead referencing the workspace directly. Since we already called the getworkspace with the exact same RNN, parameters, input size and sequence length, so calling it again would clearly be redundant.
Except, wait, it isn't actually redundant. Seemingly at random, CUDNN will actually ask for _more_ workspace in this situation, and throw up when it doesn't get what it wants. Calling getworkspace here immediately fixes this issue.
Why didn't we spot this earlier? Everything about this issue suggested a memory-related problem, perhaps an early free related to the CuArrays pooling and GC. Both because it's non-deterministic, but also because the bug _never presented itself with GC disabled_. This didn't so much lead us down the garden path as drag us through the gate and into the back of a van.
Our conclusion is that CUDNN's RNNs must have some built-in time/space tradeoffs. If memory pressure on the GPU is low, it will ask for more workspace and avoid some extra redundancy. Our test setup increased the likelihood of GC running in between invocations of the same RNN, causing this problem. Interesting to consider how this plays out with systems that allocate a bunch of memory up front and do their own pooling, as most ML frameworks do.
Anyway, if anyone at NVIDIA reads this: please, by god, add some error strings to your library.
MikeInnes
on 3 Oct 2019
👀11
❤5
Was this page helpful?
0 / 5 - 0 ratings
Related issues
MikeInnes
·
4Comments
Sheemon7
·
3Comments
ageron
·
4Comments
MikeInnes
·
6Comments
ExpandingMan
·
6Comments
Most helpful comment
This is fixed now.