Julia: New type system: stackoverflow when defining method

[mauro3]

I'm pretty sure this only showed up after the #18457 merge (maybe add to list in #19998):

    _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.0-dev.2153 (2017-01-18 05:52 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit b8971ea4f* (0 days old master)
|__/                   |  x86_64-pc-linux-gnu

julia> immutable TT{X,Y} end

julia> f{X,Y}(::Type{TT{X,Y}},x::X,y::Y) = 1
f (generic function with 1 method)

julia> f{X}(::Type{TT{X,X}},x::X) = 100
ERROR: StackOverflowError:

julia> f(TT{Int,Int},1,2)
1

julia> f(TT{Int,Int},1) # it still works fine!
100

julia> methods(f)
# 2 methods for generic function "f":
f{X}(::Type{TT{X,X}}, x::X) in Main at REPL[3]:1
f{X,Y}(::Type{TT{X,Y}}, x::X, y::Y) in Main at REPL[2]:1

[ajkeller34]

I may have run into something similar to this, although this is a much more self-contained report than I was able to generate. Specifically, I encountered a StackOverflowError just when defining a new method for fma in Unitful, at this commit/line. I worked around it temporarily in subsequent commits.

edit: I could only see that it was a stack overflow if precompilation was off, otherwise precompilation would fail with little explanation.

[mauro3]

It took me some digging too... In my case, the work around was to wrap the function definition in a try-end block. This works as the method gets still defined.

[vtjnash]

It's actual a subtyping assertion failure, the StackOverflowError is just accidentally catching the memory corruption and thinking it can continue on a non-debug build.

julia> f{X}(::Type{TT{X,X}},x::X) = 100
Assertion failed: (i >= 0 && i < sizeof(st->stack) * 8), function statestack_get, file /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c, line 104.

signal (4): Illegal instruction: 4
while loading no file, in expression starting on line 0
abort at /Volumes/Lion/Users/jameson/Documents/julia/usr/lib/libjulia-debug.0.6.0.dylib (unknown line)
__assert_rtn at /Volumes/Lion/Users/jameson/Documents/julia/usr/lib/libjulia-debug.0.6.0.dylib (unknown line)
statestack_get at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:104
pick_union_element at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:282
subtype_union at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:298
subtype_ufirst at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:306
var_lt at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:350
subtype at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:666
subtype at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:677
subtype_union at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:299
subtype_ufirst at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:306
var_lt at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:350
subtype at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:666
subtype at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:677
subtype_union at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:299
subtype_ufirst at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:306
var_lt at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:350
subtype at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:666
subtype at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:677
subtype_union at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:299
subtype_ufirst at /Volumes/Lion/Users/jameson/Documents/julia/src/subtype.c:306

[JeffBezanson]

I came across this case that looks like it might have the same root cause:

typeintersect(Tuple{Int, Ref{Pair{K,V}}} where V where K,
              Tuple{Any, Ref{Pair{T,T}} where T })

[nalimilan]

I wonder whether I get the same bug with CategoricalArrays. Loading the package with Julia master gives a stack overflow unless I remove this convert definition.

[TotalVerb]

Might this be related?

julia> typeintersect(Type{NTuple{N,E}} where E where N, Type{NTuple{N,E} where N} where E)
ERROR: StackOverflowError:
Stacktrace:
 [1] typeintersect(::Any, ::Any) at ./reflection.jl:241

[SimonDanisch]

I think I got something similar... The range of errors I can produce with this example is quite interesting:

# Build types
ints = (Int, Int32, UInt, UInt32)
typealias Ints Union{ints...}
vecs = []
for i = 2:4, t in ints
    push!(vecs, NTuple{i, t})
end
typealias Vecs Union{vecs...}

# Lets produce some errors:

# this one just froze one time, succeeded other times:
T = Type{Tuple{V, I}} where V <: Vecs where I <: Ints
# this one gave me a ReadOnlyMemoryError()
typeintersect(T, T)

test{V <: Vecs, I <: Ints}(a::Type{Tuple{V, I}}) = I
# Freezes or gives a stackoverflow:
test{V <: Vecs, I <: Ints}(a::Type{Tuple{V, I}}) = I

Update:
Just tried fresh master, first run of the example code only errored for the method redefinition with this stacktrace:

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x64f6e046 -- subtype_unionall at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:461
while loading no file, in expression starting on line 0
subtype_unionall at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:461
subtype at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:703
subtype_unionall at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:465
subtype at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:700
subtype_unionall at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:465
subtype at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:700
exists_subtype at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:808 [inlined]
forall_exists_subtype at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:836
jl_subtype_env at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:887 [inlined]
jl_subtype at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:904 [inlined]
jl_types_equal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\subtype.c:911
jl_typemap_lookup_by_type_ at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\typemap.c:653 [inlined]
jl_typemap_assoc_by_type at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\typemap.c:730
jl_typemap_insert at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\typemap.c:988
jl_method_table_insert at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:1348
jl_method_def at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\method.c:684
eval at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\interpreter.c:276
eval_body at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\interpreter.c:600
jl_interpret_toplevel_thunk at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\interpreter.c:664
jl_toplevel_eval_flex at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\toplevel.c:592 [inlined]
jl_toplevel_eval at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\toplevel.c:600
jl_toplevel_eval_in at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\builtins.c:614
eval at .\boot.jl:236
unknown function (ip: 0000000061C6F085)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:248 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:2217
eval_user_input at .\REPL.jl:66
unknown function (ip: 00000000162C613A)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:248 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:2217
macro expansion at .\REPL.jl:97 [inlined]
#1 at .\event.jl:73
unknown function (ip: 00000000162C3833)
jl_call_method_internal at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia_internal.h:248 [inlined]
jl_apply_generic at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\gf.c:2217
jl_apply at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\julia.h:1411 [inlined]
start_task at /home/Administrator/buildbot/slave/package_win6_2-x64/build/src/home/Administrator/buildbot/slave/package_win6_2-x64/build/src\task.c:261
Allocations: 3656213 (Pool: 3654837; Big: 1376); GC: 4

Second run after restarting Julia typeintersect produces the ReadOnlyMemory. So it seems unstable, which means I might eventually also reproduce this freeze when constructing T.

[SimonDanisch]

bump

[SimonDanisch]

Seems like my example is the only one still segfaulting. Should I open a new issue and close this?

[JeffBezanson]

Has the same assertion failure as #21191.

[martinholters]

Looks like this is mostly fixed now (0.6.0-pre.beta.91), the only example still crashing is https://github.com/JuliaLang/julia/issues/20103#issuecomment-276257007, which looks closely related to #21191 indeed (but I only checked the code pieces given here, not the links to packages). And

julia> typeintersect(Type{NTuple{N,E}} where E where N, Type{NTuple{N,E} where N} where E)
Type{Tuple{}} where E

doesn't crash, but I think it should give Union{}?

I'll submit tests for these, and then I think we can close this.

[SimonDanisch]

My example still segfaults on 8934aab :(

[JeffBezanson]

That seems to be the same as #21191. Fixed (at least at band-aid level) by 8208f86545e07c9581ac21082c8ecfbfb872b8d3.

[SimonDanisch]

confirmed, 8208f86 fixes it :)
Thanks!!