Julia: fatal error in type inference (type bound)

[nalimilan]

I get this on 0.7 and 1.0 and master when running the following line in a custom CSV.jl branch (commit https://github.com/JuliaData/CSV.jl/pull/317/commits/b1933f120da28c4cccc557babb04ed011a18e03d). A simple change (https://github.com/JuliaData/CSV.jl/pull/317/commits/a4c15114e9ebc3e4e07a071cb665dd5d60a4dcad) is enough to work around the problem.

julia> CSV.read("test/testfiles/baseball.csv", categorical=true)
ERROR: fatal error in type inference (type bound)
Stacktrace:
 [1] detect(::Array{Type,1}, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Array{Int64,1}, ::Parsers.Delimited{false,Parsers.Quoted{Parsers.Strip{Parsers.Sentinel{typeof(Parsers.defaultparser),Parsers.Trie{0x00,false,missing,2,Tuple{}}}}},Parsers.Trie{0x00,false,missing,8,Tuple{Parsers.Trie{0x2c,true,missing,8,Tuple{}},Parsers.Trie{0x0a,true,missing,8,Tuple{}},Parsers.Trie{0x0d,true,missing,8,Tuple{Parsers.Trie{0x0a,true,missing,8,Tuple{}}}}}}}, ::NamedTuple{(),Tuple{}}, ::Dict{Type,Type}, ::Bool, ::Bool, ::Base.RefValue{Int64}, ::Bool) at /home/milan/.julia/dev/CSV/src/typedetection.jl:88
 [2] #File#1(::Int64, ::Bool, ::Int64, ::Nothing, ::Int64, ::Nothing, ::Bool, ::Nothing, ::Bool, ::Array{String,1}, ::String, ::String, ::Bool,::Char, ::Nothing, ::Nothing, ::Char, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Dict{Type,Type}, ::Symbol, ::Bool, ::Bool, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Type, ::String) at /home/milan/.julia/dev/CSV/src/CSV.jl:181
 [3] Type at ./none:0 [inlined]
 [4] #read#101(::Bool, ::Dict{Int64,Function}, ::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:categorical,),Tuple{Bool}}}, ::Function, ::String, ::Type) at /home/milan/.julia/dev/CSV/src/CSV.jl:304
 [5] #read at ./none:0 [inlined] (repeats 2 times)
 [6] top-level scope at none:0

Cf. https://github.com/JuliaData/CSV.jl/pull/317.

[SebastianM-C]

I also noticed this with DataFrames v0.14.1 and Query v0.10.0. MWE:

a = DataFrame(:a=>[1.,2,3],:b=>[1,1,2],:c=>["a","b","c"])

categorical!(a,:a)
categorical!(a,:b)

a |> @filter(_.b == 1) |> DataFrame

[Keno]

There's an inference bug of some sort here:

  │          (c = (Core.getfield)(%30, 1))::Union{Array{String,1}, CategoricalArray}                                                                              │
  │    %32 = (Core.getfield)(%30, 2)::Tuple{Int64,Tuple{Symbol,Symbol,Symbol}}                                                                                    │
  │    %33 = (DataFrames.getvector)(c)::Union{CategoricalArray{Float64,1,UInt32,Float64,CategoricalValue{Float64,UInt32},Union{}}, Array{String,1}}               │

The inference on getvector is invalid. It's essentially just the identity on these types.

[Keno]

Further reduction:

julia> Base._methods_by_ftype(Tuple{typeof(DataFrames.getvector),CategoricalArrays.CategoricalArray}, 4, typemax(UInt), min_valid, max_valid)
1-element Array{Any,1}:
 svec(Tuple{typeof(getvector),CategoricalArray}, svec(), getvector(x) in DataFrames at /home/keno/.julia/packages/DataFrames/1PqZ3/src/other/tables.jl:12)

This is an invalid method match, because there's another applicable method right before it

getvector(x::AbstractVector) = x
getvector(x) = collect(x)

Reducing further, we see:

julia> typeintersect(CategoricalArrays.CategoricalArray, AbstractVector)
Union{}

when the correct answer is:

CategoricalArray{<:Any, 1, <: Integer, <:Any, <:Any, <:Any} 

as shown by

julia> CategoricalArray{<:Any, 1, <: Integer, <:Any, <:Any, <:Any} <: AbstractVector
true

julia> CategoricalArray{<:Any, 1, <: Integer, <:Any, <:Any, <:Any} <: CategoricalArrays.CategoricalArray
true

Minimal reproducer:

struct Concrete{N, C, U} <: AbstractArray{Union{C, U}, N}
end
typeintersect(Concrete, AbstractVector)

[nalimilan]

Fixed by #29406:

julia> struct Concrete{N, C, U} <: AbstractArray{Union{C, U}, N}
       end

julia> typeintersect(Concrete, AbstractVector)
Concrete{1,C,U} where U where C

[nalimilan]

Though the commit referenced above doesn't seem to be included in #29406. @JeffBezanson is this still relevant?

[JeffBezanson]

It looks like all but one of the test cases on jb/fix29501 are also now fixed on master (by #29406). The reduction here is among the cases fixed. I'd try checking the case in the original report, and this can be closed if it's fixed.

[nalimilan]

OK, I confirm the original bug is fixed.

[ablaom]

I'm still getting an error on a variation of @SebastianM-C example:

````
julia> a = DataFrame(:a=>[1.,2,3],:b=>[1,1,2],:c=>["a","b","c"])
julia> categorical!(a,:a)
julia> categorical!(a,:b)
julia> using Tables
julia> t = Tables.columntable(a)
(a = CategoricalValue{Float64,UInt32}[1.0, 2.0, 3.0], b = CategoricalValue{Int64,UInt32}[1, 1, 2], c = ["a", "b", "c"])
julia> DataFrame(t)
ERROR: fatal error in type inference (type bound)
Stacktrace:
[1] fromcolumns(::NamedTuple{(:a, :b, :c),Tuple{CategoricalArray{Float64,1,UInt32,Float64,CategoricalValue{Float64,UInt32},Union{}},CategoricalArray{Int64,1,UInt32,Int64,CategoricalValue{Int64,UInt32},Union{}},Array{String,1}}}) at /Users/anthony/.julia/packages/DataFrames/z2XOB/src/other/tables.jl:13
[2] DataFrame(::NamedTuple{(:a, :b, :c),Tuple{CategoricalArray{Float64,1,UInt32,Float64,CategoricalValue{Float64,UInt32},Union{}},CategoricalArray{Int64,1,UInt32,Int64,CategoricalValue{Int64,UInt32},Union{}},Array{String,1}}}) at /Users/anthony/.julia/packages/DataFrames/z2XOB/src/other/tables.jl:17
[3] top-level scope at none:0

Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:,:b)
OS: macOS (x86_64-apple-darwin14.5.0)
CPU: Intel(R) Core(TM) i5-3427U CPU @ 1.80GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, ivybridge)
Environment:
JULIA_PATH = /Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia
````

[nalimilan]

It works on Julia 1.1.0. https://github.com/JuliaLang/julia/pull/29406 hasn't been backported to 1.0 yet as it's not clear whether it could break code there. It's marked for triage though.