Mlj.jl: DimensionMismatch in evaluate()

[CameronBieganek]

I'm not sure what's going on here, but I think this should work:

using DataFrames
using RDatasets
using MLJ
using MLJLinearModels

iris = dataset("datasets", "iris")
df = filter(r -> r.Species != "virginica", iris)
y = df.Species
X = select(df, Not(:Species))

model = LogisticClassifier(penalty=:none)
logistic_machine = machine(model, X, y)
holdout = Holdout(shuffle=true, rng=1)

logistic_auc1 = evaluate(
    model, X, y,
    resampling = holdout,
    measure = auc
)

logistic_auc2 = evaluate!(
    logistic_machine,
    resampling = holdout,
    measure = auc
)

At the REPL:

julia> logistic_auc = evaluate(
           model, X, y,
           resampling = holdout,
           measure = auc
       )
ERROR: DimensionMismatch("new dimensions (5, 3) must be consistent with array size 5")
Stacktrace:
 [1] (::Base.var"#throw_dmrsa#196")(::Tuple{Int64,Int64}, ::Int64) at ./reshapedarray.jl:41
 [2] reshape at ./reshapedarray.jl:45 [inlined]
 [3] reshape at ./reshapedarray.jl:116 [inlined]
 [4] apply_X(::Array{Float64,2}, ::Array{Float64,1}, ::Int64) at /Users/bieganek/.julia/packages/MLJLinearModels/Av1bE/src/utils.jl:42
 [5] predict(::LogisticClassifier, ::Tuple{Array{Float64,1},NTuple{4,Symbol},Int64,CategoricalArray{String,1,UInt32,String,CategoricalValue{String,UInt32},Union{}}}, ::DataFrame) at /Users/bieganek/.julia/packages/MLJLinearModels/Av1bE/src/mlj/interface.jl:73
 [6] predict(::Machine{LogisticClassifier}, ::DataFrame) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/operations.jl:29
 [7] (::MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}})(::Machine{LogisticClassifier}, ::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:701
 [8] (::MLJBase.var"#193#194"{MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}},Dict{Int64,Machine{LogisticClassifier}},Int64})(::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:564
 [9] mapreduce_first at ./reduce.jl:384 [inlined]
 [10] _mapreduce(::MLJBase.var"#193#194"{MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}},Dict{Int64,Machine{LogisticClassifier}},Int64}, ::typeof(vcat), ::IndexLinear, ::UnitRange{Int64}) at ./reduce.jl:395
 [11] _mapreduce_dim at ./reducedim.jl:312 [inlined]
 [12] #mapreduce#580 at ./reducedim.jl:307 [inlined]
 [13] mapreduce at ./reducedim.jl:307 [inlined]
 [14] _evaluate!(::MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}}, ::Dict{Int64,Machine{LogisticClassifier}}, ::CPU1{Nothing}, ::Int64, ::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:563
 [15] evaluate!(::Machine{LogisticClassifier}, ::Array{Tuple{Array{Int64,1},Array{Int64,1}},1}, ::Nothing, ::Nothing, ::Int64, ::Int64, ::Array{MLJBase.AUC,1}, ::typeof(predict), ::CPU1{Nothing}, ::Bool) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:719
 [16] evaluate!(::Machine{LogisticClassifier}, ::Holdout, ::Nothing, ::Nothing, ::Int64, ::Int64, ::Array{MLJBase.AUC,1}, ::Function, ::CPU1{Nothing}, ::Bool) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:780
 [17] evaluate!(::Machine{LogisticClassifier}; resampling::Holdout, measures::Nothing, measure::MLJBase.AUC, weights::Nothing, operation::Function, acceleration::CPU1{Nothing}, rows::Nothing, repeats::Int64, force::Bool, check_measure::Bool, verbosity::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:526
 [18] evaluate(::LogisticClassifier, ::DataFrame, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Symbol,MLJModelInterface.MLJType,Tuple{Symbol,Symbol},NamedTuple{(:resampling, :measure),Tuple{Holdout,MLJBase.AUC}}}) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:544
 [19] top-level scope at REPL[12]:1

julia> logistic_auc = evaluate!(
           logistic_machine,
           resampling = holdout,
           measure = auc
       )
ERROR: DimensionMismatch("new dimensions (5, 3) must be consistent with array size 10")
Stacktrace:
 [1] (::Base.var"#throw_dmrsa#196")(::Tuple{Int64,Int64}, ::Int64) at ./reshapedarray.jl:41
 [2] reshape at ./reshapedarray.jl:45 [inlined]
 [3] reshape at ./reshapedarray.jl:116 [inlined]
 [4] apply_X(::Array{Float64,2}, ::Array{Float64,1}, ::Int64) at /Users/bieganek/.julia/packages/MLJLinearModels/Av1bE/src/utils.jl:42
 [5] predict(::LogisticClassifier, ::Tuple{Array{Float64,1},NTuple{4,Symbol},Int64,CategoricalArray{String,1,UInt32,String,CategoricalValue{String,UInt32},Union{}}}, ::DataFrame) at /Users/bieganek/.julia/packages/MLJLinearModels/Av1bE/src/mlj/interface.jl:73
 [6] predict(::Machine{LogisticClassifier}, ::DataFrame) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/operations.jl:29
 [7] (::MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}})(::Machine{LogisticClassifier}, ::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:701
 [8] (::MLJBase.var"#193#194"{MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}},Dict{Int64,Machine{LogisticClassifier}},Int64})(::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:564
 [9] mapreduce_first at ./reduce.jl:384 [inlined]
 [10] _mapreduce(::MLJBase.var"#193#194"{MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}},Dict{Int64,Machine{LogisticClassifier}},Int64}, ::typeof(vcat), ::IndexLinear, ::UnitRange{Int64}) at ./reduce.jl:395
 [11] _mapreduce_dim at ./reducedim.jl:312 [inlined]
 [12] #mapreduce#580 at ./reducedim.jl:307 [inlined]
 [13] mapreduce at ./reducedim.jl:307 [inlined]
 [14] _evaluate!(::MLJBase.var"#get_measurements#209"{Array{Tuple{Array{Int64,1},Array{Int64,1}},1},Nothing,Int64,Array{MLJBase.AUC,1},typeof(predict),Bool,DataFrame,CategoricalArray{String,1,UInt8,String,CategoricalValue{String,UInt8},Union{}}}, ::Dict{Int64,Machine{LogisticClassifier}}, ::CPU1{Nothing}, ::Int64, ::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:563
 [15] evaluate!(::Machine{LogisticClassifier}, ::Array{Tuple{Array{Int64,1},Array{Int64,1}},1}, ::Nothing, ::Nothing, ::Int64, ::Int64, ::Array{MLJBase.AUC,1}, ::typeof(predict), ::CPU1{Nothing}, ::Bool) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:719
 [16] evaluate!(::Machine{LogisticClassifier}, ::Holdout, ::Nothing, ::Nothing, ::Int64, ::Int64, ::Array{MLJBase.AUC,1}, ::Function, ::CPU1{Nothing}, ::Bool) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:780
 [17] evaluate!(::Machine{LogisticClassifier}; resampling::Holdout, measures::Nothing, measure::MLJBase.AUC, weights::Nothing, operation::Function, acceleration::CPU1{Nothing}, rows::Nothing, repeats::Int64, force::Bool, check_measure::Bool, verbosity::Int64) at /Users/bieganek/.julia/packages/MLJBase/LxAsa/src/resampling.jl:526
 [18] top-level scope at REPL[13]:1

Package versions:

(@v1.4) pkg> status DataFrames RDatasets MLJ MLJLinearModels
Status `~/.julia/environments/v1.4/Project.toml`
  [a93c6f00] DataFrames v0.21.0
  [add582a8] MLJ v0.11.3
  [6ee0df7b] MLJLinearModels v0.3.4
  [ce6b1742] RDatasets v0.6.8

[tlienart]

Thanks for that, looks like I did something silly in the recent cleanup of MLJLM, I’ll look into it tomorrow, apologies for the nuisance.

In the mean time, wrt your discourse post, you could try using the scikitlearn logistic for now, and could try using a learning network with a node operation predict and the next node operation mode, you should then be able to evaluate metrics on each. I’m typing this on a mobile without having had a chance to try it out so take it with a pinch of salt. Anthony will know but he’s away for the week afaik.

[OkonSamuel]

@CameronBieganek. just set model.multi_class=true that should fix it.
@tlienart. This is related to https://github.com/alan-turing-institute/MLJLinearModels.jl/issues/71

[tlienart]

Yes but I set something specifically to automatically switch so that people didn't have to specify this manually so probably still a bug :) I'll have a look now.

[tlienart]

Ok actually the problem is a bit more subtle.

on your data:

julia> unique(y)
2-element Array{String,1}:
 "setosa"
 "versicolor"

julia> classes(y[1])
3-element CategoricalArray{String,1,UInt8}:
 "setosa"
 "versicolor"
 "virginica"

This is because you filter to only get two classes but then you don't adjust the levels. So there's an ambiguity as to how many levels you want: 2 or 3?

MLJ considers you've not specified anything specifically so it's still 3, MLJLinearModels on the other hand, internally, looks at the number of unique values that are present in y and sees that there's only 2 of those...

Next steps:

• on your side, use levels!(y, unique(y)) before calling evaluate for your example
• on MLJLM side, MLJLM looks at the data to figure out the number of classes but MLJ knows that number of classes, so it looks like I should just add a keyword argument to allow explicit specification of the number of classes, I'll think about it.

[tlienart]

superseded by #542

[CameronBieganek]

This is because you filter to only get two classes but then you don't adjust the levels.

Yes, my apologies. I meant to call droplevels!(y), but when it came time to do it, it slipped my mind. I guess I have a short memory. :joy: