Mlj.jl: Getting PosDefException in LDA

[darrencl]

Hi,

I'm training my model using LDA algorithm from MultivariateStats.jl for classification and I got this error. I am not sure if this is the right place to ask for this (instead of MultivariateStats.jl).

PosDefException: matrix is not positive definite; Cholesky factorization failed.

Stacktrace:
 [1] chkposdef at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/lapack.jl:50 [inlined]
 [2] sygvd!(::Int64, ::Char, ::Char, ::Array{Float64,2}, ::Array{Float64,2}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/lapack.jl:5075
 [3] eigen! at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.1/LinearAlgebra/src/symmetric.jl:646 [inlined]
 [4] mclda_solve!(::Array{Float64,2}, ::Array{Float64,2}, ::Symbol, ::Int64, ::Float64) at /home/darren/.julia/packages/MultivariateStats/BYMwD/src/lda.jl:175
 [5] mclda_solve at /home/darren/.julia/packages/MultivariateStats/BYMwD/src/lda.jl:164 [inlined]
 [6] #multiclass_lda#36(::Symbol, ::Int64, ::Float64, ::Function, ::MultivariateStats.MulticlassLDAStats{Float64}) at /home/darren/.julia/packages/MultivariateStats/BYMwD/src/lda.jl:160
 [7] #fit#35 at ./none:0 [inlined]
 [8] (::getfield(StatsBase, Symbol("#kw##fit")))(::NamedTuple{(:method, :outdim, :regcoef, :covestimator_within, :covestimator_between),Tuple{Symbol,Int64,Float64,StatsBase.SimpleCovariance,StatsBase.SimpleCovariance}}, ::typeof(StatsBase.fit), ::Type{MultivariateStats.MulticlassLDA}, ::Int64, ::Array{Float64,2}, ::Array{Int64,1}) at ./none:0
 [9] fit(::LDA, ::Int64, ::DataFrame, ::CategoricalArray{String,1,UInt32,String,CategoricalString{UInt32},Union{}}) at /home/darren/.julia/packages/MLJModels/nWDbp/src/MultivariateStats.jl:311
 [10] #fit!#140(::Array{Int64,1}, ::Int64, ::Bool, ::Function, ::Machine{LDA}) at /home/darren/.julia/packages/MLJBase/t7MaX/src/machines.jl:180
 [11] (::getfield(StatsBase, Symbol("#kw##fit!")))(::NamedTuple{(:rows,),Tuple{Array{Int64,1}}}, ::typeof(fit!), ::Machine{LDA}) at ./none:0
 [12] top-level scope at ./In[63]:34

The exact same data works on DecisionTreeClassifier. My mathematics is not that strong, so if someone could enlighten me on what is expected in my data, it would be much appreciated!

Thanks!

[OkonSamuel]

@darrencl dropping the data u used to fit the model would be helpful.

[darrencl]

@OkonSamuel Thanks for your reply! But, how can I do that? Note that this error occurs in fit! function in the LDA machine.

I have partition my data into train and test, then only use the train set to fit and try to predict the test set like below, so why should I drop the train set when I don't use it in the testing?

...
train, test = partition(eachindex(y), 0.8, shuffle=true)
...
mach = machine(lda_model, X_real, y)
fit!(mach, rows=train) # err here
y_pred = predict(mach, X_real[test,:])

Also, why this works in DecisionTreeClassifier but not in LDA?

[OkonSamuel]

@darrencl. The train partition is what fit! uses to fit the machine.
The error is caused by singularity in the within class covariance estimate (i.e rank(Sw) < nfeatures ) used in the underlying algorithm. In this case DecisionTreeClassifier would still fit the model.
Whether or not singularity would occur depends on the number of features, number of samples, number of classes used in training and existence of dependent variables in the dataset (which was why i needed to see your data).
The LDA model from MultivariateStats package has a regcoef parameter that you could make use of to ensure non-singularity. Just reset the regcoef parameter as shown below or better still use a different CovarianceEstimator

lda_model =@load LDA pkg=MultivariateStats
lda.regcoef = 1e-6

train, test = partition(eachindex(y), 0.8, shuffle=true)
mach = machine(lda_model, X, y)

fit!(mach, rows=train) 
y_pred = predict(mach, X[test,:])

[darrencl]

@OkonSamuel Thanks a lot for the help and explanation. It really helps!

Apology that I misunderstood 'dropping the data'. I thought you mean drop/remove, that's why I got confused :laughing:. Anyway, the data I use is confidential, so I couldn't share it, but it basically just a spectrum data, i.e. set of Float64s.