Mlj.jl: SVC won't tune cost
Describe the bug
To Reproduce
data = DataFrame(urldownload("https://storage.googleapis.com/download.tensorflow.org/data/creditcard.csv")
data_fraud = filter(row -> row[:Class] == 1, data)
data_notfraud = filter(row -> row[:Class] == 0, data)
#split into training and test data, training wi th 1 extra row due to odd numbber of non-fraudulent claims and each with even number of fraudulent claims
data_train = vcat(data_fraud[1:round(Int, nrow(data_fraud)/2),:], data_notfraud[1:round(Int, nrow(data_notfraud)/2),:])
sort!(data_train, :Time)
data_test = vcat(data_fraud[round(Int, nrow(data_fraud)/2)+1:nrow(data_fraud),:], data_notfraud[round(Int, nrow(data_notfraud)/2)+1:nrow(data_notfraud),:])
sort!(data_test, :Time)
#Setup train and test arrays/vectors
X_train = DataFrames.select(data_train, Not(:Class))
X_test = DataFrames.select(data_test, Not(:Class))
y_train = categorical(data_train.Class)
y_train_int = data_train.Class
y_test = categorical(data_test.Class)
y_test_int = data_test.Class
model_svm = @load SVC
svc = machine(model_svm, X_train, y_train)
r = range(model_svm, :cost, lower=0.001, upper=1.0, scale=:log)
self_tuning_svm_model = TunedModel(model=model_svm,
resampling = CV(nfolds=5),
tuning = Grid(resolution=3),
range = r,
measure = misclassification_rate)
self_tuning_svm = machine(self_tuning_svm_model, X_train, y_train)
fit!(self_tuning_svm)
Expected behavior
Expected to tune cost hyperparameter, instead crashes with Segmentation Fault:11
Additional context
Versions
kbjarnason
All 8 comments
@kbjarnason. What version of julia and MLJ are you using
OkonSamuel
on 23 May 2020
Julia version: 1.4.1
MLJ version: 0.11.2
Kind Regards,
Kristian Bjarnason
On 23 May 2020, at 9:32 pm, Okon Samuel notifications@github.com wrote:
@kbjarnason https://github.com/kbjarnason. What version of julia and MLJ are you using
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/alan-turing-institute/MLJ.jl/issues/551#issuecomment-633123822, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIEN72XBPSRY2ECXDDKYC5DRTAQFJANCNFSM4NINEBNQ.
kbjarnason
on 23 May 2020
@kbjarnason It seems this has been resolved on slack
OkonSamuel
on 24 May 2020
I haven’t tried tuning an svm yet but hopefully yes!
Kind Regards,
Kristian Bjarnason
On 24 May 2020, at 2:36 am, Okon Samuel notifications@github.com wrote:
@kbjarnason https://github.com/kbjarnason It seems this has been resolved on slack
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/alan-turing-institute/MLJ.jl/issues/551#issuecomment-633159502, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIEN72Q6IZT4JU2PCRCV2DTRTBTXLANCNFSM4NINEBNQ.
kbjarnason
on 24 May 2020
tuning of SVC works here on my PC. Provided the data is standardized before use
```julia
julia> mymodel1 = @pipeline Std_SVC(std_model = Standardizer(),
svc = SVC())
Std_SVC(
std_model = Standardizer(
features = Symbol[],
ignore = false,
ordered_factor = false,
count = false),
svc = SVC(
kernel = LIBSVM.Kernel.RadialBasis,
gamma = -1.0,
weights = nothing,
cost = 1.0,
degree = 3,
coef0 = 0.0,
tolerance = 0.001,
shrinking = true,
probability = false)) @ 1…51
julia> svc = machine(mymodel1, X_train, y_train)
Machine{Std_SVC} @ 1…86
julia> r = range(mymodel1, :(svc.cost), lower=0.001, upper=1.0, scale=:log)
MLJBase.NumericRange(Float64, :(svc.cost), ... )
julia> self_tuning_svm_model = TunedModel(model=mymodel1,
resampling = CV(nfolds=5),
tuning = Grid(resolution=3),
range = r,
measure = misclassification_rate)
DeterministicTunedModel(
model = Std_SVC(
std_model = Standardizer @ 1…01,
svc = SVC @ 1…20),
tuning = Grid(
goal = nothing,
resolution = 3,
shuffle = true,
rng = Random._GLOBAL_RNG()),
resampling = CV(
nfolds = 5,
shuffle = false,
rng = Random._GLOBAL_RNG()),
measure = misclassification_rate(),
weights = nothing,
operation = MLJModelInterface.predict,
range = NumericRange(
field = :(svc.cost),
lower = 0.001,
upper = 1.0,
origin = 0.5005,
unit = 0.4995,
scale = :log),
train_best = true,
repeats = 1,
n = nothing,
acceleration = CPU1{Nothing}(nothing),
acceleration_resampling = CPU1{Nothing}(nothing),
check_measure = true) @ 5…94
julia> self_tuning_svm = machine(self_tuning_svm_model, X_train, y_train)
Machine{DeterministicTunedModel{Grid,…}} @ 4…58
julia> fit!(self_tuning_svm, verbosity=6)
Below is the response i got
```julia
[ Info: Training Machine{DeterministicTunedModel{Grid,…}} @ 5…57.
[ Info: Attempting to evaluate 3 models.
Evaluating over 3 metamodels: 0%[> ] ETA: N/A[ Info: Training Machine{Resampler{CV,…}} @ 4…18.
[ Info: Training Machine{Std_SVC} @ 1…10.
[ Info: Training NodalMachine{Standardizer} @ 1…47.
[ Info: Training NodalMachine{SVC} @ 7…00.
Evaluating over 5 folds: 20%[=====> ] ETA: 0:06:57[ Info: Training Machine{Std_SVC} @ 1…10.
[ Info: Training NodalMachine{Standardizer} @ 8…60.
[ Info: Training NodalMachine{SVC} @ 1…19.
Evaluating over 5 folds: 40%[==========> ] ETA: 0:04:55[ Info: Training Machine{Std_SVC} @ 1…10.
[ Info: Training NodalMachine{Standardizer} @ 1…37.
[ Info: Training NodalMachine{SVC} @ 5…99.
Evaluating over 5 folds: 60%[===============> ] ETA: 0:03:15[ Info: Training Machine{Std_SVC} @ 1…10.
[ Info: Training NodalMachine{Standardizer} @ 4…47.
[ Info: Training NodalMachine{SVC} @ 7…09.
Evaluating over 5 folds: 80%[====================> ] ETA: 0:01:37[ Info: Training Machine{Std_SVC} @ 1…10.
[ Info: Training NodalMachine{Standardizer} @ 8…32.
[ Info: Training NodalMachine{SVC} @ 1…97.
Evaluating over 5 folds: 100%[=========================] Time: 0:08:21
hyperparameters: (std_model = (features = Symbol[], ignore = false, ordered_factor = false, count = false), svc = (kernel = LIBSVM.Kernel.RadialBasis, gamma = -1.0, weights = nothing, cost = 1.0, degree = 3, coef0 = 0.0, tolerance = 0.001, shrinking = true, probability = false))
result: (measure = [misclassification_rate], measurement = [0.0013482692384317713])
Evaluating over 3 metamodels: 33%[========> ] ETA: 0:16:47[ Info: Updating Machine{Resampler{CV,…}} @ 4…18.
[ Info: Training Machine{Std_SVC} @ 4…76.
[ Info: Training NodalMachine{Standardizer} @ 1…92.
[ Info: Training NodalMachine{SVC} @ 3…71.
Evaluating over 5 folds: 20%[=====> ] ETA: 0:00:23[ Info: Training Machine{Std_SVC} @ 4…76.
[ Info: Training NodalMachine{Standardizer} @ 1…29.
[ Info: Training NodalMachine{SVC} @ 2…38.
Evaluating over 5 folds: 40%[==========> ] ETA: 0:00:18[ Info: Training Machine{Std_SVC} @ 4…76.
[ Info: Training NodalMachine{Standardizer} @ 8…58.
[ Info: Training NodalMachine{SVC} @ 8…01.
Evaluating over 5 folds: 60%[===============> ] ETA: 0:00:13[ Info: Training Machine{Std_SVC} @ 4…76.
[ Info: Training NodalMachine{Standardizer} @ 1…39.
[ Info: Training NodalMachine{SVC} @ 6…93.
Evaluating over 5 folds: 80%[====================> ] ETA: 0:00:07[ Info: Training Machine{Std_SVC} @ 4…76.
[ Info: Training NodalMachine{Standardizer} @ 1…44.
[ Info: Training NodalMachine{SVC} @ 6…31.
Evaluating over 5 folds: 100%[=========================] Time: 0:00:35
hyperparameters: (std_model = (features = Symbol[], ignore = false, ordered_factor = false, count = false), svc = (kernel = LIBSVM.Kernel.RadialBasis, gamma = -1.0, weights = nothing, cost = 0.0010000000000000002, degree = 3, coef0 = 0.0, tolerance = 0.001, shrinking = true, probability = false))
result: (measure = [misclassification_rate], measurement = [0.0017274686672644763])
Evaluating over 3 metamodels: 67%[================> ] ETA: 0:04:30[ Info: Updating Machine{Resampler{CV,…}} @ 4…18.
[ Info: Training Machine{Std_SVC} @ 1…49.
[ Info: Training NodalMachine{Standardizer} @ 6…24.
[ Info: Training NodalMachine{SVC} @ 1…50.
Evaluating over 5 folds: 20%[=====> ] ETA: 0:36:39[ Info: Training Machine{Std_SVC} @ 1…49.
[ Info: Training NodalMachine{Standardizer} @ 4…35.
[ Info: Training NodalMachine{SVC} @ 5…51.
Evaluating over 5 folds: 40%[==========> ] ETA: 0:27:43[ Info: Training Machine{Std_SVC} @ 1…49.
[ Info: Training NodalMachine{Standardizer} @ 1…36.
[ Info: Training NodalMachine{SVC} @ 9…26.
Evaluating over 5 folds: 60%[===============> ] ETA: 0:13:02[ Info: Training Machine{Std_SVC} @ 1…49.
[ Info: Training NodalMachine{Standardizer} @ 1…72.
[ Info: Training NodalMachine{SVC} @ 7…00.
Evaluating over 5 folds: 80%[====================> ] ETA: 0:05:08[ Info: Training Machine{Std_SVC} @ 1…49.
[ Info: Training NodalMachine{Standardizer} @ 1…09.
[ Info: Training NodalMachine{SVC} @ 1…69.
Evaluating over 5 folds: 100%[=========================] Time: 0:21:45
hyperparameters: (std_model = (features = Symbol[], ignore = false, ordered_factor = false, count = false), svc = (kernel = LIBSVM.Kernel.RadialBasis, gamma = -1.0, weights = nothing, cost = 0.0316227766016838, degree = 3, coef0 = 0.0, tolerance = 0.001, shrinking = true, probability = false))
result: (measure = [misclassification_rate], measurement = [0.0017274686672644763])
Evaluating over 3 metamodels: 100%[=========================] Time: 0:30:44
[ Info: Training Machine{Std_SVC} @ 1…21.
[ Info: Training NodalMachine{Standardizer} @ 1…58.
[ Info: Features standarized:
[ Info: :Time mu=52404.06846717789 sigma=21105.891389395143
[ Info: :V1 mu=-0.24912997191915257 sigma=1.8138022132834126
[ Info: :V2 mu=0.02049834563565114 sigma=1.6101275597114075
[ Info: :V3 mu=0.6743434126322257 sigma=1.264602483901866
[ Info: :V4 mu=0.13670722032461008 sigma=1.3216741833889702
[ Info: :V5 mu=-0.28267985705042487 sigma=1.304747365721006
[ Info: :V6 mu=0.07847387088465514 sigma=1.2824722349684927
[ Info: :V7 mu=-0.1177886408671241 sigma=1.1646058900628178
[ Info: :V8 mu=0.0648447088715078 sigma=1.2335252466217246
[ Info: :V9 mu=-0.08883888142212973 sigma=1.089852697112262
[ Info: :V10 mu=-0.022276876067042074 sigma=1.050090086042523
[ Info: :V11 mu=0.20809289999976968 sigma=1.047495314095711
[ Info: :V12 mu=0.03119914663778642 sigma=1.0164442145577053
[ Info: :V13 mu=-0.01879793069139625 sigma=0.9979300103553298
[ Info: :V14 mu=0.029725206461356114 sigma=0.9128962666106761
[ Info: :V15 mu=0.2288666327967404 sigma=0.9254793110065512
[ Info: :V16 mu=-0.007946100988015351 sigma=0.8754566066059178
[ Info: :V17 mu=0.04302915348810576 sigma=0.8780109076435942
[ Info: :V18 mu=-0.08353726279212678 sigma=0.8314236851433924
[ Info: :V19 mu=-0.01705308140533924 sigma=0.8084443827168185
[ Info: :V20 mu=0.04220566399929354 sigma=0.7193732830649423
[ Info: :V21 mu=-0.03984833177803252 sigma=0.7211066188231148
[ Info: :V22 mu=-0.11794899463349022 sigma=0.6346234175484644
[ Info: :V23 mu=-0.033196024494739636 sigma=0.5898355036519893
[ Info: :V24 mu=0.011300486949693891 sigma=0.595740760694303
[ Info: :V25 mu=0.13087995655154708 sigma=0.4372768169961001
[ Info: :V26 mu=0.02155496841817364 sigma=0.49234146520726885
[ Info: :V27 mu=0.0006210959116538015 sigma=0.3886537267552494
[ Info: :V28 mu=0.002203761383392946 sigma=0.3061566145828678
[ Info: :Amount mu=90.68276101794893 sigma=246.46435777432401
[ Info: Training NodalMachine{SVC} @ 1…94.
..
WARNING: using -h 0 may be faster
*..*
optimization finished, #iter = 4618
nu = 0.002033
obj = -195.101443, rho = -0.858773
nSV = 2242, nBSV = 115
Total nSV = 2242
Machine{DeterministicTunedModel{Grid,…}} @ 5…57
But don't take my word, you can try it out for yourself
OkonSamuel
on 24 May 2020
just reposting what i said in slack
testing the data using MLJ's SVMClassifier (gotten by interfacing to ScikitLearn.jl), ScikitLearn.jl SVC(I used ScikitLearn.jl package here to fit and tune) and LIBSVM svmtrain is extremely slow and sometimes segfaults if force-cancelled with CTRL+C indicating that the issue is not with MLJ tuning facility but with the nature of the data.(For SVM models the data has to be scaled)
Sklearn has a gamma = scale parameter set by default that scales the dataset by 1 / (n_features * X.var()). I have opened an issue at MLJModels suggesting this as the default in LIBSVM implementation.
OkonSamuel
on 25 May 2020
@kbjarnason is it okay if i close this?
OkonSamuel
on 25 May 2020
Go for it! Thanks for the help :)
On Mon, 25 May 2020, 18:49 Okon Samuel, notifications@github.com wrote:
@kbjarnason https://github.com/kbjarnason is it okay if i close this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/alan-turing-institute/MLJ.jl/issues/551#issuecomment-633649906,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIEN72WWZJTQ2XXJB54Q4BTRTKOQFANCNFSM4NINEBNQ
.
kbjarnason
on 25 May 2020
Related issues
tlienart
·
3Comments
CameronBieganek
·
6Comments
tlienart
·
6Comments
ablaom
·
4Comments
ExpandingMan
·
7Comments