Mlj.jl: Help with coercing strings for binary data into Continuous variables

[drcxcruz]

Describe the bug

To Reproduce

Data from example one of https://cran.r-project.org/web/packages/InvariantCausalPrediction/InvariantCausalPrediction.pdf

julia> schema(setScientificTypes(X))
┌───────────┬──────────────────────────────────┬──────────────────┐
│ _.names │ _.types │ _.scitypes │
├───────────┼──────────────────────────────────┼──────────────────┤
│ gender │ CategoricalString{UInt32} │ Multiclass{2} │
│ ethnicity │ CategoricalString{UInt32} │ Multiclass{3} │
│ score │ Float64 │ Continuous │
│ fcollege │ CategoricalString{UInt32} │ Multiclass{2} │
│ mcollege │ CategoricalString{UInt32} │ Multiclass{2} │
│ home │ CategoricalString{UInt32} │ Multiclass{2} │
│ urban │ CategoricalString{UInt32} │ Multiclass{2} │
│ unemp │ Float64 │ Continuous │
│ wage │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ tuition │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ income │ CategoricalString{UInt32} │ Multiclass{2} │
│ region │ CategoricalString{UInt32} │ Multiclass{1} │
└───────────┴──────────────────────────────────┴──────────────────┘
_.nrows = 99

Float64.(Y)
99-element Array{Float64,1}:
10.0
20.0
30.0
0.0
1.0
-->

```julia
xbr = XGBoostRegressor()

    r1 = range(xbr, :max_depth, lower = 3, upper = 10)
    r2 = range(xbr, :num_round, lower = 1, upper = 25)

    tm = TunedModel(
        model = xbr,
        tuning = Grid(resolution = 7),
        resampling = CV(rng = 11),
        ranges = [r1, r2],
        measure = rms
    )

    mtm = machine(tm, setScientificTypes(X), Float64.(y[:, 1]))
    fit!(mtm)  #freezing

Expected behavior

Additional context

Versions

[OkonSamuel]

@drcxcruz. The data simulated in the link you gave doesn't match what you used above. From the data simulated in the link. I can't reproduce your error.

[drcxcruz]

thank you for your time. my apologies, the data set in question is example 3.

#

# 3rd example:

# College Distance data

library(AER)
data("CollegeDistance")
CD <- CollegeDistance

define two experimental settings by

distance to closest 4-year college

ExpInd <- list()
ExpInd[[1]] <- which(CD$distance < quantile(CD$distance,0.5))
ExpInd[[2]] <- which(CD$distance >= quantile(CD$distance,0.5))

target variable is binary (did education lead at least to BA degree?)

Y <- as.factor(CD$education>=16)

use these predictors

X <- CD[,c("gender","ethnicity","score","fcollege","mcollege","home",
"urban","unemp","wage","tuition","income","region")]

[OkonSamuel]

@drcxcruz. Now i can reproduce. The reason you got this warning is because of the datatype of X. (setScientificTypes() isn't a valid function). XGBoost only accepts X that is of scitypeTable{#s23} where #s23<:(AbstractArray{#s25,1} where #s25<:ScientificTypes.Continuous) (i.e only Floats colunms are allowed not strings or integers columns). But the columns [:gender, :ethinicity, :fcollege, :mcollege, :home, :urban, :income, :region] are not Float columns.

Since most of these variables are meant to be Bool columns. You can run the following julia code to make X compatible with model scitype.

using DataFrames, MLJ
## note X is the Feature matrix
cond = eltype.(eachcol(X) ) .== String  #returns true for columns that are strings
col_names = names(X)[condition] #get column names having  String type
for col in col_names
       X[!, col] .= Float64.(MLJ.int(categorical(X[!, col])))
end

After doing this the code below runs

 xbr = XGBoostRegressor()

        r1 = range(xbr, :max_depth, lower = 3, upper = 10)
        r2 = range(xbr, :num_round, lower = 1, upper = 25)

        tmr = TunedModel(
            model = xbr,
            tuning = Grid(resolution = 7),
            resampling = CV(rng = 11),
            ranges = [r1, r2],
            measure = rms
        )

        mtmr = machine(tmr, X, Float64.(y) )
        fit!(mtmr)
 xbc = XGBoostClassifier()

        r1 = range(xbc, :max_depth, lower = 3, upper = 10)
        r2 = range(xbc, :num_round, lower = 1, upper = 25)

        tmc = TunedModel(
            model = xbc,
            tuning = Grid(resolution = 7),
            resampling = CV(rng = 11),
            ranges = [r1, r2],
            measure = rms
        )

        mtmc = machine(tmc, X, categorical(y) )
        fit!(mtmc) 

[drcxcruz]

thank you very much! i will try it out.

[ablaom]

A slightly more MLJ-idiomatic workflow doing the same thing (assuming all strings are binary): Use coerce(X, Textual=>OrderedFactor) (in place coerce! works for data frames) to convert your strings to ordered factors (OrderedFactor{2} in your case) and then use the transformer OneHotEncoder(drop_last=true) to make these into Continuous.

A helpful first step in any workflow is schema(X) to inspect scitypes. scitype(X) is a kind of more compressed version of the same info used for type comparisons.

[drcxcruz]

Thank you for your teachings. It is very interesting and helpful. Sorry, I still not able to get it working, Next is the test I am running today. I feel this simple approach following MLJ lab tutorial should work. Is the MultiClass scientific type support by the XGBooster? If not, should it be supported? is there a "autotype(df, :string_to_orderedfactor))"? It would bey very useful to have autotype(df, :string_to_orderedfactor)).

Also noticed, that MLJ incorrectly defined the following scientific types. it appears that running autotype more than once in the same data rame breaks the scientific types.

| wage │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ tuition │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ income │ CategoricalString{UInt32} │ Multiclass{2} │

is the the above is a bug? :)

using DataFrames, MLJ, Combinatorics, CategoricalArrays, PrettyPrinting
@load XGBoostRegressor
@load XGBoostClassifier

dfX = CSV.File("X3small.csv") |> DataFrame
dfY = CSV.File("Y3small.csv") |> DataFrame
dfE = CSV.File("ExpInd3small.csv") |> DataFrame

=

julia> schema(dfX)
schema(dfX)
┌───────────┬─────────┬────────────┐
│ _.names │ _.types │ _.scitypes │
├───────────┼─────────┼────────────┤
│ gender │ String │ Textual │
│ ethnicity │ String │ Textual │
│ score │ Float64 │ Continuous │
│ fcollege │ String │ Textual │
│ mcollege │ String │ Textual │
│ home │ String │ Textual │
│ urban │ String │ Textual │
│ unemp │ Float64 │ Continuous │
│ wage │ Float64 │ Continuous │
│ tuition │ Float64 │ Continuous │
│ income │ String │ Textual │
│ region │ String │ Textual │
└───────────┴─────────┴────────────┘
_.nrows = 99
=#

function setScientificTypes(df::DataFrame)
df = coerce(df, autotype(df, :few_to_finite))
df = coerce(df, autotype(df, :discrete_to_continuous))
df = coerce(df, autotype(df, :string_to_multiclass))
return (df)
end

#

dfX = setScientificTypes(dfX)

=

julia> schema(dfX)
┌───────────┬──────────────────────────────────┬──────────────────┐
│ _.names │ _.types │ _.scitypes │
├───────────┼──────────────────────────────────┼──────────────────┤
│ gender │ CategoricalString{UInt32} │ Multiclass{2} │
│ ethnicity │ CategoricalString{UInt32} │ Multiclass{3} │
│ score │ Float64 │ Continuous │
│ fcollege │ CategoricalString{UInt32} │ Multiclass{2} │
│ mcollege │ CategoricalString{UInt32} │ Multiclass{2} │
│ home │ CategoricalString{UInt32} │ Multiclass{2} │
│ urban │ CategoricalString{UInt32} │ Multiclass{2} │
│ unemp │ Float64 │ Continuous │
│ wage │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ tuition │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ income │ CategoricalString{UInt32} │ Multiclass{2} │
│ region │ CategoricalString{UInt32} │ Multiclass{1} │
└───────────┴──────────────────────────────────┴──────────────────┘
_.nrows = 99
=#

xbr = XGBoostRegressor()

r1 = range(xbr, :max_depth, lower = 3, upper = 10)
r2 = range(xbr, :num_round, lower = 1, upper = 25)

tm = TunedModel(
model = xbr,
tuning = Grid(resolution = 7),
resampling = CV(rng = 11),
ranges = [r1, r2],
measure = rms
)

mtm = machine(tm, dfX, Float64.(dfY.Y[:, 1]))

=

┌ Warning: The scitype of X, in machine(model, X, y) or machine(model, X, y, w) is incompatible with model:
│ scitype(X) = Table{Union{AbstractArray{Continuous,1}, AbstractArray{Multiclass{3},1}, AbstractArray{Multiclass{1},1}, AbstractArray{Multiclass{2},1}, AbstractArray{OrderedFactor{4},1}}}
│ input_scitype(model) = Table{#s23} where #s23<:(AbstractArray{#s25,1} where #s25<:Continuous).
└ @ MLJBase C:\Users\BCP.juliapro\JuliaPro_v1.4.0-1\packages\MLJBase\ESDzL\src\machines.jl:49
=#

[OkonSamuel]

is there a "autotype(df, :string_to_orderedfactor))

@drcxcruz. Sorry MLJ does't support that. To see the list of currently available autotype conversions see here

Also noticed, that MLJ incorrectly defined the following scientific types. it appears that running autotype more than once in the same data rame breaks the scientific types.

| wage │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ tuition │ CategoricalValue{Float64,UInt32} │ OrderedFactor{4} │
│ income │ CategoricalString{UInt32} │ Multiclass{2} │

is the the above is a bug? :)

No. those columns have few values and by following the :few_to_finite rule there are converted into an appropriate Finite subtype (e.g Multiclass or OrderedFactor)

[OkonSamuel]

@drcxcruz.
I wrote some code to help explain better. The code runs on my pc using the data set provided("Although i just downloaded the first 100 rows of the dataset to my PC")

julia> using MLJ, CSV, DataFrames

julia> data = CSV.File("put/path/todata/data.csv") |> DataFrame

# I just copied the feaatures you listed
julia> features = Symbol.(["gender","ethnicity","score","fcollege","mcollege","home",
       "urban","unemp","wage","tuition","income","region"]) 

## Extracted the features_matrix(X) and target(y)
julia> y, X = unpack(data, ==(:education), colname -> colname in features);

julia> all(names(X) .== features) # just to confirm

julia> y = categorical(y .>= 16);

julia> xgc = @load XGBoostClassifier;

#check if the data is compactible with XGboost classifier
julia> matching(xgc, X, y) ## this returns false meaning it isn't.

julia> scitype(y) <: info(xgc).target_scitype #returns true meaning target is okay

julia> scitype(X) <: info(xgc).input_scitype #returns false meaning input is not okay

#check what input_type(feature_matrix) it wants
julia> info(xgc).input_scitype #so it wants all columns to be of Continuous scitype
## meaning it wants columns thats are Float columns. It wont even consider accepting
## Integer columns talkness of Categorical columns.

#Let's check the columns that violate this

julia> schema(X)
┌───────────┬─────────┬────────────┐
│ _.names   │ _.types │ _.scitypes │
├───────────┼─────────┼────────────┤
│ gender    │ String  │ Textual    │
│ ethnicity │ String  │ Textual    │
│ score     │ Float64 │ Continuous │
│ fcollege  │ String  │ Textual    │
│ mcollege  │ String  │ Textual    │
│ home      │ String  │ Textual    │
│ urban     │ String  │ Textual    │
│ unemp     │ Float64 │ Continuous │
│ wage      │ Float64 │ Continuous │
│ tuition   │ Float64 │ Continuous │
│ income    │ String  │ Textual    │
│ region    │ String  │ Textual    │
└───────────┴─────────┴────────────┘
_.nrows = 100

## so [:gender, :ethinicity, :fcollege, :mcollege, :home, :urban, :income, :region] are not Continous columns
## They are Textual (String type) columns)

julia> first(X, 5)
5×12 DataFrame
│ Row │ gender │ ethnicity │ score   │ fcollege │ mcollege │ home   │ urban  │ unemp   │ wage    │ tuition │ income │ region │
│     │ String │ String    │ Float64 │ String   │ String   │ String │ String │ Float64 │ Float64 │ Float64 │ String │ String │
├─────┼────────┼───────────┼─────────┼──────────┼──────────┼────────┼────────┼─────────┼─────────┼─────────┼────────┼────────┤
│ 1   │ male   │ other     │ 39.15   │ yes      │ no       │ yes    │ yes    │ 6.2     │ 8.09    │ 0.88915 │ high   │ other  │
│ 2   │ female │ other     │ 48.87   │ no       │ no       │ yes    │ yes    │ 6.2     │ 8.09    │ 0.88915 │ low    │ other  │
│ 3   │ male   │ other     │ 48.74   │ no       │ no       │ yes    │ yes    │ 6.2     │ 8.09    │ 0.88915 │ low    │ other  │
│ 4   │ male   │ afam      │ 40.4    │ no       │ no       │ yes    │ yes    │ 6.2     │ 8.09    │ 0.88915 │ low    │ other  │
│ 5   │ female │ other     │ 40.48   │ no       │ no       │ no     │ yes    │ 5.6     │ 8.09    │ 0.88915 │ low    │ other  │

## It seems we want [:gender, :ethinicity, :fcollege, :mcollege, :home, :urban, :income, :region] to be treated as categorical columns
## Since they have no natural ordering we apply the following fix.

julia> X = coerce(X, Textual => Multiclass);

## But this still dosen't match what XGBoostClassifier requires?
## We can solving by OneHot encoding the categorical columns before applying XGBoostClassifier
## This can be done togther by building a linear Pipeline as thus
## Note only add prediction_type=:probabilistic for probabilistic responses which XGBoostClassfier is 
julia> pipe = @pipeline MyPipe(hot=OneHotEncoder(),
                       xgc = xgc) prediction_type=:probabilistic

#Let's wrap all this in a tuning model
julia> r1 = range(pipe, :(xgc.max_depth), lower = 3, upper = 10)

julia> r2 = range(pipe, :(xgc.num_round), lower = 1, upper = 25)

julia> tmr = TunedModel(
                   model = pipe,
                   tuning = Grid(resolution = 7),
                   resampling = CV(rng = 11),
                   ranges = [r1, r2],
                   measure = cross_entropy #don't use rms for probabilistic responses
               )

## Finally put these in a machine and fit the machine

julia> mtmr = machine(tmr, X, y)

julia> fit!(mtmr)

I hope this is helpful

[drcxcruz]

Impressive! I will take your Julia code and "expand" it consider any kind of X. That is, my goal is to run the booster for any kind of data set the user provides to the algorithm. The education data set was only one example.

THANK YOU!

[OkonSamuel]

@drcxcruz. I'll be closing this now.