Mlj.jl: Programmatic creation of pipelines

[CameronBieganek]

Consider the following code:

julia> using MLJ

julia> function make_pipeline(model, name::AbstractString; drop_last=false)
           @pipeline(
               OneHotEncoder(drop_last=drop_last),
               model,
               name=name
           )
       end
ERROR: LoadError: UndefVarError: drop_last not defined
Stacktrace:
 [1] top-level scope at REPL[5]:1
 [2] eval at ./boot.jl:331 [inlined]
 [3] eval(::Expr) at ./client.jl:449
 [4] pipeline_preprocess(::Module, ::Expr, ::Vararg{Any,N} where N) at /Users/bieganek/.julia/packages/MLJBase/2UxSl/src/composition/models/pipelines.jl:310
 [5] pipeline_(::Module, ::Expr, ::Vararg{Any,N} where N) at /Users/bieganek/.julia/packages/MLJBase/2UxSl/src/composition/models/pipelines.jl:439
 [6] @pipeline(::LineNumberNode, ::Module, ::Vararg{Any,N} where N) at /Users/bieganek/.julia/packages/MLJBase/2UxSl/src/composition/models/pipelines.jl:562
in expression starting at REPL[5]:2

I'm not sure precisely what's going on here, but it looks like @pipeline was only designed to be called at the top-level, which makes it more laborious to create a bunch of related pipelines that use different supervised ML models for the target prediction.

I'm not sure how feasible this is, but it would be nice if there were a simple interface for creating pipelines without using a macro (in the spirit of this Discourse comment).

Version Info:

(@v1.4) pkg> status MLJ MLJBase MLJModels MLJLinearModels
Status `~/.julia/environments/v1.4/Project.toml`
  [add582a8] MLJ v0.12.0
  [a7f614a8] MLJBase v0.14.2
  [6ee0df7b] MLJLinearModels v0.5.0
  [d491faf4] MLJModels v0.11.0

julia> versioninfo()
Julia Version 1.4.0
Commit b8e9a9ecc6 (2020-03-21 16:36 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = vim

[ablaom]

Yes, unfortunately, the arguments of @pipeline get evaluated in top-level scope of the calling module, which limits its use 😞 .

A non-macro solution is difficult to imagine here because a new struct is being defined. I thought about defining a "universal" composite model type whose single field is a dictionary of component models (with a type parameter that is different for each new "architecture") but this seemed unnatural and I did not pursue it. For the special case of pipelines, it might be worth revisiting, I suppose.

Note that for your particular use-case, the pipeline's fields are mutable. So you can do things like this:

pipe = @pipeline OneHotEncoder RidgeRegressor 
for rgs in [`RidgeRegressor`, `KNNRegressor`, ...]
    pipe.ridge_regressor = rgs
    evaluate(pipe, X, y, measure=rms)
end

[CameronBieganek]

pipe = @pipeline OneHotEncoder RidgeRegressor 
for rgs in [`RidgeRegressor`, `KNNRegressor`, ...]
    pipe.ridge_regressor = rgs
    evaluate(pipe, X, y, measure=rms)
end

That's not a bad solution! Though it has the odd side effect that my random forest model lives in the ridge_regressor field of the pipeline.

I thought about defining a "universal" composite model type whose single field is a dictionary of component models (with a type parameter that is different for each new "architecture") but this seemed unnatural and I did not pursue it.

Just thinking out loud here... It seems like this is a case where a little type instability is not a big deal. So we could have a "universal" composite type called Pipeline with non-concrete fields that looks something like this:

struct Pipeline
    steps::AbstractVector
end

Pipeline(args...) = Pipeline([args...])

pipe = Pipeline(
    OneHotEncoder(),
    RandomForestClassifier()
)

[DilumAluthge]

This would be type-stable:

struct Pipeline{T <: Tuple}
    steps::T
end

[ablaom]

That's not a bad solution! Though it has the odd side effect that my random forest model lives in the ridge_regressor field of the pipeline.

Yeah. I originally thought of generating names like model1, model2, and so forth. But I reckon that for the main-case the status-quo makes inspecting fitted_params and reports easier.

And yes, your suggestion is more-or-less what I had in mind. But how, for example, do you access the nested objects? A lot currently depends on the getproperty access. Maybe if steps is a named tuple? Then you could do pipe.steps.classifier (but pipe.classifier is nicer)??

[CameronBieganek]

Yeah, maybe it would make sense for steps to be a named tuple. And then maybe getproperty could be overloaded?

getproperty(p::Pipeline, s::Symbol) = getproperty(p.steps, s)

And to build on @DilumAluthge's suggestion, perhaps the struct definition could be

struct Pipeline{T <: NamedTuple}
    steps::T
end

However, I don't claim to have an understanding of the broader implications...

[ablaom]

Good idea about the overloading!

BTW I should have said that you can always roll your own composites using the learning network syntax. Using @from_network you could use whatever names for the fields you like. Or, to do what you want without macros in for your use case:

mutable struct OneHotWrapper{P} <: ProbabilisticComposite
    probabilistic_model::P
    drop_last::Bool
end

function MLJ.fit(wrapper::OneHotWrapper, verbosity, X, y)
    Xs = source(X)
    ys = source(y)

    hot = OneHotEncoder(drop_last=wrapper.drop_last)
    mach1 = machine(hot, Xs)
    W = transform(mach1, Xs)

    mach2 = machine(wrapper.probabilistic_model, W, ys)
    yhat = predict(mach2, W)

    mach = machine(Probabilistic(), Xs, ys, predict=yhat)
    fit!(mach, verbosity=verbosity - 1)
    return mach()
end


@load DecisionTreeClassifier
wrapper = OneHotWrapper(DecisionTreeClassifier(), true)

X, y = @load_iris;

# add a categorical feature:
bogus = coerce(rand(["blue", "red"], length(y)), Multiclass);
X = merge(X, (bogus=bogus,))

julia> evaluate(wrapper, X, y, measure=cross_entropy)
Evaluating over 6 folds: 100%[=========================] Time: 0:00:00
┌───────────────┬───────────────┬───────────────────────────────────────────── ⋯
│ _.measure     │ _.measurement │ _.per_fold                                   ⋯
├───────────────┼───────────────┼───────────────────────────────────────────── ⋯
│ cross_entropy │ 3.12          │ [2.22e-16, 2.22e-16, 4.33, 2.88, 4.33, 7.21] ⋯
└───────────────┴───────────────┴───────────────────────────────────────────── ⋯

editted to add a categorical feature to the demo data

[CameronBieganek]

Ok, good to know! I haven't played around with the learning network stuff yet. Though of course the appeal of syntax like

pipe = Pipeline(
    OneHotEncoder(),
    RandomForestClassifier()
)

is that you get to avoid the "boiler plate" calls to machine, transform, and predict.

[CameronBieganek]

@ablaom If you don't mind, I'd like to play around with this. I will need to do some experimentation, of course, and then I can put together an RFC pull request.

cc-ing @ExpandingMan since he's commented about @from_network and @pipeline.

[ablaom]

Happy of course for you to have a go! And I believe you are well-equipped to make a valuable contribution here, if you have the time.

I should warn you there is quite a lot of functionality built into the current @pipeline and my own priorities are currently in other areas. I think a POC would be a fantastic start, and I am happy to provide guidance. However, I should say I may not have the resources to move a POC to a complete solution without a lot of further assistance.

Let me know if you'd like to zoom at some point in the process. This can sometimes save a lot of time.

[CameronBieganek]

Ok, sounds good!

[CameronBieganek]

@ablaom Quick question. According to the @pipeline docstring,

At most one of the models may be a supervised model, but this model can appear in any position.

Off the top of my head, I can't think of a use-case for not having the supervised model as the last model in the pipeline. (Not counting target transformations.) Can you give me an example where the supervised model is not the last?

[ExpandingMan]

If you consider a transformation of the output (i.e. an inverse transformation of something that was applied at the input) to be a model (really it's just the application of a model), then this would be a very common case in which this is needed.

[CameronBieganek]

Sure, but that's already handled explicitly in @pipeline by the target and inverse keyword arguments. So, I'm looking for an example where the supervised model is not the last model in the sequence of arguments supplied in the vararg positional arguments to @pipeline. 🙂

[ablaom]

To clarify, when specifying a target tranformation this implies two transformations - one before passing target to supervised model for training - and a corresponding inverse transformation, applied to the output of predict of the supervised model. (If you specify target=t and t isa Unsupervised then it is tacitly assumed that the model supports both a transform and inverse_transform method (or error is thrown). If target=f and f isa Function then user is obliged to provide inverse=g as well.) So, if I want to apply a transformation at the end of the pipeline, that is not the inverse of a transform I'm happy to apply at the beginning, then I need to specify that as part of the pipeline and not with inverse=.... (Indeed, you are disallowed from specifying inverse=... without target=.... ) For example, I may want to compute the mode of the probabilisitic predictions of my supervised model.

[CameronBieganek]

For example, I may want to compute the mode of the probabilisitic predictions of my supervised model.

Yeah, that's a good example. However, that example is already covered by setting the keyword argument operation to operation=predict_mode. 😄

Can you think of another example? I'm just wondering what use cases we're trying to support by allowing the supervised model to not be the last model in the varargs.

[ablaom]

Yeah, well, suppose you want a weighted mode, that is, lower threshold for certain classes. So you want to add a custom function to the end.

[CameronBieganek]

I had a little time over the U.S. Thanksgiving holiday to look at this, but since then I've been busy. However, I'm definitely still planning to work on this. There's a bit of work involved, but I have a pretty clear idea of what to do. Here's the skeleton of my plan:

Define pipeline types and constructor

struct DeterministicPipeline{N <: NamedTuple} <: DeterministicComposite
    steps::N
end

struct ProbabilisticPipeline{N <: NamedTuple} <: ProbabilisticComposite
    steps::N
end

struct UnsupervisedPipeline{N <: NamedTuple} <: UnsupervisedComposite
    steps::N
end

# Overload getproperty so the steps of a pipeline can be accessed like `pipe.random_forest`.
Base.getproperty(...) = #  ...

function Pipeline(...)
    # ...
end

Pipeline will take the place of @pipeline and construct one of the pipeline types, depending on the input arguments. The argument checking that currently occurs in @pipeline will occur in the Pipeline constructor.

Add fit methods

function MLJBase.fit(...)
    # Re-use the current `linear_learning_network_machine()` function,
    # which is already well tested.
end

[ablaom]

@CameronBieganek Many thanks for delving into this challenging part of the code base!

One concern I have about this is how the new pipelines object are constructed. Currently, we have a show that generates something like this:

julia> pipe = @pipeline Standardizer DecisionTreeRegressor name = MyPipe
MyPipe(
    standardizer = Standardizer(
            features = Symbol[],
            ignore = false,
            ordered_factor = false,
            count = false),
    decision_tree_regressor = DecisionTreeRegressor(
            max_depth = -1,
            min_samples_leaf = 5,
            min_samples_split = 2,
            min_purity_increase = 0.0,
            n_subfeatures = 0,
            post_prune = false,
            merge_purity_threshold = 1.0)) @934

This presents the pipeline object exactly as it can be constructed, and is also suggestive of how nested parameters can be accessed. However, without macro magic, your constructor going to look a little different. Your proposal to overload getproperty allows the user to access nested hyperparameters as if the component models were fields of the composite, but it does not allow the user to construct them that way, which creates some cognitive dissonance, no?

[CameronBieganek]

My plan is to have a few different signatures for the Pipeline constructor. One of them will be essentially the same as the current @pipeline macro, e.g.

pipe = Pipeline(
    X -> coerce(X, :age => Continuous),
    OneHotEncoder(drop_last=true),
    RandomForestRegressor(n_trees=100);
    target = v -> log.(v),
    inverse = v -> exp.(v)
)

Like the current @pipeline, the above call will auto-generate the names for the steps in the pipeline. However, I would like to add two other methods that allow the user to specify their own names for the steps in the pipeline. I think this would be the signatures for those two methods:

Pipeline(args::Pair...; kwargs...) = # ...
Pipeline(args::Tuple...; kwargs...) = # ...

Then the original pipeline above could be written either as this:

pipe = Pipeline(
    :foo => X -> coerce(X, :age => Continuous),
    :bar => OneHotEncoder(drop_last=true),
    :baz => RandomForestRegressor(n_trees=100);
    target = v -> log.(v),
    inverse = v -> exp.(v)
)

or as this:

pipe = Pipeline(
    (:foo, X -> coerce(X, :age => Continuous)),
    (:bar, OneHotEncoder(drop_last=true)),
    (:baz, RandomForestRegressor(n_trees=100));
    target = v -> log.(v),
    inverse = v -> exp.(v)
)

We could maybe allow strings in addition to symbols when declaring step names, but that could lead to issues with creating the named tuple if the strings are not valid identifiers. I wanted to use keyword arguments for defining the step names, like this:

pipe = Pipeline(
    coerce = X -> coerce(X, :age => Continuous),
    one_hot = OneHotEncoder(),
    random_forest = RandomForestRegressor()
)

but of course that doesn't work so well, since we also need to support the actual keyword arguments like target, inverse, operation, etc.

@ablaom Does that alleviate your concerns, or am I missing something? :)

[ablaom]

Thanks for that. That sounds good but I don't think I was clear about my concern, which I am struggling to articulate succinctly. Perhaps you can help me out by explaining how you propose display a Pipeline instance in the REPL?

Currently all composite models are displayed as they can be constructed (see my previous comment) and this works quite uniformly, no matter how deep the composition. For example:

julia> tree = DecisionTreeRegressor()
DecisionTreeRegressor(
    max_depth = -1,
    min_samples_leaf = 5,
    min_samples_split = 2,
    min_purity_increase = 0.0,
    n_subfeatures = 0,
    post_prune = false,
    merge_purity_threshold = 1.0) @574

julia> forest = EnsembleModel(atom=tree)
DeterministicEnsembleModel(
    atom = DecisionTreeRegressor(
            max_depth = -1,
            min_samples_leaf = 5,
            min_samples_split = 2,
            min_purity_increase = 0.0,
            n_subfeatures = 0,
            post_prune = false,
            merge_purity_threshold = 1.0),
    atomic_weights = Float64[],
    bagging_fraction = 0.8,
    rng = Random._GLOBAL_RNG(),
    n = 100,
    acceleration = CPU1{Nothing}(nothing),
    out_of_bag_measure = Any[]) @557

julia> pipe = @pipeline Standardizer() forest name=MyPipe
MyPipe(
    standardizer = Standardizer(
            features = Symbol[],
            ignore = false,
            ordered_factor = false,
            count = false),
    deterministic_ensemble_model = DeterministicEnsembleModel(
            atom = DecisionTreeRegressor @574,
            atomic_weights = Float64[],
            bagging_fraction = 0.8,
            rng = Random._GLOBAL_RNG(),
            n = 100,
            acceleration = CPU1{Nothing}(nothing),
            out_of_bag_measure = Any[])) @709

julia> show(pipe, 3)
MyPipe(
    standardizer = Standardizer(
            features = Symbol[],
            ignore = false,
            ordered_factor = false,
            count = false),
    deterministic_ensemble_model = DeterministicEnsembleModel(
            atom = DecisionTreeRegressor(
                    max_depth = -1,
                    min_samples_leaf = 5,
                    min_samples_split = 2,
                    min_purity_increase = 0.0,
                    n_subfeatures = 0,
                    post_prune = false,
                    merge_purity_threshold = 1.0),
            atomic_weights = Float64[],
            bagging_fraction = 0.8,
            rng = Random._GLOBAL_RNG(),
            n = 100,
            acceleration = CPU1{Nothing}(nothing),
            out_of_bag_measure = Any[])) @709

It seems that your proposal breaks this possibility. But perhaps all composites should be displayed in a different way - one that suggests how the different levels can be accessed and not how they are constructed??

[CameronBieganek]

Maybe we could print something like this:

julia> Pipeline(("scaler", Standardizer()), ("regressor", RandomForestRegressor())
Pipeline(
    ("scaler", Standardizer(
            features = Symbol[],
            ignore = false,
            ordered_factor = false,
            count = false)),
    ("regressor", RandomForestRegressor(
            max_depth = -1,
            min_samples_leaf = 1,
            min_samples_split = 2,
            min_purity_increase = 0.0,
            n_subfeatures = -1,
            n_trees = 10,
            sampling_fraction = 0.7,
            pdf_smoothing = 0.0)))

I think that's basically how scikit-learn pipelines print, if I recall correctly.

The other option is to provide a vararg keyword argument version of Pipeline, so then we would have

julia> Pipeline(scaler=Standardizer(), regressor=RandomForestRegressor())
Pipeline(
    scaler = Standardizer(
            features = Symbol[],
            ignore = false,
            ordered_factor = false,
            count = false),
    regressor = RandomForestRegressor(
            max_depth = -1,
            min_samples_leaf = 1,
            min_samples_split = 2,
            min_purity_increase = 0.0,
            n_subfeatures = -1,
            n_trees = 10,
            sampling_fraction = 0.7,
            pdf_smoothing = 0.0))

So the printing stays basically the same as the current status quo, and that way the printed output would be a valid constructor call. Of course the downside is that we would have to disallow user-named steps that conflict with target, inverse, invert_last, prediction_type, or operation, but I think that's probably not too big of a deal. Though it makes the interface slightly confusing in that a keyword argument can be either a step in the pipeline or an actual keyword argument...

EDIT: For the first case, this is what would happen if you use Pipeline with auto-generated step names:

julia> Pipeline(Standardizer(), RandomForestRegressor())
Pipeline(
    ("standardizer", Standardizer(
            features = Symbol[],
            ignore = false,
            ordered_factor = false,
            count = false)),
    ("random_forest_regressor", RandomForestRegressor(
            max_depth = -1,
            min_samples_leaf = 1,
            min_samples_split = 2,
            min_purity_increase = 0.0,
            n_subfeatures = -1,
            n_trees = 10,
            sampling_fraction = 0.7,
            pdf_smoothing = 0.0)))

[ablaom]

@CameronBieganek Thanks for the clarification, which is re-assuring. The first kind of display is fine; as you say the second could lead to some confusion. I'm imagining this might be tricky to work this into the existing show methods, but not impossible.

[CameronBieganek]

Since we're discussing Pipeline, I might as well float a related idea. What do you think about making a few changes to enable removing the target, inverse, operation, prediction_type, and invert_last keyword arguments? Then Pipeline could unambiguously take keyword arguments to specify steps instead of tuples or pairs. Let me elaborate:

target and inverse

We could get rid of these by adding a model wrapper like TransformedTargetRegressor. Then a pipeline could look like this:

pipe = Pipeline(
    standardizer = Standardizer(),
    random_forest = TransformedTargetRegressor(
        RandomForestRegressor(),
        target = y -> log.(y),
        inverse = z -> exp.(z)
    )
)

Perhaps that's a little less pretty, but it does allow us to get rid of the target and inverse keyword arguments to Pipeline.

(I stole TransformedTargetRegressor from sklearn.)

prediction_type

I don't have a complete picture of what this argument is used for, but it seems like in most cases the prediction type is inferred from the last model in the pipeline. Perhaps there are cases where a user defines their own model type but they don't define the prediction_type trait?

operation

With regular models, you can often choose which predict method to call on the model; e.g., with a RandomForestClassifier, you can call either predict or predict_mode. I don't know if this always works, but that seems to extend to composite models at least in some cases. For example, this works:

import DataFrames
using MLJ
@load RandomForestClassifier pkg=DecisionTree

X = DataFrames.DataFrame(rand(100, 3))
y = categorical(rand([0, 1], 100))

Xs = source(X)
ys = source(y)

standardizer = machine(Standardizer(), Xs)
W = transform(standardizer, Xs)

rf = machine(RandomForestClassifier(), W, ys)
yhat = predict(rf, W)

mach = machine(Probabilistic(), Xs, ys; predict=yhat)
fit!(mach)
predict(mach, X)
predict_mode(mach, X)

So, I'm not sure to what extent the operation argument is needed for Pipeline. Likely there are use cases that I'm not aware of yet. 🙂

invert_last

Not sure about this one...

[ablaom]

I think breaking out the target transformer as a separate wrapper makes a lot of sense. I have found that some users are unfamiliar with the concept of learned target transformations and get confused by them in the @pipeline context, because it is bundled within another concept - linear pipeline. Also, the business about operation and invert_last addressed in this wrapper; they came about essentially because one cannot immediately apply the target inverse transform to a probabilistic prediction (well, in theory, you could but that would be a mammoth effort to implement) so you either want to call predict_mode/predict_mean/predict_median in that case to get a point prediction, or, follow the supervised model with another transforming element that extracts point predictions and declare invert_last. I now think the use case for "follow the supervised model with another transforming element " is too small to worry about. If one really wants it, one can do a custom learning network (and this is the general philosophy I'm aiming at - pipelines should be simple and intuitive - if you want something special, roll your own with a learning network). We could also just insist prediction_type of a target-transformer-wrapper is always :deterministic.

Assuming we break out the target-transformer-wrapper, we would still need prediction_type=..., unless we insist that the supervised model appears at the end of the line. I agree that for the sake of simplifying the API, we could insist it be at the end; again if you want different, roll your own learning network.

@CameronBieganek Is a target-transformer-wrapper something you could find time to work on?I would call it TransformedTargetModel or similar, for consistency with the other wrappers TunedModel and EnsembleModel. It's constructor might look like TransformedTargetModel(model=..., transformer=..., inverse=..., operation=...) with inverse only specified for non-learned transformations, as in the current pipeline API, no? If model <: Probabilistic then one must specify operation different from predict.

[CameronBieganek]

Sounds good! Yeah, I'd be happy to add a TransformedTargetModel to the PR. Unfortunately, this task has currently slipped a bit on my list of priorities, but hopefully I'll get a chance to work on it one of these days.