Mlj.jl: Not loading to all workers: @everywhere @load RandomForestClassifier pkg = DecisionTree

[drcxcruz]

Describe the bug
It appears that
@everywhere @load RandomForestClassifier pkg = DecisionTree

does not load the MLJ package in all the workers.

To Reproduce

using Distributed:  @everywhere, addprocs, @spawnat
addprocs(4)  # get ready for parallelism
@everywhere using MLJ

@everywhere @load RandomForestClassifier  pkg = DecisionTree


 pipeRandomForestClassifier = @pipeline RandomForestClassifierPipe(
                selector = FeatureSelector(),
                hot = OneHotEncoder(),
                tree = RandomForestClassifier()) prediction_type = :probabilistic


ERROR: LoadError: LoadError: UndefVarError: RandomForestClassifier not defined


# If only do 
@load RandomForestClassifier  pkg = DecisionTree

# I do not get the error above but I get the error below

@everywhere function train_ResultXY!(tmRandomForestClassifier, rfResultXY, Xy, E)
    rfResultXY = machine(tmRandomForestClassifier, Xy, E)
    fit!(rfResultXY)
    ŷ_rfResultXY = MLJ.predict(rfResultXY, Xy)
    ŷMode_rfResultXY = [mode(ŷ_rfResultXY[i]) for i in 1:length(ŷ_rfResultXY)]
    return ((ŷMode_rfResultXY = ŷMode_rfResultXY, rfResultXY = rfResultXY))
end

BrefXY = @spawnat :any train_ResultXY!(tmRandomForestClassifier,  rfResultXY, Xy, E)

fXY = fetch(BrefXY)

ERROR: LoadError: On worker 3:
UndefVarError: RandomForestClassifierPipe not defined
deserialize_datatype at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:1211
handle_deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:788
deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:735
deserialize_datatype at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:1235
handle_deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:788
deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:735
deserialize_datatype at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:1235
handle_deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:790
deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:735
handle_deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:795
deserialize at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Serialization\src\Serialization.jl:735 [inlined]
deserialize_msg at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\messages.jl:99
#invokelatest#1 at .\essentials.jl:712 [inlined]
invokelatest at .\essentials.jl:711 [inlined]
message_handler_loop at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\process_messages.jl:185
process_tcp_streams at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\process_messages.jl:142
#97 at .\task.jl:358
Stacktrace:
 [1] #remotecall_fetch#141 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\remotecall.jl:390 [inlined]
 [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\remotecall.jl:382
 [3] #remotecall_fetch#144 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\remotecall.jl:417 [inlined]
 [4] remotecall_fetch at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\remotecall.jl:417 [inlined]
 [5] call_on_owner at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\remotecall.jl:490 [inlined]
 [6] fetch(::Distributed.Future) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.4\Distributed\src\remotecall.jl:529
 [7] InvariantEnvironmentPrediction(::DataFrame, ::Array{Float64,1}, ::DataFrame, ::Int64, ::Int64, ::Float64, ::Bool) at C:\Users\BCP\github\ICP\InvariantEnvironmentPrediction.jl:73

Expected behavior
Run random forest in multiple workers in parallel.

Additional context

Versions
(@v1.4) pkg> status
Status C:\Users\BCP\.julia\environments\v1.4\Project.toml
[336ed68f] CSV v0.6.1
[324d7699] CategoricalArrays v0.7.7
[861a8166] Combinatorics v1.0.1
[d58978e5] Dagger v0.8.0
[a93c6f00] DataFrames v0.20.2
[7806a523] DecisionTree v0.10.1
[31c24e10] Distributions v0.23.2
[38e38edf] GLM v1.3.9
[09f84164] HypothesisTests v0.10.0
[7073ff75] IJulia v1.21.2
[add582a8] MLJ v0.11.2
[a7f614a8] MLJBase v0.13.5
[e80e1ace] MLJModelInterface v0.2.6
[d491faf4] MLJModels v0.9.10
[e1d29d7a] Missings v0.4.3
[54e16d92] PrettyPrinting v0.2.0
[612083be] Queryverse v0.5.0
[6f49c342] RCall v0.13.6
[2913bbd2] StatsBase v0.33.0
[bd369af6] Tables v1.0.4
[009559a3] XGBoost v0.4.3

[ablaom]

The first problem may be this: Each worker has its own environment. You may have DecisionTree.jl in the master environment but you need activate an appropriate environment on every worker to give it access to that package (which @load will attempt to load with a import DecisionTree command). Have you done this?

There are a couple of ways to do this. One is use the --project option when you run julia itself. See https://docs.julialang.org/en/v1/manual/getting-started/ for details. To do it from the REPL, see this tutorial: https://github.com/mbauman/ParallelWorkshop2019/blob/master/070%20Distributed.jl

[ablaom]

Re the second problem, it would be good to have a complete minimum working example, including (dummy) data.

[drcxcruz]

hi,

thank you!

I did not try the import yet. i will do so, thanks. my goal is to create a utility that is easily executed from the RELP or another julia program. I would have prefer not to use --project option, if possible.

Here is my module that is trying to run random forest in parallel since each training is taking a few seconds to complete. I have to train 100s of random forests :) in the algorithm depending of user input.

abloam.zip

function setScientificTypes!(X::DataFrame)
for c in 1:ncol(X)
if unique(X[:,c])[1] === missing
println("setScientificTypes: Column :", names(X)[c], " has missing values. They are not allowed.")
return (missing)
end
# remove the missing type that MLJ modeling does not like
if typeof(X[:,c]) == Array{Union{Missing,Float64},1}
X[!,c] = Float64.(X[!,c])
end
if typeof(X[:,c]) == Array{Union{Missing,Int64},1}
X[!,c] = Int64.(X[!,c])
end
if typeof(X[:,c]) == Array{Union{Missing,String},1}
X[!,c] = String.(X[!,c])
end
end

coerce!(X, autotype(X, :discrete_to_continuous))
coerce!(X, autotype(X, :string_to_multiclass))
return (X)

end # setScientificTypes!

[ablaom]

I'm very sorry, but without a small minimum example exposing your problem , it will be some time before I can address your issue. Or are you not sure you still have a bug?

[ablaom]

Ideally provide:

• small self-contained code that fails
• the complete error message and trace
• the output of running versioninfo()
• the output of running using Pkg; Pkg.status()

Thanks

[drcxcruz]

julia> versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: AMD Ryzen 5 PRO 3400GE w/ Radeon Vega Graphics
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, znver1)
Environment:
JULIA_NUM_THREADS = 4

julia> using Pkg; Pkg.status()
Status C:\Users\BCP\.julia\environments\v1.4\Project.toml
[336ed68f] CSV v0.6.1
[324d7699] CategoricalArrays v0.7.7
[861a8166] Combinatorics v1.0.1
[d58978e5] Dagger v0.8.0
[a93c6f00] DataFrames v0.20.2
[7806a523] DecisionTree v0.10.1
[31c24e10] Distributions v0.23.2
[38e38edf] GLM v1.3.9
[09f84164] HypothesisTests v0.10.0
[7073ff75] IJulia v1.21.2
[add582a8] MLJ v0.11.2
[a7f614a8] MLJBase v0.13.5
[e80e1ace] MLJModelInterface v0.2.6
[d491faf4] MLJModels v0.9.10
[e1d29d7a] Missings v0.4.3
[54e16d92] PrettyPrinting v0.2.0
[612083be] Queryverse v0.5.0
[2913bbd2] StatsBase v0.33.0
[bd369af6] Tables v1.0.4
[009559a3] XGBoost v0.4.3

[drcxcruz]

hi,

Here is the minimal code. There are NO errors in the code below. it appears that @spawnat is working as expected. However, there is an error later in the fetch.

using Distributed: @everywhere, addprocs, @spawnat
addprocs(4) # get ready for parallelism
@everywhere using MLJ
@load RandomForestClassifier pkg = DecisionTree

@everywhere function fitter(tmRandomForestClassifier, X, y)
mtm = machine(tmRandomForestClassifier, X, y)
fit!(mtm)
ŷ = MLJ.predict(mtm, X)
return (ŷ)
end

#

using CSV, DataFrames
cd("C:\Users\BCP\github\ICP\Test")

X = CSV.File("treeX1.csv") |> DataFrame
Y = CSV.File("treeY1.csv") |> DataFrame

Y = Y[:,1] |> categorical

pipeRandomForestClassifier = @pipeline RandomForestClassifierPipe(
selector = FeatureSelector(),
hot = OneHotEncoder(),
tree = RandomForestClassifier()) prediction_type = :probabilistic

cases = [[Symbol(names(X)[j]) for j in 1:i] for i in 1:ncol(X)]
r1 = range(pipeRandomForestClassifier, :(selector.features), values=cases)

tmRandomForestClassifier = TunedModel(
model = pipeRandomForestClassifier,
range=r1,
measures=[cross_entropy, BrierScore()],
resampling=CV(nfolds=9)
)

julia> spawnref = @spawnat :any fitter(tmRandomForestClassifier, X, Y)
Distributed.Future(2, 1, 46, nothing)

[drcxcruz]

Now, let us do the fetch which is generating the error.

julia> result = fetch(spawnref)
ERROR: On worker 2:
UndefVarError: RandomForestClassifierPipe not defined
deserialize_datatype at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:1211
handle_deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:788
deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:735deserialize_datatype at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:1235
handle_deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:788
deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:735handle_deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:803
deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:735 [inlined]
deserialize_global_from_main at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\clusterserialize.jl:160

3 at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\clusterserialize.jl:72 [inlined]

foreach at .abstractarray.jl:1919
deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\clusterserialize.jl:72handle_deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:878
deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:735handle_deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:792
deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:735handle_deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:795
deserialize at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Serialization\src\Serialization.jl:735 [inlined]
deserialize_msg at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\messages.jl:99

invokelatest#1 at .\essentials.jl:712 [inlined]

invokelatest at .\essentials.jl:711 [inlined]
message_handler_loop at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\process_messages.jl:185
process_tcp_streams at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\process_messages.jl:142

97 at .\task.jl:358

Stacktrace:
[1] #remotecall_fetch#141 at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\remotecall.jl:390 [inlined]
[2] remotecall_fetch(::Function, ::Distributed.Worker, ::Distributed.RRID) at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\remotecall.jl:382
[3] #remotecall_fetch#144 at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\remotecall.jl:417 [inlined]
[4] remotecall_fetch at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\remotecall.jl:417 [inlined]
[5] call_on_owner at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\remotecall.jl:490 [inlined]
[6] fetch(::Distributed.Future) at D:buildbot\worker\package_win64build\usr\sharejulia\stdlib\v1.4\Distributed\src\remotecall.jl:529

[drcxcruz]

Problem 1:
You mentioned that
@load RandomForestClassifier pkg = DecisionTree

is not enough and I need to also do
import DecisionTree

However, import DecisionTree does not define RandomForestClassifier so an error still occurs. I do not see how to combine import and @load to define RandomForestClassifier in all the workers. what is the syntax to do so?

Problem 2: How do we define the pipe in all the workers? The syntax relating to @pipeline does not seem to support token @everywhere. I tried placing @everywhere in the pipeline creation but I got syntax error.

pipeRandomForestClassifier = _@everywhere_ @pipeline RandomForestClassifierPipe(
selector = FeatureSelector(),
hot = OneHotEncoder(),
tree = RandomForestClassifier()) prediction_type = :probabilistic

thanks for your time.

[ablaom]

X = CSV.File("treeX1.csv") |> DataFrame
Y = CSV.File("treeY1.csv") |> DataFrame

I don't have these files. Can you substitute similar data constructed in your code?

[ablaom]

is not enough and I need to also do
import DecisionTree

No, you misunderstand. You don't need to do import DecisionTree. The @load call does that for you. But that won't work unless DecisionTree is in the active environment of the worker making the @load call. You can check this by sending a Pkg.status() to each worker.

[drcxcruz]

hi

OK, how do we make DecisionTree active environment of the worker? :) Enclosed are the data files. thank you

abloam.zip

[OkonSamuel]

@drcxcruz are you running the code on your PC or on a cluster. Please check that you have DecisionTree installed in julia

[drcxcruz]

hi,

I am running in my Lenovo Think Centre desktop. I ran the code successfully many times without any parallelism. However, the parallelism is not working. I am running the code in Microsoft Visual Studio with the Julia extension.

Yes, I have the packages in my Julia. As stated before, below are the versions of my Julia and Julia packages. Thank you for your time.

julia> versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: AMD Ryzen 5 PRO 3400GE w/ Radeon Vega Graphics
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-8.0.1 (ORCJIT, znver1)
Environment:
JULIA_NUM_THREADS = 4

julia> using Pkg; Pkg.status()
Status C:\Users\BCP.julia\environments\v1.4\Project.toml
[336ed68f] CSV v0.6.1
[324d7699] CategoricalArrays v0.7.7
[861a8166] Combinatorics v1.0.1
[d58978e5] Dagger v0.8.0
[a93c6f00] DataFrames v0.20.2
[7806a523] DecisionTree v0.10.1
[31c24e10] Distributions v0.23.2
[38e38edf] GLM v1.3.9
[09f84164] HypothesisTests v0.10.0
[7073ff75] IJulia v1.21.2
[add582a8] MLJ v0.11.2
[a7f614a8] MLJBase v0.13.5
[e80e1ace] MLJModelInterface v0.2.6
[d491faf4] MLJModels v0.9.10
[e1d29d7a] Missings v0.4.3
[54e16d92] PrettyPrinting v0.2.0
[612083be] Queryverse v0.5.0
[2913bbd2] StatsBase v0.33.0
[bd369af6] Tables v1.0.4
[009559a3] XGBoost v0.4.3

[OkonSamuel]

Try this

julia> using Distributed
julia> addprocs(2)
julia> @everywhere using MLJ
julia> @everywhere forest_model = MLJ.@load RandomForestClassifier pkg=DecisionTree
julia> @everywhere pipeRandomForestClassifier = @pipeline RandomForestClassifierPipe(
                       selector = FeatureSelector(),
                       hot = OneHotEncoder(),
                       tree = RandomForestClassifier()) prediction_type = :probabilistic

The rest of your code should just work

[drcxcruz]

hi

Impressive work you do like the last time you helped me out!

I have made the change as you suggested. It is working now using native Julia REPL and not MS Visual Studio. I will do more testing in native Julia REPL. thank you

Let me ask you a related question. it appears that @spawnat does not work from MS Visual Studio. I would imagine that @spawnat works fine from JuliaPro. I used to code in JuliaPro but it created some package conflicts that did not allow me to update MLJ packages to newest versions. Thus, I un-installed JuliaPro and started using MS visual studio. Now, I am able to update MLJ packages but the parallelism does not work from Visual Studio. What editor do you recommend? if you use JuliaPro, how did you solve the package conflicts in order to update MLJ packages?

Again, thank you guys for all your help. As a present, I did the following Jupyter lab showcasing MLJ: https://notes.quantecon.org/submission/5e653f616479ca001038be0f

[OkonSamuel]

I mainly use the Julia REPL although sometimes i use Visual studio Code (Although i haven't used it in awhile) or Jupyter notebooks. Maybe you should try updating/ reinstalling your VSCode.

As a present, I did the following Jupyter lab showcasing MLJ: https://notes.quantecon.org/submission/5e653f616479ca001038be0f

Cool.

PS: Sometimes julia language support for VSCode doesn't fully install due to Internet connection issues and may cause it not to behave as expected. Make sure you properly installed julia language support 0.15.32 or above from VSCode market place. If that doesn't still work i guess you have to open an issue at https://github.com/julia-vscode/julia-vscode

[OkonSamuel]

As a present, I did the following Jupyter lab showcasing MLJ: https://notes.quantecon.org/submission/5e653f616479ca001038be0f

@tlienart. Is there any possibility of adding a link to this in MLJTutorials ??

[tlienart]

Yes sounds good, also we have a new guy starting today who will help out with the tutorials so hopefully it'll be a bit faster on that front given I'm a big bottleneck...

[OkonSamuel]

I'll be closing this now.