Climatemachine.jl: Windows tests failing ; Linux, MacOS tests passing
Description
Tests are failing on Windows platforms, but passing on Linux and MacOS systems. This can be seen on multiple PRs, e.g. #932 and #924 . Is this a known problem ?
cc @bischtob @simonbyrne @charleskawczynski @kpamnany
For #932, the
bors try output
Azure Pipelines / CLIMA
succeeded 10 minutes ago in 2h 11m 7s
https://github.com/climate-machine/CLIMA/runs/565320767
Azure pipeline results
CLIMA (Windows Julia 1.3) Failing after 88m — Windows Julia 1.3 failed
https://github.com/climate-machine/CLIMA/pull/932/checks?check_run_id=565340992
The test summary suggests failure in the MPI Jobs category:
Test Summary: | Pass Error Total
MPI Jobs | 12 1 13
For CLIMA Developers
- [ ] This feature can be added (if it cannot be, explain why in a comment below -- lack of technical expertise, not relevant to the scope of this project, too ambitious)
- [ ] There is a timeline for when this feature can be implemented
- [ ] The feature has been (or will be) implemented (Please link the PR)
akshaysridhar
All 14 comments
The test that is failing appears to be:
https://github.com/climate-machine/CLIMA/blob/518e242cb87131d45c1bdfb183b8d8295407280b/test/Mesh/runtests.jl#L24
The full error is:
┌ Info: Running MPI test...
45.300055 seconds (48 allocations: 1.891 KiB)
│ n = 5
│ f = "mpi_connect_1d.jl"
â”” cmd = `mpiexec -n 5 'C:\julia-1.3\bin\julia.exe' -Cnative '-JC:\julia-1.3\lib\julia\sys.dll' --check-bounds=yes -g1 --startup-file=no '--project=C:\Users\VSSADM~1\AppData\Local\Temp\jl_xCUw8B\Project.toml' --code-coverage=none 'D:\a\1\s\test\Mesh\mpi_connect_1d.jl'`
fatal: error thrown and no exception handler available.
fatal: error thrown and no exception handler available.
jlbacktrace at C:\julia-1.3\bin\libjulia.dll (unknown line)
jlbacktrace at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_gc_pool_alloc at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_gc_pool_alloc at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_new_uninitialized_datatype at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_new_uninitialized_datatype at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_compute_fieldtypes at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_compute_fieldtypes at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_inst_concrete_tupletype_v at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_inst_concrete_tupletype_v at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_f_tuple at C:\julia-1.3\bin\libjulia.dll (unknown line)
jl_f_tuple at C:\julia-1.3\bin\libjulia.dll (unknown line)
unknown function (ip: 00000000059407F4)
unknown function (ip: 00000000052AEBA0)
unknown function (ip: 00000000052AEE9E)
jl_apply_generic at C:\julia-1.3\bin\libjulia.dll (unknown line)
unknown function (ip: 00000000061A07F4)
unknown function (ip: 0000000005B0EBA0)
jl_apply_generic at C:\julia-1.3\bin\libjulia.dll (unknown line)
Aborting: mpiexec on fv-az379 failed to communicate with smpd manager on fv-az379
MPI Jobs: Error During Test at D:\a\1\s\test\testhelpers.jl:34
Test threw exception
Expression: begin
run(cmd)
true
end
failed process: Process(`mpiexec -n 5 'C:\julia-1.3\bin\julia.exe' -Cnative '-JC:\julia-1.3\lib\julia\sys.dll' --check-bounds=yes -g1 --startup-file=no '--project=C:\Users\VSSADM~1\AppData\Local\Temp\jl_xCUw8B\Project.toml' --code-coverage=none 'D:\a\1\s\test\Mesh\mpi_connect_1d.jl'`, ProcessExited(4294967295)) [4294967295]
Stacktrace:
[1] pipeline_error at .\process.jl:525 [inlined]
[2] #run#565(::Bool, ::typeof(run), ::Cmd) at .\process.jl:440
[3] run at .\process.jl:438 [inlined]
[4] macro expansion at .\util.jl:155 [inlined]
[5] runmpi(::Array{Tuple{Int64,String},1}, ::String) at D:\a\1\s\test\testhelpers.jl:34
[6] top-level scope at D:\a\1\s\test\Mesh\runtests.jl:37
[7] top-level scope at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.3\Test\src\Test.jl:1107
[8] top-level scope at D:\a\1\s\test\Mesh\runtests.jl:23
59.754717 seconds (2.33 M allocations: 114.105 MiB, 0.23% gc time)
simonbyrne
on 7 Apr 2020
I have seen similar errors in #855. Could it be something to do with updating MPI in #933?
lcw
on 7 Apr 2020
Should we revert #933? It seems that master is flaky, and this will slow down PR merges if this is the case.
charleskawczynski
on 7 Apr 2020
Now #855 is passing on Windows. I just kept rerunning the windows CI until it passed. It depends on #933. Should I bors r+ #855?
lcw
on 7 Apr 2020
I don't see any reason to wait on merging that.
kpamnany
on 7 Apr 2020
I did see the failure once on Saturday before #933 was merged: see Attempt 1 here:
which was run on https://github.com/climate-machine/CLIMA/commit/9677227cbc53f6da77acb9939c8269d0cc2283d1
simonbyrne
on 7 Apr 2020
Is there any point retrying the Windows runs before #942 is merged in?
blallen
on 7 Apr 2020
CLIMA (Windows Julia 1.3) Failing after 93m — Windows Julia 1.3 failed
942 appears to have failed the windows check as well
akshaysridhar
on 7 Apr 2020
I ran master on my local Windows machine and all the Mesh tests pass. Also, looking at the logs, the exact test that's failing seems to vary. Here two of the Mesh tests failed, reporting an internal error (in MPI?).
Anyone have any ideas?
kpamnany
on 8 Apr 2020
On the linux test --
2020-04-08T06:29:23.6956413Z Err:1 http://security.ubuntu.com/ubuntu xenial-security/main amd64 libgc1c2 amd64 1:7.4.2-7.3ubuntu0.1
2020-04-08T06:29:23.6956955Z 503 Service Unavailable
2020-04-08T06:29:23.8923043Z Get:1 http://security.ubuntu.com/ubuntu xenial-security/main amd64 libgc1c2 amd64 1:7.4.2-7.3ubuntu0.1 [82.1 kB]
2020-04-08T06:29:23.8977628Z Err:2 http://azure.archive.ubuntu.com/ubuntu xenial/main amd64 gfortran amd64 4:5.3.1-1ubuntu1
2020-04-08T06:29:23.8978799Z 503 Service Unavailable
2020-04-08T06:29:23.9020010Z Get:3 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu xenial/main amd64 libobjc4 amd64 9.2.1-17ubuntu1~16.04 [48.8 kB]
I did see the failure once on Saturday before #933 was merged... which was run on 9677227
Maybe it was updating LazyArrays then? Should we try reverting #929? Or both #929 and #933 in case it's an interaction. I'll start a branch with these reverted to see.
charleskawczynski
on 8 Apr 2020
Lets just allow failures on Windows until someone has time to track this down.
simonbyrne
on 8 Apr 2020
How can we tell bors to avoid the windows build? Right now we specify
status = [
"CLIMA",
"ci/slurmci"
]
Also, it's only one data point but, #944 is passing on Windows
charleskawczynski
on 8 Apr 2020
Related issues
jkozdon
·
9Comments
glwagner
·
3Comments
bischtob
·
4Comments
jkozdon
·
8Comments
smarras79
·
5Comments
Most helpful comment
Lets just allow failures on Windows until someone has time to track this down.