Julia: Building from source with system dependencies has numerous failures

Created on 17 Sep 2018  ยท  21Comments  ยท  Source: JuliaLang/julia

Attempting to switch on USE_SYSTEM_* in turn for various components produces the following build or testsuite failures:

pcre and mpfr both fail the stress test:

pcre:

Test                          (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
compiler/validation                (4) |     3.52 |   0.19 |  5.4 |     154.15 |   215.89
strings/search                     (7) |     5.34 |   0.24 |  4.4 |     246.64 |   215.89
strings/io                         (9) |     6.26 |   0.23 |  3.8 |     257.39 |   221.52
unicode/utf8                       (7) |     0.67 |   0.00 |  0.0 |       2.25 |   215.89
      From worker 2:    Skipping Base.Filesystem.JL_O_RANDOM
      From worker 2:    Skipping Base.Filesystem.JL_O_SEQUENTIAL
      From worker 2:    Skipping Base.Filesystem.JL_O_SHORT_LIVED
      From worker 2:    Skipping Base.Filesystem.JL_O_TEMPORARY
      From worker 2:    Skipping Base.active_repl
      From worker 2:    Skipping Base.active_repl_backend
strings/util                       (8) |     9.20 |   0.19 |  2.0 |     309.89 |   222.77
worlds                             (7) |     4.18 |   0.07 |  1.6 |     131.74 |   215.89
strings/types                      (4) |     8.30 |   0.26 |  3.1 |    1183.92 |   216.64
      From worker 2:    Skipping Base.active_repl
      From worker 2:    Skipping Base.active_repl_backend
compiler/compiler                  (3) |    15.84 |   0.37 |  2.3 |     690.13 |   240.75
      From worker 2:    Skipping Base.Filesystem.JL_O_RANDOM
      From worker 2:    Skipping Base.Filesystem.JL_O_SEQUENTIAL
      From worker 2:    Skipping Base.Filesystem.JL_O_SHORT_LIVED
      From worker 2:    Skipping Base.Filesystem.JL_O_TEMPORARY
      From worker 2:    Skipping Base.active_repl
      From worker 2:    Skipping Base.active_repl_backend
keywordargs                        (8) |     8.55 |   0.06 |  0.7 |     111.67 |   225.67
triplequote                        (8) |     0.04 |   0.00 |  0.0 |       1.66 |   225.67
ambiguous                          (2) |    18.73 |   0.40 |  2.1 |    1153.26 |   241.27
intrinsics                         (8) |     0.42 |   0.01 |  1.3 |      26.16 |   225.77
char                               (3) |     3.83 |   0.16 |  4.3 |     227.22 |   254.04
strings/basic                      (6) |    25.26 |   0.40 |  1.6 |     666.98 |   242.88
iobuffer                           (3) |     4.97 |   0.07 |  1.5 |     117.25 |   269.21
staged                             (6) |     6.80 |   0.17 |  2.4 |     266.15 |   244.02
hashing                            (8) |    35.58 |   0.70 |  2.0 |    1475.74 |   261.31
subtype                            (4) |    46.69 |   1.15 |  2.5 |    1797.38 |   694.52
tuple                              (8) |    13.15 |   0.19 |  1.4 |     264.37 |   278.21
dict                               (2) |    59.54 |   1.55 |  2.6 |    1971.63 |   300.71
reduce                             (4) |    25.98 |   0.99 |  3.8 |    1308.81 |   694.52
intfuncs                           (4) |     4.33 |   0.04 |  1.0 |     139.69 |   694.52
simdloop                           (4) |     3.30 |   0.11 |  3.2 |     266.22 |   694.52
vecelement                         (4) |    20.97 |   0.63 |  3.0 |    1326.66 |   694.52
core                               (9) |   107.83 |  10.95 | 10.2 |   10730.47 |   621.39
offsetarray                        (3) |    95.76 |   2.78 |  2.9 |    3357.09 |   349.64
copy                               (3) |     3.22 |   0.09 |  2.7 |     158.77 |   354.75
rational                           (4) |    20.51 |   0.56 |  2.7 |     854.60 |   694.52
reducedim                          (8) |    67.57 |   2.20 |  3.3 |    3260.58 |   332.75
fastmath                           (4) |    13.93 |   0.44 |  3.1 |     534.03 |   694.52
functional                         (8) |    13.78 |   0.46 |  3.4 |     604.60 |   347.16
numbers                            (7) |   146.10 |   2.80 |  1.9 |    5016.75 |   388.21
operators                          (8) |     8.58 |   0.28 |  3.3 |     503.21 |   360.43
path                               (7) |     3.61 |   0.02 |  0.6 |      37.99 |   413.22
parse                              (7) |     7.35 |   0.29 |  4.0 |     363.09 |   414.13
abstractarray                      (2) |   119.79 |   4.72 |  3.9 |    5288.67 |   406.29
math                               (3) |    83.55 |   1.75 |  2.1 |    3010.11 |   414.48
bigfloat                           (3) |     0.95 |   0.02 |  2.5 |      40.31 |   414.48
      From worker 7:    WARNING: replacing module Foo.
loading                            (7) |    48.87 |   2.20 |  4.5 |    3985.50 |   427.63
ccall                              (8) |    60.06 |  41.82 | 69.6 |     783.99 |   390.25
      From worker 7:           [stdio passthrough ok]
iterators                          (4) |    70.42 |   2.64 |  3.8 |    3827.68 |   719.09
bigint                             (2) |    10.08 |   0.26 |  2.6 |     310.46 |   410.27
backtrace                          (8) |     2.11 |   0.03 |  1.3 |      36.32 |   398.46
version                            (8) |     1.94 |   0.05 |  2.6 |      99.16 |   399.62
file                               (4) |     5.89 |   0.18 |  3.1 |     264.31 |   733.49
namedtuple                         (8) |     4.74 |   0.17 |  3.5 |     197.66 |   401.43
arrayops                           (6) |   196.56 |   7.18 |  3.7 |    9208.54 |   535.19
spawn                              (7) |    25.69 |   0.39 |  1.5 |     577.72 |   436.00
floatapprox                        (7) |     2.53 |   0.08 |  3.1 |     152.77 |   436.92
mpfr                               (4) |    23.94 |   0.57 |  2.4 |    1193.60 |   778.85
regex                              (4) |     0.76 |   0.02 |  3.3 |      36.53 |   778.85
reflection                         (7) |     7.59 |   0.32 |  4.2 |     469.47 |   445.36
float16                            (4) |     3.16 |   0.09 |  2.7 |     159.65 |   778.85
sysinfo                            (4) |     0.23 |   0.01 |  5.2 |      15.33 |   778.85
combinatorics                      (7) |     1.30 |   0.05 |  3.5 |      53.01 |   448.13
env                                (4) |     0.76 |   0.01 |  1.5 |      19.67 |   778.85
complex                            (6) |    22.57 |   0.66 |  2.9 |     853.61 |   535.19
mod2pi                             (6) |     0.73 |   0.01 |  2.0 |      26.55 |   535.19
sorting                            (3) |    49.32 |   1.45 |  2.9 |    2377.99 |   450.04
euler                              (6) |     1.75 |   0.09 |  5.0 |     130.47 |   535.19
rounding                           (7) |     8.80 |   0.32 |  3.6 |     515.73 |   466.38
read                               (2) |    43.48 |   1.97 |  4.5 |    3298.00 |   627.56
goto                               (2) |     0.12 |   0.00 |  0.0 |       3.82 |   627.73
llvmcall                           (2) |     0.85 |   0.02 |  2.9 |      55.60 |   627.73
llvmcall2                          (2) |     0.07 |   0.00 |  0.0 |       2.00 |   627.73
grisu                              (2) |     4.89 |   0.12 |  2.5 |     212.25 |   627.73
errorshow                          (6) |    13.41 |   0.42 |  3.2 |     568.20 |   538.46
some                               (2) |     2.31 |   0.06 |  2.4 |      99.19 |   627.73
meta                               (6) |     2.38 |   0.09 |  3.8 |      89.59 |   539.43
stacktraces                        (2) |     5.09 |   0.16 |  3.2 |     203.13 |   627.73
sets                               (7) |    24.21 |   0.70 |  2.9 |     763.24 |   477.86
      From worker 7:    Warning: threaded loop executed in order
docs                               (6) |    14.32 |   0.65 |  4.5 |     546.64 |   563.70
enums                              (6) |    10.61 |   1.02 |  9.6 |     384.46 |   564.53
broadcast                          (8) |    82.81 |   5.03 |  6.1 |    4726.82 |   527.19
threads                            (7) |    33.13 |   0.38 |  1.1 |     236.46 |   486.01
      From worker 2:    Test Summary: | Pass  Total
      From worker 2:    @deprecate    |    6      6
      From worker 2:    Test Summary: | Pass  Total
      From worker 2:    firstcaller   |    1      1
      From worker 2:    Test Summary:              | Broken  Total
      From worker 2:    parser syntax deprecations |      1      1
misc                               (2) |    53.78 |   3.02 |  5.6 |    1908.49 |   656.34
int                                (8) |    25.85 |   0.45 |  1.7 |     273.94 |   548.45
bitarray                           (9) |   229.27 |  14.02 |  6.1 |   12585.36 |   629.53
inline                             (9) |     1.57 |   0.10 |  6.3 |      13.80 |   630.17
bitset                             (2) |    11.25 |   0.09 |  0.8 |      70.89 |   656.79
show                               (3) |    90.40 |   3.55 |  3.9 |    2788.23 |   513.42
cartesian                          (3) |     0.42 |   0.00 |  0.0 |       1.16 |   513.52
checked                            (7) |    31.07 |   0.10 |  0.3 |      48.95 |   518.24
osutils                            (3) |     1.27 |   0.00 |  0.0 |       0.61 |   514.31
error                              (2) |     6.56 |   0.24 |  3.6 |      81.89 |   657.27
iostream                           (3) |     3.57 |   0.12 |  3.3 |      71.10 |   514.36
specificity                        (3) |     0.43 |   0.03 |  6.0 |       4.20 |   514.42
ranges                             (4) |   106.62 |   8.70 |  8.2 |    8564.29 |   830.43
boundscheck                        (9) |    16.66 |   0.13 |  0.8 |      20.74 |   630.99
secretbuffer                       (2) |     3.51 |   1.04 | 29.5 |      40.49 |   657.41
syntax                             (9) |     6.34 |   0.12 |  1.9 |      99.07 |   632.53
codegen                            (3) |    11.15 |   0.98 |  8.7 |      26.95 |   533.74
floatfuncs                         (8) |    34.26 |   1.68 |  4.9 |     661.74 |   565.20
channels                           (7) |    27.74 |   2.65 |  9.5 |     530.75 |   585.53
logging                            (2) |    25.48 |   0.61 |  2.4 |     467.10 |   657.41
reinterpretarray                   (4) |    29.89 |   1.58 |  5.3 |     760.70 |   863.30
asyncmap                           (3) |    29.08 |   1.65 |  5.7 |     696.63 |   533.74
missing                            (9) |    34.04 |   1.45 |  4.3 |     567.98 |   651.17
subarray                           (5) |   490.57 |  32.71 |  6.7 |   24178.32 |   730.59
Pkg/resolve                        (4) |   111.53 |  12.19 | 10.9 |    6461.94 |   916.42
cmdlineargs                        (6) |   339.49 |   0.46 |  0.1 |     282.26 |   564.53
LinearAlgebra/schur                (6) |    89.05 |   4.56 |  5.1 |    2071.92 |   584.50
LinearAlgebra/qr                   (9) |   349.23 |  27.03 |  7.7 |   10402.17 |   878.09
LinearAlgebra/matmul               (4) |   251.77 |  15.46 |  6.1 |    5256.24 |   968.18
LinearAlgebra/special              (6) |    84.62 |   3.75 |  4.4 |    1518.12 |   622.92
SparseArrays/higherorderfns        (8) |   492.97 |  30.73 |  6.2 |   12247.26 |   752.01
LinearAlgebra/eigen                (9) |   119.19 |  10.42 |  8.7 |    2868.32 |   953.38
LinearAlgebra/bunchkaufman         (4) |   146.49 |   9.02 |  6.2 |    3612.94 |  1011.61
LinearAlgebra/svd                  (6) |   175.11 |  11.80 |  6.7 |    3901.14 |   710.16
SparseArrays/sparse                (7) |   650.06 | 105.60 | 16.2 |   10478.00 |   730.51
LinearAlgebra/tridiag              (9) |   193.28 |   9.98 |  5.2 |    2925.90 |  1002.51
LinearAlgebra/lapack               (8) |   201.86 |  13.52 |  6.7 |    3894.11 |   865.50
SparseArrays/sparsevector          (2) |   730.09 |  34.43 |  4.7 |   18190.98 |   905.14
LinearAlgebra/bidiag               (4) |   246.43 |  14.00 |  5.7 |    4614.16 |  1126.11
LinearAlgebra/dense                (5) |   684.14 |  59.40 |  8.7 |   15176.97 |  1060.49
LinearAlgebra/generic              (2) |    88.59 |   5.39 |  6.1 |    1830.64 |   928.02
LinearAlgebra/uniformscaling       (4) |    74.30 |   4.72 |  6.3 |    1282.67 |  1160.13
LinearAlgebra/hessenberg           (2) |    18.78 |   1.07 |  5.7 |     519.05 |   931.70
LinearAlgebra/adjtrans             (2) |    94.01 |   6.48 |  6.9 |    2253.27 |   939.31
LinearAlgebra/lu                   (9) |   267.55 |  21.77 |  8.1 |    4766.23 |  1124.69
LinearAlgebra/cholesky             (7) |   306.55 |  17.56 |  5.7 |    6416.01 |   853.47
LinearAlgebra/blas                 (4) |   126.59 |   7.11 |  5.6 |    1936.44 |  1228.89
LinearAlgebra/givens               (9) |    29.65 |   1.48 |  5.0 |     336.92 |  1126.14
LinearAlgebra/pinv                 (2) |    48.39 |   4.35 |  9.0 |    1260.57 |  1031.24
LinearAlgebra/lq                   (5) |   206.12 |  20.89 | 10.1 |    6268.26 |  1230.05
Dates/query                        (5) |     4.97 |   0.11 |  2.1 |      23.10 |  1235.89
Dates/adjusters                    (2) |    20.35 |   0.68 |  3.3 |     271.59 |  1035.46
LinearAlgebra/symmetric            (8) |   366.30 |  24.82 |  6.8 |    5999.72 |  1005.37
LinearAlgebra/structuredbroadcast  (7) |    96.84 |   7.25 |  7.5 |    2231.46 |   893.03
Dates/types                        (7) |    10.60 |   0.38 |  3.6 |      68.31 |   909.90
Dates/rounding                     (8) |    19.56 |   0.90 |  4.6 |     125.10 |  1013.34
Dates/ranges                       (2) |    54.79 |   3.70 |  6.8 |    1093.80 |  1047.39
Dates/periods                      (5) |    76.51 |   5.72 |  7.5 |     965.24 |  1256.53
Dates/conversions                  (2) |     7.52 |   0.21 |  2.9 |      58.59 |  1048.29
CRC32c                             (2) |     7.02 |   1.25 | 17.9 |     114.69 |  1048.76
Base64                             (5) |    12.71 |   0.96 |  7.6 |     190.64 |  1315.90
LinearAlgebra/diagonal             (6) |   498.22 |  38.09 |  7.6 |    9931.00 |   911.45
Dates/arithmetic                   (8) |    40.82 |   2.37 |  5.8 |     649.40 |  1035.79
Dates/io                           (7) |    50.39 |   2.67 |  5.3 |     867.50 |   922.62
DelimitedFiles                     (2) |    24.34 |   1.67 |  6.9 |     501.67 |  1048.76
Libdl                              (2) |     0.78 |   0.02 |  3.1 |      28.97 |  1048.76
Future                             (8) |     4.40 |   0.23 |  5.2 |      83.15 |  1036.29
Logging                            (2) |    14.05 |   1.03 |  7.4 |     350.58 |  1048.76
Markdown                           (8) |    21.74 |   1.73 |  7.9 |     479.97 |  1037.42
InteractiveUtils                   (7) |    31.53 |   2.02 |  6.4 |     587.83 |   977.88
Dates/accessors                    (9) |   173.97 |   9.29 |  5.3 |    3765.97 |  1128.21
FileWatching                       (6) |    49.28 |   8.22 | 16.7 |     195.11 |   918.34
Profile                            (7) |    12.74 |   0.71 |  5.6 |     305.13 |   981.47
      From worker 4:    The following 'Resetting the helper list...' warning is expected:
Mmap                               (2) |    36.95 |  25.06 | 67.8 |     296.74 |  1048.76
โ”Œ Warning: Resetting the helper list is currently unsupported:
โ”‚ ignoring all git credential helpers
โ”” @ LibGit2 ~/julia/src/julia/usr/share/julia/stdlib/v1.0/LibGit2/src/gitcredential.jl:229
Serialization                      (2) |    20.33 |   1.86 |  9.2 |     672.75 |  1048.76
Printf                             (8) |    99.25 |  10.08 | 10.2 |    2987.53 |  1064.66
โ”Œ Warning: TerminalMenus: Unable to enter raw mode: ArgumentError("stream is closed or unusable")
โ”” @ REPL.TerminalMenus ~/julia/src/julia/usr/share/julia/stdlib/v1.0/REPL/src/TerminalMenus/util.jl:21
      From worker 9:
โ”Œ Warning: TerminalMenus: Unable to enter raw mode: ArgumentError("stream is closed or unusable")
โ”” @ REPL.TerminalMenus ~/julia/src/julia/usr/share/julia/stdlib/v1.0/REPL/src/TerminalMenus/util.jl:21
      From worker 9:
REPL                               (9) |   135.78 |  10.01 |  7.4 |    2211.22 |  1203.76
Random                             (6) |   146.50 |   9.37 |  6.4 |    2722.16 |  1006.78
UUIDs                              (6) |     0.42 |   0.00 |  0.0 |       6.92 |  1006.78
Unicode                            (6) |    16.64 |   0.59 |  3.5 |     247.89 |  1018.78
Statistics                         (2) |   133.77 |   9.77 |  7.3 |    2396.21 |  1055.93
โ”Œ Warning: Omitting `@test_deprecated` tests which can't yet
โ”‚ be tested in --depwarn=error mode
โ”” @ Main.Test77Main_Test ~/julia/src/julia/usr/share/julia/stdlib/v1.0/Test/test/runtests.jl:736
Test                               (9) |    88.86 |   2.32 |  2.6 |     388.82 |  1204.68
SuiteSparse                        (8) |   201.04 |  12.39 |  6.2 |    3520.37 |  1142.39
LibGit2/libgit2                    (4) |   483.44 |   6.14 |  1.3 |    1532.96 |  1603.24
Distributed                        (5) |   369.16 |   0.23 |  0.1 |      26.94 |  1315.90
SHA                                (7) |   323.17 |   2.16 |  0.7 |     647.92 |  1030.83
LinearAlgebra/triangular           (3) |  1505.82 |  89.13 |  5.9 |   35050.41 |  1407.03

Executing tests that run on node 1 only:
precompile                         (1) |    43.08 |   0.22 |  0.5 |     466.87 |   239.95
SharedArrays                       (1) |    40.55 |   1.43 |  3.5 |    1756.68 |   268.86
stress: Test Failed at /build/julia/src/julia/test/stress.jl:98
  Expression: begin
    ccall(:kill, Cvoid, (Cint, Cint), getpid(), 2)
    for i = 1:10
        Libc.systemsleep(0.1)
        ccall(:jl_gc_safepoint, Cvoid, ())
    end
end
    Expected: InterruptException
  No exception thrown
Stacktrace:
 [1] top-level scope at /build/julia/src/julia/test/stress.jl:98
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1038
 [4] include at ./sysimg.jl:29 [inlined]
 [5] include(::String) at /build/julia/src/julia/test/testdefs.jl:13
 [6] macro expansion at /build/julia/src/julia/test/testdefs.jl:22 [inlined]
 [7] macro expansion at /build/julia/src/julia/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [8] macro expansion at /build/julia/src/julia/test/testdefs.jl:21 [inlined]
 [9] macro expansion at ./util.jl:289 [inlined]
 [10] top-level scope at /build/julia/src/julia/test/testdefs.jl:19 [inlined]
 [11] top-level scope at ./none:0
stress: Error During Test at none:1
  Test threw exception
  Expression: stress
  Some tests did not pass: 118 passed, 1 failed, 0 errored, 0 broken.

Test Summary:                       |     Pass  Error  Broken     Total
  Overall                           | 37408366      1  327816  37736183
    compiler/validation             |       26                       26
    strings/search                  |      522                      522
    strings/io                      |    12749                    12749
    unicode/utf8                    |       19                       19
    strings/util                    |      449                      449
    worlds                          |       64                       64
    strings/types                   |  2302688                  2302688
    compiler/compiler               |      545              1       546
    keywordargs                     |      127                      127
    triplequote                     |       28                       28
    ambiguous                       |       62              2        64
    intrinsics                      |       34                       34
    char                            |     1471                     1471
    strings/basic                   |    87281                    87281
    iobuffer                        |      200                      200
    staged                          |       59                       59
    hashing                         |    18762                    18762
    subtype                         |   337379             15    337394
    tuple                           |      483                      483
    dict                            |   144314                   144314
    reduce                          |      292                      292
    intfuncs                        |     4386                     4386
    simdloop                        |      248                      248
    vecelement                      |      533                      533
    core                            |  8423072                  8423072
    offsetarray                     |      324                      324
    copy                            |      512                      512
    rational                        |    97519                    97519
    reducedim                       |      673                      673
    fastmath                        |      811                      811
    functional                      |       95                       95
    numbers                         |  1380107                  1380107
    operators                       |    12922                    12922
    path                            |      233             12       245
    parse                           |    10291                    10291
    abstractarray                   |     3972                     3972
    math                            |  1505744                  1505744
    bigfloat                        |       32                       32
    loading                         |   156014                   156014
    ccall                           |     5244                     5244
    iterators                       |     1497                     1497
    bigint                          |     2061                     2061
    backtrace                       |       25              1        26
    version                         |     2464                     2464
    file                            |      781                      781
    namedtuple                      |      154              1       155
    arrayops                        |     1766                     1766
    spawn                           |      130              4       134
    floatapprox                     |       49                       49
    mpfr                            |      760                      760
    regex                           |       28                       28
    reflection                      |      330                      330
    float16                         |      124                      124
    sysinfo                         |        2                        2
    combinatorics                   |       98                       98
    env                             |       56                       56
    complex                         |     8250              2      8252
    mod2pi                          |       80                       80
    sorting                         |     4864                     4864
    euler                           |       12                       12
    rounding                        |      880                      880
    read                            |     3085                     3085
    goto                            |       19                       19
    llvmcall                        |       13                       13
    llvmcall2                       |        6                        6
    grisu                           |      683                      683
    errorshow                       |      172                      172
    some                            |       62                       62
    meta                            |       55                       55
    stacktraces                     |       43                       43
    sets                            |      736              1       737
    docs                            |      197                      197
    enums                           |       84                       84
    broadcast                       |      422                      422
    threads                         |   190579                   190579
    misc                            |  1279371                  1279371
    int                             |    10727                    10727
    bitarray                        |   898011                   898011
    inline                          |       13                       13
    bitset                          |      186                      186
    show                            |      329                      329
    cartesian                       |        7                        7
    checked                         |     1219                     1219
    osutils                         |       26                       26
    error                           |       30                       30
    iostream                        |       27                       27
    specificity                     |      126                      126
    ranges                          | 12108035         327680  12435715
    boundscheck                     |                          No tests
    secretbuffer                    |       12                       12
    syntax                          |      579              1       580
    codegen                         |       94                       94
    floatfuncs                      |      134                      134
    channels                        |      182                      182
    logging                         |      115                      115
    reinterpretarray                |      114                      114
    asyncmap                        |      292                      292
    missing                         |      403              1       404
    subarray                        |      280                      280
    Pkg/resolve                     |      182                      182
    cmdlineargs                     |      206                      206
    LinearAlgebra/schur             |      390                      390
    LinearAlgebra/qr                |     3118                     3118
    LinearAlgebra/matmul            |      597                      597
    LinearAlgebra/special           |      947                      947
    SparseArrays/higherorderfns     |     7073             72      7145
    LinearAlgebra/eigen             |      406                      406
    LinearAlgebra/bunchkaufman      |     5145                     5145
    LinearAlgebra/svd               |      412                      412
    SparseArrays/sparse             |     2144                     2144
    LinearAlgebra/tridiag           |     1159                     1159
    LinearAlgebra/lapack            |      780                      780
    SparseArrays/sparsevector       |     9919                     9919
    LinearAlgebra/bidiag            |     1750                     1750
    LinearAlgebra/dense             |     7467              2      7469
    LinearAlgebra/generic           |      408                      408
    LinearAlgebra/uniformscaling    |      313                      313
    LinearAlgebra/hessenberg        |       40                       40
    LinearAlgebra/adjtrans          |      252                      252
    LinearAlgebra/lu                |     1191                     1191
    LinearAlgebra/cholesky          |     2186                     2186
    LinearAlgebra/blas              |      628                      628
    LinearAlgebra/givens            |     1840                     1840
    LinearAlgebra/pinv              |      288                      288
    LinearAlgebra/lq                |     1253                     1253
    Dates/query                     |      988                      988
    Dates/adjusters                 |     3147                     3147
    LinearAlgebra/symmetric         |     1966                     1966
    LinearAlgebra/structuredbroadcast |      408                      408
    Dates/types                     |      167                      167
    Dates/rounding                  |      296                      296
    Dates/ranges                    |   349123                   349123
    Dates/periods                   |      681                      681
    Dates/conversions               |      160                      160
    CRC32c                          |      664                      664
    Base64                          |     1015                     1015
    LinearAlgebra/diagonal          |     1531                     1531
    Dates/arithmetic                |      312                      312
    Dates/io                        |      258                      258
    DelimitedFiles                  |       81                       81
    Libdl                           |       63              1        64
    Future                          |       33                       33
    Logging                         |       34                       34
    Markdown                        |      231                      231
    InteractiveUtils                |      104                      104
    Dates/accessors                 |  7723858                  7723858
    FileWatching                    |      510                      510
    Profile                         |       10                       10
    Mmap                            |      131                      131
    Serialization                   |      106                      106
    Printf                          |      738                      738
    REPL                            |      984              5       989
    Random                          |   203068                   203068
    UUIDs                           |       10                       10
    Unicode                         |      752                      752
    Statistics                      |      574                      574
    Test                            |      285             14       299
    SuiteSparse                     |      759                      759
    LibGit2/libgit2                 |      666              1       667
    Distributed                     |                          No tests
    SHA                             |        1                        1
    LinearAlgebra/triangular        |    33194                    33194
    precompile                      |       89                       89
    SharedArrays                    |      110                      110
    stress                          |               1                 1
    FAILURE

The global RNG seed was 0xa3d406745cd6ad08ccaffb2bb9e1ccea.

Error in testset stress:
Error During Test at none:1
  Test threw exception
  Expression: stress
  Some tests did not pass: 118 passed, 1 failed, 0 errored, 0 broken.
ERROR: LoadError: Test run finished with errors
in expression starting at /build/julia/src/julia/test/runtests.jl:61

mpfr:

Test                          (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB)
compiler/validation                (4) |     9.52 |   0.36 |  3.8 |     154.03 |   215.82
strings/search                     (7) |    14.52 |   0.44 |  3.1 |     246.68 |   215.82
unicode/utf8                       (7) |     1.09 |   0.00 |  0.0 |       2.25 |   215.82
strings/io                         (9) |    17.36 |   0.54 |  3.1 |     257.39 |   221.89
      From worker 2:    Skipping Base.Filesystem.JL_O_RANDOM
      From worker 2:    Skipping Base.Filesystem.JL_O_SEQUENTIAL
      From worker 2:    Skipping Base.Filesystem.JL_O_SHORT_LIVED
      From worker 2:    Skipping Base.Filesystem.JL_O_TEMPORARY
      From worker 2:    Skipping Base.active_repl
      From worker 2:    Skipping Base.active_repl_backend
worlds                             (9) |     9.27 |   0.19 |  2.0 |     132.57 |   223.74
strings/util                       (8) |    27.57 |   0.59 |  2.1 |     309.89 |   222.52
strings/types                      (4) |    19.64 |   0.61 |  3.1 |    1183.92 |   216.50
      From worker 2:    Skipping Base.active_repl
      From worker 2:    Skipping Base.active_repl_backend
      From worker 2:    Skipping Base.Filesystem.JL_O_RANDOM
      From worker 2:    Skipping Base.Filesystem.JL_O_SEQUENTIAL
      From worker 2:    Skipping Base.Filesystem.JL_O_SHORT_LIVED
      From worker 2:    Skipping Base.Filesystem.JL_O_TEMPORARY
      From worker 2:    Skipping Base.active_repl
      From worker 2:    Skipping Base.active_repl_backend
compiler/compiler                  (3) |    40.73 |   1.16 |  2.9 |     690.09 |   241.13
ambiguous                          (2) |    41.32 |   0.95 |  2.3 |    1153.28 |   241.18
triplequote                        (2) |     0.15 |   0.00 |  0.0 |       1.66 |   241.43
intrinsics                         (2) |     1.41 |   0.01 |  0.4 |      23.02 |   241.95
keywordargs                        (9) |    17.57 |   0.11 |  0.6 |     111.47 |   226.97
char                               (3) |    12.67 |   0.63 |  5.0 |     227.22 |   254.43
strings/basic                      (6) |    58.43 |   0.93 |  1.6 |     667.02 |   243.43
iobuffer                           (3) |    14.67 |   0.19 |  1.3 |     117.25 |   269.30
staged                             (6) |    15.98 |   0.29 |  1.8 |     266.15 |   244.63
hashing                            (9) |    84.08 |   1.87 |  2.2 |    1445.07 |   261.82
tuple                              (9) |    24.98 |   0.53 |  2.1 |     264.26 |   277.89
subtype                            (4) |   132.49 |   3.09 |  2.3 |    1797.38 |   694.62
dict                               (2) |   133.49 |   3.99 |  3.0 |    1970.74 |   302.38
reduce                             (9) |    62.72 |   2.35 |  3.7 |    1215.56 |   432.69
intfuncs                           (9) |    15.26 |   0.18 |  1.2 |     132.14 |   432.69
simdloop                           (9) |    12.03 |   0.45 |  3.8 |     261.24 |   432.69
core                               (7) |   253.20 |  23.66 |  9.3 |   10726.55 |   618.90
vecelement                         (9) |    47.51 |   1.65 |  3.5 |    1325.62 |   432.69
offsetarray                        (3) |   222.78 |   5.97 |  2.7 |    3356.87 |   349.89
copy                               (3) |     9.81 |   0.29 |  3.0 |     158.77 |   354.37
rational                           (7) |    39.08 |   1.45 |  3.7 |     883.32 |   618.90
reducedim                          (4) |   165.27 |   5.61 |  3.4 |    3315.79 |   694.62
fastmath                           (7) |    32.50 |   0.94 |  2.9 |     516.64 |   618.90
functional                         (4) |    34.12 |   1.01 |  3.0 |     637.04 |   694.62
operators                          (4) |    25.96 |   1.19 |  4.6 |     527.16 |   694.62
numbers                            (8) |   364.72 |   6.24 |  1.7 |    5053.62 |   388.96
path                               (4) |    11.51 |   0.06 |  0.5 |      35.58 |   694.62
parse                              (4) |    18.90 |   0.49 |  2.6 |     435.20 |   694.62
abstractarray                      (2) |   281.42 |  11.13 |  4.0 |    5278.09 |   405.94
ccall                              (8) |   120.58 |  82.56 | 68.5 |     781.08 |   404.97
math                               (3) |   203.76 |   3.71 |  1.8 |    3008.74 |   414.69
bigfloat                           (8) |     0.97 |   0.05 |  5.3 |      14.28 |   404.97
iterators                          (7) |   174.61 |   8.43 |  4.8 |    3839.43 |   618.90
bigint                             (2) |    24.38 |   0.65 |  2.7 |     310.37 |   419.30
      From worker 8:           [stdio passthrough ok]
backtrace                          (7) |     4.29 |   0.06 |  1.4 |      33.20 |   618.90
      From worker 4:    WARNING: replacing module Foo.
loading                            (4) |   108.50 |   5.04 |  4.6 |    3567.07 |   694.62
version                            (4) |     4.24 |   0.14 |  3.4 |      98.13 |   694.62
file                               (2) |    20.55 |   0.66 |  3.2 |     358.37 |   423.94
namedtuple                         (4) |    13.84 |   0.38 |  2.8 |     210.90 |   694.62
arrayops                           (6) |   480.10 |  17.12 |  3.6 |    9209.84 |   536.76
spawn                              (8) |    45.58 |   0.76 |  1.7 |     600.03 |   418.00
floatapprox                        (8) |     6.70 |   0.18 |  2.7 |     153.87 |   418.99
reflection                         (8) |    29.19 |   1.01 |  3.5 |     469.78 |   434.38
regex                              (8) |     2.99 |   0.11 |  3.7 |      36.22 |   434.63
float16                            (8) |     8.85 |   0.22 |  2.5 |     109.87 |   441.00
combinatorics                      (8) |     5.21 |   0.09 |  1.7 |      53.56 |   443.35
sysinfo                            (8) |     1.14 |   0.06 |  5.3 |      14.99 |   443.63
mpfr                               (2) |    81.85 |   1.78 |  2.2 |    1285.43 |   472.34
env                                (8) |     3.86 |   0.04 |  1.0 |      19.65 |   443.79
read                               (7) |   126.78 |   7.11 |  5.6 |    3178.49 |   759.98
mod2pi                             (7) |     4.00 |   0.24 |  5.9 |      32.74 |   760.79
complex                            (6) |    82.11 |   2.01 |  2.4 |     853.82 |   536.76
rounding                           (2) |    34.20 |   1.21 |  3.5 |     526.32 |   473.41
euler                              (7) |     6.41 |   0.58 |  9.0 |     133.48 |   768.95
sorting                            (3) |   157.33 |   4.14 |  2.6 |    2385.30 |   448.07
goto                               (3) |     0.34 |   0.00 |  0.0 |       1.11 |   448.07
llvmcall                           (3) |     1.31 |   0.05 |  3.9 |      17.83 |   448.33
llvmcall2                          (3) |     0.28 |   0.00 |  0.0 |       2.00 |   448.38
grisu                              (3) |    14.97 |   0.37 |  2.4 |     185.69 |   450.13
some                               (3) |     5.46 |   0.14 |  2.5 |      92.86 |   456.32
meta                               (3) |     6.13 |   0.24 |  3.8 |      92.27 |   456.50
errorshow                          (2) |    51.45 |   1.86 |  3.6 |     655.74 |   483.05
stacktraces                        (3) |    14.85 |   0.52 |  3.5 |     240.63 |   462.17
sets                               (7) |    76.36 |   3.46 |  4.5 |     762.69 |   780.58
      From worker 7:    Warning: threaded loop executed in order
docs                               (2) |    41.90 |   1.89 |  4.5 |     652.58 |   496.82
enums                              (2) |    20.32 |   0.92 |  4.5 |     327.73 |   504.07
broadcast                          (4) |   235.53 |  11.55 |  4.9 |    4717.70 |   755.91
threads                            (7) |    64.42 |   4.56 |  7.1 |     238.76 |   791.42
      From worker 3:    Test Summary: | Pass  Total
      From worker 3:    @deprecate    |    6      6
      From worker 3:    Test Summary: | Pass  Total
      From worker 3:    firstcaller   |    1      1
      From worker 3:    Test Summary:              | Broken  Total
      From worker 3:    parser syntax deprecations |      1      1
misc                               (3) |    98.72 |   6.97 |  7.1 |    1804.86 |   559.47
int                                (4) |    38.84 |   0.62 |  1.6 |     236.55 |   778.24
bitset                             (3) |    18.34 |   0.28 |  1.5 |      73.20 |   559.47
inline                             (3) |     2.45 |   0.10 |  4.0 |      30.99 |   559.47
ranges                             (8) |   217.98 |  14.79 |  6.8 |    8552.28 |   524.17
checked                            (7) |    43.11 |   0.26 |  0.6 |      78.18 |   828.18
cartesian                          (7) |     0.73 |   0.00 |  0.0 |       8.99 |   828.25
show                               (6) |   195.05 |   8.10 |  4.2 |    2808.78 |   571.95
osutils                            (7) |     1.54 |   0.00 |  0.0 |       0.61 |   828.28
iostream                           (7) |     3.52 |   0.32 |  9.1 |      67.13 |   828.42
error                              (8) |     6.50 |   0.12 |  1.9 |      72.62 |   525.46
specificity                        (8) |     0.40 |   0.00 |  0.0 |       4.71 |   525.46
floatfuncs                         (4) |    34.54 |   1.54 |  4.5 |     662.73 |   795.75
secretbuffer                       (7) |     3.57 |   1.19 | 33.2 |      32.93 |   828.46
boundscheck                        (3) |    21.20 |   0.00 |  0.0 |       0.90 |   559.47
codegen                            (8) |     8.78 |   1.08 | 12.3 |      26.83 |   525.46
syntax                             (7) |     8.74 |   0.24 |  2.7 |     100.64 |   828.46
channels                           (6) |    29.10 |   3.65 | 12.6 |     555.13 |   655.88
bitarray                           (9) |   592.55 |  27.95 |  4.7 |   12450.13 |   583.75
logging                            (3) |    31.76 |   0.93 |  2.9 |     457.77 |   559.47
reinterpretarray                   (4) |    35.22 |   2.21 |  6.3 |     690.14 |   813.97
asyncmap                           (7) |    37.39 |   1.70 |  4.6 |     710.01 |   828.46
missing                            (8) |    46.93 |   1.91 |  4.1 |     679.52 |   528.38
subarray                           (5) |  1014.56 |  62.94 |  6.2 |   24177.84 |   734.57
Pkg/resolve                        (4) |   130.51 |  14.70 | 11.3 |    6311.65 |   874.10
cmdlineargs                        (2) |   413.00 |   1.29 |  0.3 |     274.76 |   531.19
LinearAlgebra/schur                (2) |   135.49 |   6.26 |  4.6 |    2397.13 |   548.52
LinearAlgebra/matmul               (4) |   318.65 |  18.89 |  5.9 |    5171.46 |   941.84
LinearAlgebra/qr                   (8) |   462.25 |  27.21 |  5.9 |   10351.28 |   757.92
LinearAlgebra/special              (2) |   111.84 |   4.44 |  4.0 |    1700.68 |   597.65
SparseArrays/higherorderfns        (6) |   622.89 |  36.50 |  5.9 |   12283.17 |   756.53
LinearAlgebra/bunchkaufman         (8) |   146.89 |   8.66 |  5.9 |    2726.68 |   805.26
LinearAlgebra/eigen                (4) |   190.79 |  11.62 |  6.1 |    3576.19 |   981.35
SparseArrays/sparse                (9) |   759.78 | 151.64 | 20.0 |   10134.73 |   828.83
LinearAlgebra/svd                  (2) |   203.91 |  12.97 |  6.4 |    4052.89 |   672.47
LinearAlgebra/lapack               (6) |   197.11 |  12.01 |  6.1 |    3678.70 |   839.48
LinearAlgebra/tridiag              (8) |   202.19 |   8.33 |  4.1 |    2922.24 |   866.04
SparseArrays/sparsevector          (3) |   858.32 |  39.28 |  4.6 |   17877.06 |   932.37
LinearAlgebra/bidiag               (4) |   263.44 |  15.53 |  5.9 |    4262.74 |  1089.36
LinearAlgebra/dense                (5) |   789.77 |  68.77 |  8.7 |   15170.40 |  1061.60
LinearAlgebra/generic              (3) |    78.97 |   4.72 |  6.0 |    1668.48 |   948.98
LinearAlgebra/hessenberg           (3) |    37.56 |   2.41 |  6.4 |     508.19 |   952.35
LinearAlgebra/uniformscaling       (4) |    91.78 |   5.37 |  5.8 |    1274.78 |  1125.23
LinearAlgebra/adjtrans             (4) |   117.13 |   8.32 |  7.1 |    1640.48 |  1143.37
LinearAlgebra/cholesky             (2) |   412.22 |  24.67 |  6.0 |    6204.20 |   795.05
LinearAlgebra/pinv                 (4) |    54.11 |   4.26 |  7.9 |    1186.28 |  1247.96
LinearAlgebra/blas                 (3) |   209.65 |  11.40 |  5.4 |    2640.84 |   986.56
LinearAlgebra/givens               (2) |    21.15 |   1.01 |  4.8 |     355.31 |   806.00
LinearAlgebra/lq                   (5) |   297.26 |  27.93 |  9.4 |    6268.08 |  1238.22
LinearAlgebra/lu                   (6) |   412.54 |  28.86 |  7.0 |    6821.57 |  1042.70
LinearAlgebra/symmetric            (8) |   391.84 |  26.09 |  6.7 |    5038.90 |   956.71
Dates/query                        (6) |     6.28 |   0.08 |  1.3 |      23.19 |  1042.98
Dates/adjusters                    (5) |    25.08 |   1.08 |  4.3 |     251.25 |  1262.91
Dates/rounding                     (5) |    11.44 |   0.50 |  4.4 |     111.32 |  1270.69
Dates/types                        (5) |     9.95 |   0.36 |  3.6 |      81.91 |  1277.62
LinearAlgebra/structuredbroadcast  (4) |   107.30 |   7.98 |  7.4 |    2232.33 |  1247.96
Dates/ranges                       (6) |    60.51 |   3.77 |  6.2 |    1129.78 |  1055.29
Dates/periods                      (8) |    66.44 |   3.25 |  4.9 |     926.75 |   984.34
Dates/conversions                  (6) |     6.62 |   0.14 |  2.2 |      56.88 |  1060.83
CRC32c                             (6) |     3.83 |   0.28 |  7.3 |      75.97 |  1062.58
Base64                             (8) |     8.52 |   1.37 | 16.1 |     178.66 |   984.42
Dates/io                           (5) |    64.81 |   3.69 |  5.7 |     863.01 |  1348.06
Dates/arithmetic                   (4) |    48.44 |   2.66 |  5.5 |     672.17 |  1247.96
DelimitedFiles                     (6) |    33.36 |   2.04 |  6.1 |     463.72 |  1071.77
Future                             (4) |     8.43 |   1.20 | 14.3 |      86.82 |  1247.96
Libdl                              (4) |     3.01 |   0.10 |  3.4 |      25.67 |  1247.96
Logging                            (4) |    17.84 |   1.41 |  7.9 |     359.76 |  1247.96
FileWatching                       (5) |    59.57 |  14.68 | 24.6 |     179.44 |  1348.06
InteractiveUtils                   (6) |    44.98 |   2.69 |  6.0 |     625.29 |  1127.43
Markdown                           (4) |    28.98 |   2.14 |  7.4 |     492.88 |  1247.96
LinearAlgebra/diagonal             (9) |   648.36 |  51.75 |  8.0 |   10777.88 |  1031.84
Profile                            (4) |    21.08 |   1.64 |  7.8 |     312.66 |  1250.49
Dates/accessors                    (2) |   218.61 |   9.18 |  4.2 |    3768.73 |   815.97
      From worker 3:    The following 'Resetting the helper list...' warning is expected:
โ”Œ Warning: Resetting the helper list is currently unsupported:
โ”‚ ignoring all git credential helpers
โ”” @ LibGit2 ~/julia/src/julia/usr/share/julia/stdlib/v1.0/LibGit2/src/gitcredential.jl:229
Mmap                               (5) |    51.83 |  38.68 | 74.6 |     275.21 |  1348.06
Serialization                      (5) |    30.09 |   3.48 | 11.6 |     753.64 |  1348.06
Printf                             (6) |   105.35 |   9.32 |  8.8 |    2958.97 |  1140.58
โ”Œ Warning: TerminalMenus: Unable to enter raw mode: ArgumentError("stream is closed or unusable")
โ”” @ REPL.TerminalMenus ~/julia/src/julia/usr/share/julia/stdlib/v1.0/REPL/src/TerminalMenus/util.jl:21
      From worker 9:    
โ”Œ Warning: TerminalMenus: Unable to enter raw mode: ArgumentError("stream is closed or unusable")
โ”” @ REPL.TerminalMenus ~/julia/src/julia/usr/share/julia/stdlib/v1.0/REPL/src/TerminalMenus/util.jl:21
      From worker 9:    
REPL                               (9) |   117.92 |  10.08 |  8.5 |    2210.09 |  1101.95
Random                             (4) |   147.27 |  10.95 |  7.4 |    2706.87 |  1368.16
UUIDs                              (4) |     0.33 |   0.00 |  0.0 |       6.64 |  1368.18
Unicode                            (4) |    21.35 |   1.34 |  6.3 |     308.39 |  1381.14
Statistics                         (5) |   124.03 |  10.94 |  8.8 |    2470.17 |  1348.06
โ”Œ Warning: Omitting `@test_deprecated` tests which can't yet
โ”‚ be tested in --depwarn=error mode
โ”” @ Main.Test97Main_Test ~/julia/src/julia/usr/share/julia/stdlib/v1.0/Test/test/runtests.jl:736
Test                               (9) |   119.12 |   1.63 |  1.4 |     417.78 |  1106.88
SuiteSparse                        (6) |   211.72 |  13.68 |  6.5 |    3369.41 |  1225.41
LibGit2/libgit2                    (3) |   520.40 |   8.16 |  1.6 |    1488.83 |  1369.19
Distributed                        (8) |   403.80 |   0.05 |  0.0 |      28.14 |   984.42
SHA                                (2) |   319.92 |   1.74 |  0.5 |     687.32 |   867.99
LinearAlgebra/triangular           (7) |  1773.89 | 111.24 |  6.3 |   35336.51 |  1502.91

Executing tests that run on node 1 only:
precompile                         (1) |    38.24 |   0.22 |  0.6 |     454.97 |   239.77
SharedArrays                       (1) |    33.86 |   1.30 |  3.8 |    1756.70 |   270.38
stress: Test Failed at /build/julia/src/julia/test/stress.jl:98
  Expression: begin
    ccall(:kill, Cvoid, (Cint, Cint), getpid(), 2)
    for i = 1:10
        Libc.systemsleep(0.1)
        ccall(:jl_gc_safepoint, Cvoid, ())
    end
end
    Expected: InterruptException
  No exception thrown
Stacktrace:
 [1] top-level scope at /build/julia/src/julia/test/stress.jl:98
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1038
 [4] include at ./sysimg.jl:29 [inlined]
 [5] include(::String) at /build/julia/src/julia/test/testdefs.jl:13
 [6] macro expansion at /build/julia/src/julia/test/testdefs.jl:22 [inlined]
 [7] macro expansion at /build/julia/src/julia/usr/share/julia/stdlib/v1.0/Test/src/Test.jl:1083 [inlined]
 [8] macro expansion at /build/julia/src/julia/test/testdefs.jl:21 [inlined]
 [9] macro expansion at ./util.jl:289 [inlined]
 [10] top-level scope at /build/julia/src/julia/test/testdefs.jl:19 [inlined]
 [11] top-level scope at ./none:0
stress: Error During Test at none:1
  Test threw exception
  Expression: stress
  Some tests did not pass: 118 passed, 1 failed, 0 errored, 0 broken.

Test Summary:                       |     Pass  Error  Broken     Total
  Overall                           | 37391103      1  327816  37718920
    compiler/validation             |       26                       26
    strings/search                  |      522                      522
    unicode/utf8                    |       19                       19
    strings/io                      |    12749                    12749
    worlds                          |       64                       64
    strings/util                    |      449                      449
    strings/types                   |  2302688                  2302688
    compiler/compiler               |      545              1       546
    ambiguous                       |       62              2        64
    triplequote                     |       28                       28
    intrinsics                      |       34                       34
    keywordargs                     |      127                      127
    char                            |     1471                     1471
    strings/basic                   |    87281                    87281
    iobuffer                        |      200                      200
    staged                          |       59                       59
    hashing                         |    18762                    18762
    tuple                           |      483                      483
    subtype                         |   337379             15    337394
    dict                            |   144314                   144314
    reduce                          |      292                      292
    intfuncs                        |     4386                     4386
    simdloop                        |      248                      248
    core                            |  8423072                  8423072
    vecelement                      |      533                      533
    offsetarray                     |      324                      324
    copy                            |      512                      512
    rational                        |    97519                    97519
    reducedim                       |      673                      673
    fastmath                        |      811                      811
    functional                      |       95                       95
    operators                       |    12922                    12922
    numbers                         |  1380107                  1380107
    path                            |      233             12       245
    parse                           |    10291                    10291
    abstractarray                   |     2752                     2752
    ccall                           |     5244                     5244
    math                            |  1505744                  1505744
    bigfloat                        |       32                       32
    iterators                       |     1497                     1497
    bigint                          |     2061                     2061
    backtrace                       |       25              1        26
    loading                         |   141854                   141854
    version                         |     2464                     2464
    file                            |      781                      781
    namedtuple                      |      154              1       155
    arrayops                        |     1766                     1766
    spawn                           |      130              4       134
    floatapprox                     |       49                       49
    reflection                      |      330                      330
    regex                           |       28                       28
    float16                         |      124                      124
    combinatorics                   |       98                       98
    sysinfo                         |        2                        2
    mpfr                            |      760                      760
    env                             |       56                       56
    read                            |     3085                     3085
    mod2pi                          |       80                       80
    complex                         |     8250              2      8252
    rounding                        |      880                      880
    euler                           |       12                       12
    sorting                         |     4864                     4864
    goto                            |       19                       19
    llvmcall                        |       13                       13
    llvmcall2                       |        6                        6
    grisu                           |      683                      683
    some                            |       62                       62
    meta                            |       55                       55
    errorshow                       |      172                      172
    stacktraces                     |       43                       43
    sets                            |      736              1       737
    docs                            |      197                      197
    enums                           |       84                       84
    broadcast                       |      422                      422
    threads                         |   190579                   190579
    misc                            |  1279371                  1279371
    int                             |    10727                    10727
    bitset                          |      186                      186
    inline                          |       13                       13
    ranges                          | 12108108         327680  12435788
    checked                         |     1219                     1219
    cartesian                       |        7                        7
    show                            |      329                      329
    osutils                         |       26                       26
    iostream                        |       27                       27
    error                           |       30                       30
    specificity                     |      126                      126
    floatfuncs                      |      134                      134
    secretbuffer                    |       12                       12
    boundscheck                     |                          No tests
    codegen                         |       94                       94
    syntax                          |      579              1       580
    channels                        |      182                      182
    bitarray                        |   896055                   896055
    logging                         |      115                      115
    reinterpretarray                |      114                      114
    asyncmap                        |      292                      292
    missing                         |      403              1       404
    subarray                        |      280                      280
    Pkg/resolve                     |      182                      182
    cmdlineargs                     |      206                      206
    LinearAlgebra/schur             |      390                      390
    LinearAlgebra/matmul            |      597                      597
    LinearAlgebra/qr                |     3118                     3118
    LinearAlgebra/special           |      947                      947
    SparseArrays/higherorderfns     |     7073             72      7145
    LinearAlgebra/bunchkaufman      |     5145                     5145
    LinearAlgebra/eigen             |      406                      406
    SparseArrays/sparse             |     2144                     2144
    LinearAlgebra/svd               |      412                      412
    LinearAlgebra/lapack            |      780                      780
    LinearAlgebra/tridiag           |     1159                     1159
    SparseArrays/sparsevector       |     9919                     9919
    LinearAlgebra/bidiag            |     1750                     1750
    LinearAlgebra/dense             |     7467              2      7469
    LinearAlgebra/generic           |      408                      408
    LinearAlgebra/hessenberg        |       40                       40
    LinearAlgebra/uniformscaling    |      313                      313
    LinearAlgebra/adjtrans          |      252                      252
    LinearAlgebra/cholesky          |     2186                     2186
    LinearAlgebra/pinv              |      288                      288
    LinearAlgebra/blas              |      628                      628
    LinearAlgebra/givens            |     1840                     1840
    LinearAlgebra/lq                |     1253                     1253
    LinearAlgebra/lu                |     1191                     1191
    LinearAlgebra/symmetric         |     1966                     1966
    Dates/query                     |      988                      988
    Dates/adjusters                 |     3147                     3147
    Dates/rounding                  |      296                      296
    Dates/types                     |      167                      167
    LinearAlgebra/structuredbroadcast |      408                      408
    Dates/ranges                    |   349123                   349123
    Dates/periods                   |      681                      681
    Dates/conversions               |      160                      160
    CRC32c                          |      664                      664
    Base64                          |     1015                     1015
    Dates/io                        |      258                      258
    Dates/arithmetic                |      312                      312
    DelimitedFiles                  |       81                       81
    Future                          |       33                       33
    Libdl                           |       63              1        64
    Logging                         |       34                       34
    FileWatching                    |      510                      510
    InteractiveUtils                |      104                      104
    Markdown                        |      231                      231
    LinearAlgebra/diagonal          |     1531                     1531
    Profile                         |       10                       10
    Dates/accessors                 |  7723858                  7723858
    Mmap                            |      131                      131
    Serialization                   |      106                      106
    Printf                          |      738                      738
    REPL                            |      984              5       989
    Random                          |   203068                   203068
    UUIDs                           |       10                       10
    Unicode                         |      752                      752
    Statistics                      |      574                      574
    Test                            |      285             14       299
    SuiteSparse                     |      759                      759
    LibGit2/libgit2                 |      666              1       667
    Distributed                     |                          No tests
    SHA                             |        1                        1
    LinearAlgebra/triangular        |    33194                    33194
    precompile                      |       89                       89
    SharedArrays                    |      110                      110
    stress                          |               1                 1
    FAILURE

The global RNG seed was 0x8130efd3ea41bc2925fc76477fec2dd0.

Error in testset stress:
Error During Test at none:1
  Test threw exception
  Expression: stress
  Some tests did not pass: 118 passed, 1 failed, 0 errored, 0 broken.
ERROR: LoadError: Test run finished with errors
in expression starting at /build/julia/src/julia/test/runtests.jl:61

libm fails with a highly uninformative internal make error:

    CC src/llvm-api.o
    LINK usr/lib/libjulia.so.1.0
make[1]: Leaving directory '/build/julia/src/julia/src'
make[1]: Entering directory '/build/julia/src/julia/ui'
Warning: git information unavailable; versioning information limited
    CC ui/repl.o
    LINK usr/bin/julia
make[1]: Leaving directory '/build/julia/src/julia/ui'
Creating usr/etc/julia/startup.jl
Copying in usr/share/man/man1/julia.1
/build/julia/src/julia/contrib/install.sh 755 /build/julia/src/julia/contrib/julia-config.jl /build/julia/src/julia/usr/share/julia/
make[1]: Entering directory '/build/julia/src/julia/base'
Warning: git information unavailable; versioning information limited
    PERL base/pcre_h.jl
    PERL base/errno_h.jl
    PERL base/build_h.jl.phony
    PERL base/file_constants.jl
    PERL base/uv_constants.jl
make[1]: *** [Makefile:168: /build/julia/src/julia/usr/lib/julia/libm.so] Error 1
make[1]: Leaving directory '/build/julia/src/julia/base'
make: *** [Makefile:63: julia-base] Error 2
make: Leaving directory '/build/julia/src/julia'

libunwind fails due to trying to do integer math on something the system version defines as a floating-point number (but if I locally patch out the whole if (!defined...) mess and unconditionally set JL_UNW_HAS_FORMAT_IP, it works fine and even passes the testsuite!):

make[1]: Entering directory '/build/julia/src/julia/src'
Warning: git information unavailable; versioning information limited
    CC src/jltypes.o
    CC src/gf.o
    CC src/ast.o
    CC src/typemap.o
    CC src/builtins.o
    CC src/module.o
    CC src/interpreter.o
    CC src/symbol.o
    CC src/dlload.o
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/module.c:7:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/gf.c:14:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/typemap.c:6:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/interpreter.c:9:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/ast.c:14:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/builtins.c:24:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:135: module.o] Error 1
make[1]: *** Waiting for unfinished jobs....
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/symbol.c:11:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:135: typemap.o] Error 1
make[1]: *** [Makefile:135: interpreter.o] Error 1
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/jltypes.c:14:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:135: symbol.o] Error 1
make[1]: *** [Makefile:135: gf.o] Error 1
make[1]: *** [Makefile:135: builtins.o] Error 1
make[1]: *** [Makefile:135: jltypes.o] Error 1
In file included from /usr/include/libunwind-x86_64.h:129,
                 from /usr/include/libunwind.h:25,
                 from /build/julia/src/julia/src/julia_internal.h:623,
                 from /build/julia/src/julia/src/dlload.c:10:
/build/julia/src/julia/src/julia_internal.h:627:35: error: floating constant in preprocessor expression
        (UNW_VERSION_MAJOR == 1 && UNW_VERSION_MINOR > 1))
                                   ^~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:135: dlload.o] Error 1
make[1]: *** [Makefile:135: ast.o] Error 1
make[1]: Leaving directory '/build/julia/src/julia/src'
make: *** [Makefile:69: julia-src-release] Error 2
make: Leaving directory '/build/julia/src/julia'
$ grep UNW_VERSION_MINOR /usr/include/libunwind-common.h 
#define UNW_VERSION_MINOR   2.1

openblas and lapack link fine, but fail the testsuite:

openblas-0.3.3-fail.log

Same results with just system openblas:

openblas-0.3.3-no-lapack-fail.log

LLVM of course fails dismally with:

    JULIA usr/lib/julia/sys-o.a
Generating precompile statements... 762 generated in  32.810796 seconds
Illegal inttoptr
      %scevgep9 = ptrtoint i32 addrspace(13)* %scevgep to i64
Illegal inttoptr
      %scevgep1011 = ptrtoint i32 addrspace(13)* %scevgep10 to i64

signal (6): Aborted
in expression starting at no file:0
gsignal at /usr/lib/libc.so.6 (unknown line)
abort at /usr/lib/libc.so.6 (unknown line)
runOnFunction at /build/julia/src/julia/src/llvm-gc-invariant-verifier.cpp:178
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /usr/lib/libLLVM-6.0.so (unknown line)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /usr/lib/libLLVM-6.0.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /usr/lib/libLLVM-6.0.so (unknown line)
operator() at /build/julia/src/julia/src/jitlayers.cpp:1182 [inlined]
jl_dump_native at /build/julia/src/julia/src/jitlayers.cpp:1191
jl_write_compiler_output at /build/julia/src/julia/src/precompile.c:84
jl_atexit_hook at /build/julia/src/julia/src/init.c:233
main at /build/julia/src/julia/ui/repl.c:234
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
_start at /build/julia/src/julia/usr/bin/julia (unknown line)
Allocations: 57622075 (Pool: 57611147; Big: 10928); GC: 126
/bin/sh: line 1: 30109 Aborted                 (core dumped) /build/julia/src/julia/usr/bin/julia -O3 -C "x86-64" --output-o /build/julia/src/julia/usr/lib/julia/sys-o.a.tmp --startup-file=no --warn-overwrite=yes --sysimage /build/julia/src/julia/usr/lib/julia/sys.ji /build/julia/src/julia/contrib/generate_precompile.jl /build/julia/src/julia/usr/lib/julia/sys.ji

But I'm sure you know and expect that, you have locally patched llvm in order to fix it. What's more alarming to me is that we were confidently assured that julia 1.0 would finally actually work with a system llvm, which has so far failed to be true.

I tried to see if I could apply the patches in question to llvm 6.0.1 but first they didn't apply cleanly, and then attempting to backport related patches for our system llvm package, resulted in llvm failing to build -- at which point I rapidly gave up.

Apparently, julia has no interest any time soon in being remotely easy to compile with a system llvm? Whatever happened to "We just have to nominate them" for llvm 6.0.1? I'm no expert at llvm, and our llvm packager is distinctly unimpressed with this whole trainwreck and refuses to "babysit llvm patches for julia".

Is this going to be the same story with llvm 7 once that promised rainbow appears?

Most helpful comment

As noted in the README, we require patched versions of our dependencies, because our dependencies have bugs. We do our best to upstream patches, but that takes time, esp if you're only using major releases.

we were confidently assured that julia 1.0 would finally actually work with a system llvm, which has so far failed to be true.

I'm sorry if you felt misled, but we were never in a position to promise this, and to the best of my knowledge we didn't.

Separately, this issue comes across as very ranty. I appreciate that your intention is to package julia for Arch Linux, but comments like

Apparently, julia has no interest any time soon in being remotely easy to compile with a system llvm? [...] Is this going to be the same story with llvm 7 once that promised rainbow appears?

are extremely off putting and make me less likely to want to help you fix these issues.

With respect to https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5, as mentioned there we're in the process of addressing these issues, but our first priority is getting it right for the generic binaries, second users building from source locally, and third integration with system package managers. We have rough plans for the third, including allowing distro packagers to re-use binaries that they already ship (assuming they are compatible), but we haven't gotten to that yet. On that point, I would ask for a bit of patience.

All 21 comments

The fact that it's almost impossible to build julia with any system dependencies, on top of which shenanigans like https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5 occur, makes it seem increasingly likely that julia's plan for Linux distribution is to ship a small linux operating system inside a VM, in order to run successfully.

What's next, are you going to bootstrap your own gcc build so that thirdparty ecosystem packages can reliably depend on the private libgfortran.so version?

Oh, also when setting up the testsuite to actually run, your make test target doesn't let me skip tests, it just runs them all. Which is only problematic because apparently you dont run them all either -- your CI slips the sockets test. I've reimplemented the testsuite that way for now, but it would be nice to know if that's actually expected to happen, and if so, what is the Makefile target for?

As noted in the README, we require patched versions of our dependencies, because our dependencies have bugs. We do our best to upstream patches, but that takes time, esp if you're only using major releases.

we were confidently assured that julia 1.0 would finally actually work with a system llvm, which has so far failed to be true.

I'm sorry if you felt misled, but we were never in a position to promise this, and to the best of my knowledge we didn't.

Separately, this issue comes across as very ranty. I appreciate that your intention is to package julia for Arch Linux, but comments like

Apparently, julia has no interest any time soon in being remotely easy to compile with a system llvm? [...] Is this going to be the same story with llvm 7 once that promised rainbow appears?

are extremely off putting and make me less likely to want to help you fix these issues.

With respect to https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5, as mentioned there we're in the process of addressing these issues, but our first priority is getting it right for the generic binaries, second users building from source locally, and third integration with system package managers. We have rough plans for the third, including allowing distro packagers to re-use binaries that they already ship (assuming they are compatible), but we haven't gotten to that yet. On that point, I would ask for a bit of patience.

stress test failure seems strange.
libm build issue could be a real bug.
libunwind and llvm are both upstream bugs so ask them why they haven't got a new release yet.
blas issue appears to be not actually using openblas (I used to get that error before making sure I'm actually loading openblas).

are extremely off putting and make me less likely to want to help you fix these issues.

On the other hand, I explicitly receive the strong impression that you all actively find the existence of distro packages to be distasteful and unsupported in the strongest terms. That makes me feel like we're fools (and unwelcome ones at that) for daring to try packaging it.

stress test failure doesn't really seems strange.

Yes -- I sometimes get nondeterministic failures there, so it takes 3 or 4 times to rebuild the package before it works. But in the case of pcre2 and mpfr it is consistently failing.

libunwind and llvm are both upstream bugs so ask them why they haven't got a new release yet.

Actually this libunwind thing seems to be remarked upon by https://github.com/JuliaLang/julia/pull/29082 (but it's still float comparison, whether you use > or !=)

As for llvm, they did release a new version, which is why I asked:

Whatever happened to "We just have to nominate them" for llvm 6.0.1?

but it's still float comparison, whether you use > or !=

It's a libunwind bug, whether any attempt is made to workaround it or not. https://github.com/libunwind/libunwind/issues/30

they did release a new version

The point release isn't the issue. We are hitting bugs from them faster than they are making releases.

distasteful and unsupported in the strongest terms

Correct and that is a statement of the fact rather than a statement of what could be done.
Given the bugs and misalignment in release schedule with LLVM, we don't expect being able to include all necessary patches in the latest point release and neither do we expect distributions to be willing to carry all the LLVM patches which necessarily mean that their julia package will likely be broken (and again as a fact, many of them are). Therefore, we have to strongly discourage users from using distro packages at this time after getting many reports about broken features.
Those messages are also only for the users and distro maintainers are still more than welcome to include dependencies that have our bug fix.

@eli-schwartz I've been able to package Julia in Fedora recently. It's annoying that I haven't been able to use neither the system's libunwind nor the system's mpfr (not sure why for the latter, need to investigate), but it works. EDIT: of course I don't use the system LLVM either, since the maintainer of that package didn't think applying Julia patches to it was a good idea. I haven't had problems with PCRE2. See https://src.fedoraproject.org/rpms/julia/blob/master/f/julia.spec if that can help.

On the other hand, I explicitly receive the strong impression that you all actively find the existence of distro packages to be distasteful and unsupported in the strongest terms. That makes me feel like we're fools (and unwelcome ones at that) for daring to try packaging it.

I wouldn't say distasteful, but your impression that we consider it unsupported iz correct. This is primarily for three reasons:

  1. Many distros ship broken versions of julia (often because of missing upstream patches, or occasionally additional patches that cause new bugs), which frustrates users and increases support volume.
  2. Many distros' release schedule takes so long that by the time the packages make it in, the packaged julia version is no longer supported, making those packages a bit useless for a fast moving project like julia (I appreciate that Arch moves faster and this is probably less applicable with 1.0, but still).
  3. Distro packaging (esp for packages) is incompatible with our reproducibility goals (we want users to be able to go back years later and get the exact same code that they were using when they originally wrote the code).

As a result of this, supporting distro package users is third priority, behind the generic binaries (first priority) and people building from source (second priority). We still have a bunch of work to do to fix those two, but we will get to improved distro packager support eventually.

I'd suggest filing separate issues about the libm and the stress failures. I don't see any PCRE-related failure (the make output mixes unrelated things AFAICT, -j1 should help).

Also, USE_SYSTEM_LIBM=0 USE_SYSTEM_OPENLIBM=1 should work if Arch includes openlibm, and that's the recommended approach to ensure computations give exactly the same results as the official binaries.

I'm hopeful that the LLVM patching process will eventually converge and we'll be compatible with some released version of LLVM. If that's all that's required, we might get there. However the problem in the past has been that distros will only let us use whichever LLVM version they picked, making compatibility far less likely. From our perspective, we're being good citizens by fixing bugs and upstreaming the patches. If we could increase compatibility by throwing in a few #ifdefs we would, and have done so in the past. But for most of these issues that is not practical.

I think at this point distros accept to package multiple LLVM versions, since Julia is not the only package which needs a specific version (e.g. Haskell or Rust also do). But it would be really nice to get all patches upstreamed and included in an LLVM point release, so that we can share a single versioned LLVM package. If the LLVM release process doesn't allow that, we should complain.

But it would be really nice to get all patches upstreamed and included in an LLVM point release, so that we can share a single versioned LLVM package. If the LLVM release process doesn't allow that, we should complain.

I'm pretty sure the llvm process does allow that, since I mentioned it in my initial comment here, and linked to https://github.com/JuliaLang/julia/pull/26626#issuecomment-376587822

Attempting to build julia using the system llvm completely fails for entirely unexpected reasons now that the system llvm6 is not the one pointed to by the /usr/lib/libLLVM.so symlink -- because Arch Linux actually has no problem providing versioned packages when needed as dependencies for things.

So, https://github.com/JuliaLang/julia/pull/29446

So, it turns out the failures when using the system openblas, are due to the fact that our openblas package is built with NO_CBLAS=1.

This makes sense I think, because we already provide libcblas.so which is obviously linked to libblas.so (and on Arch Linux, openblas is a drop-in replacement for the reference blas, providing a symlink to libblas.so as well as package provides/conflicts markers for blas). I think it's reasonable for packages which need cblas symbols, to link/dlopen cblas in order to get them instead of just sort of assuming every openblas lib out there has a convenience copy of cblas statically linked into it.

Julia already expects and copes with lapack being external to openblas, as build_h.jl contains both libblas_name and liblapack_name. I think it should do the same with libcblas...

In the meantime I got the package to build and pass the tests by lying and saying:

override LIBBLAS = -lcblas
override LIBBLASNAME = libcblas

Feel free to file a PR to support this with a less hacky solution.

Since we build our own BLAS by default, could we link it explicitly as cblas instead of as blas since technically cblas is what we need to link against? That seems like it would solve this problem without breaking our usage of our own OpenBLAS build.

Well, I'm currently testing this as a solution, and then building with USE_SYSTEM_CBLAS: https://github.com/eli-schwartz/julia/commit/0c442318196389d653ee21eba65d8c4f7beb72a0

Related on building from the source; I'm encountering test failures in Sockets and Distributed when building from the source on Nixos. When the failing tests are reordered such that the failing ones are at the top, everything passes just fine, so it appears some of the tests affect the state of the other ones.

I have that issue too, which I fixed by emulating one of the the CI settings (all of which explicitly don't use the apparently unsupported make test target) that disables the sockets test in order to claim success. Presumably those tests don't matter.

Better file a separate issue. A big issue with lots of problems doesn't help tracking progress.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

wilburtownsend picture wilburtownsend  ยท  3Comments

iamed2 picture iamed2  ยท  3Comments

m-j-w picture m-j-w  ยท  3Comments

ararslan picture ararslan  ยท  3Comments

Keno picture Keno  ยท  3Comments