Julia: test of Sockets fails on mac, with julia 1.3.0-rc4

[daviehh]

]test Sockets works on 1.2.0 but on 1.3.0-rc4 stops after the "put! in listenany(defaultport)". Running the test code manually, it looks like the testset UDPSocket hangs:

https://github.com/JuliaLang/julia/blob/220681f21fc5aaf83b7c3aee0a2b8d6186805b2c/stdlib/Sockets/test/runtests.jl#L274

errors with ERROR: IOError: send: message too long (EMSGSIZE), which is caught, then

https://github.com/JuliaLang/julia/blob/220681f21fc5aaf83b7c3aee0a2b8d6186805b2c/stdlib/Sockets/test/runtests.jl#L284

hangs

[ViralBShah]

versioninfo()?

[daviehh]

Julia Version 1.3.0-rc4.1
Commit 8c4656b97a (2019-10-15 14:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

may be related to macOS version: mine is 10.15 (19A602), whereas travis test running on 10.14 is ok

[ararslan]

may be related to macOS version

Could be. I can't reproduce on Mojave (10.14).

[dkarrasch]

I can reproduce on macOS 10.15.

julia> versioninfo()
Julia Version 1.3.0-rc4.1
Commit 8c4656b97a (2019-10-15 14:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

[KristofferC]

Can someone try 1.2 with this Mac version.

[daviehh]

@KristofferC
While in julia 1.2.0 ]test Sockets runs fine, it's not julia 1.2 per se because the tests for UV_EMSGSIZE is added for 1.3 and not present in 1.2 (https://github.com/JuliaLang/julia/blob/release-1.2/stdlib/Sockets/test/runtests.jl). The UV_EMSGSIZE test also fails on 1.2 on macOS 10.15 on my end.

On macOS 10.15, for julia 1.3.0-rc4, changing the line
https://github.com/JuliaLang/julia/blob/220681f21fc5aaf83b7c3aee0a2b8d6186805b2c/stdlib/Sockets/test/runtests.jl#L281

to

msg = Vector{UInt8}("fedcba9876543210"^36)

(and the corresponding test to @test fetch(tsk) == Vector{UInt8}("fedcba9876543210"^36))) fixes this problem; according to https://github.com/JuliaLang/julia/blob/220681f21fc5aaf83b7c3aee0a2b8d6186805b2c/stdlib/Sockets/test/runtests.jl#L263

maybe now in macOS 10.15 sending empty msg does not work, and one needs to send minimum reassembly buffer?

Self-contained relevant test:

using Sockets, Random, Test

defaultport = rand(2000:4000)

let localhost = getaddrinfo("localhost")
    global randport
    randport, server = listenany(localhost, defaultport)
    @async connect("localhost", randport)
    s1 = accept(server)
    @test_throws ErrorException("client TCPSocket is not in initialization state") accept(
        server,
        s1,
    )
    @test_throws Base._UVError("listen", Base.UV_EADDRINUSE) listen(randport)
    port2, server2 = listenany(localhost, randport)
    @test randport != port2
    close(server)
    close(server2)
end

a = UDPSocket()
b = UDPSocket()
bind(a, ip"127.0.0.1", randport)
bind(b, ip"127.0.0.1", randport + 1)

msg = Vector{UInt8}("1234"^16377)
# @test_throws(
#     Base._UVError("send", Base.UV_EMSGSIZE),
#     send(b, ip"127.0.0.1", randport, msg),
# )
pop!(msg)
tsk = @async recv(a)

try
    send(b, ip"127.0.0.1", randport, msg)
catch ex
    if !(ex isa Base.IOError && ex.code == Base.UV_EMSGSIZE) ||
       Sys.islinux() || Sys.iswindows()
                    # this is allowed failure on some platforms which might further restrict
                    # the maximum packet size being sent (even locally), such as BSD's `sysctl net.inet.udp.maxdgram`
        rethrow()
    end
    # empty!(msg)
    msg = Vector{UInt8}("fedcba9876543210"^36) # The minimum reassembly buffer size for IPv4 is 576 bytes
    send(b, ip"127.0.0.1", randport, msg) # check that the socket is still alive
end

@test fetch(tsk) == Vector{UInt8}("fedcba9876543210"^36))

I commented out the @test_throws part since julia v1.2 throws a different error string than 1.3-rc4

[vtjnash]

minimum there refers to the threshold: this is the smallest value permissible for the maximum message size

[daviehh]

@vtjnash Thanks! Looks like sending a non-empty msg is enough for the test to pass on macOS 10.15:

using Sockets, Random, Test

defaultport = rand(2000:4000)

let localhost = getaddrinfo("localhost")
    global randport
    randport, server = listenany(localhost, defaultport)
    @async connect("localhost", randport)
    s1 = accept(server)
    @test_throws ErrorException("client TCPSocket is not in initialization state") accept(
        server,
        s1,
    )
    @test_throws Base._UVError("listen", Base.UV_EADDRINUSE) listen(randport)
    port2, server2 = listenany(localhost, randport)
    @test randport != port2
    close(server)
    close(server2)
end

a = UDPSocket()
b = UDPSocket()
bind(a, ip"127.0.0.1", randport)
bind(b, ip"127.0.0.1", randport + 1)

msg = Vector{UInt8}("1234"^16377)
# @test_throws(
#     Base._UVError("send", Base.UV_EMSGSIZE),
#     send(b, ip"127.0.0.1", randport, msg),
# )
pop!(msg)
tsk = @async recv(a)

try
    send(b, ip"127.0.0.1", randport, msg)
catch ex
    if !(ex isa Base.IOError && ex.code == Base.UV_EMSGSIZE) ||
       Sys.islinux() || Sys.iswindows()
                    # this is allowed failure on some platforms which might further restrict
                    # the maximum packet size being sent (even locally), such as BSD's `sysctl net.inet.udp.maxdgram`
        rethrow()
    end
    # empty!(msg)
    msg = Vector{UInt8}("0")
    send(b, ip"127.0.0.1", randport, msg) # check that the socket is still alive
end

print(fetch(tsk) == Vector{UInt8}("0"))

[StefanKarpinski]

Marking as a 1.3-blocker until this gets fixed or determined to be not a blocker.

[daviehh]

Did some digging, looks like the reason is that upon receiving an empty udp message, the async fetch does not return until a non-empty message is received.

let this file be test_rx.jl:

using Sockets

a = UDPSocket()
bind(a, ip"127.0.0.1", 6000)
tsk = @async recv(a)

print(fetch(tsk))

and let this be test_tx.jl:

using Sockets

b = UDPSocket()
bind(b, ip"127.0.0.1", 6001)

msg = Vector{UInt8}()
send(b, ip"127.0.0.1", 6000, msg)

println("sent empty msg; press enter to send 1")
readline()

msg = Vector{UInt8}("1")
send(b, ip"127.0.0.1", 6000, msg)

Now run julia test_rx.jl in one terminal and julia test_tx.jl in another, test_rx continues to wait after the empty msg is sent. Pressing enter in test_tx to send a non-empty message, then test_rx's fetch returns the empty msg sent before.

[vtjnash]

Catalena changes some of the networking stack (deprecates Network Kernel Extension), so perhaps they introduced an error here?

[daviehh]

@vtjnash quickly glancing through, looks like julia is using libuv for this? The issue may not be entirely the os's error: quick test of similar code to send empty udp message works ok with python, e.g.

receiving:

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(("127.0.0.1", 6000))

while True:
    data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
    print("received message:", data)

sending:

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(b"", ("127.0.0.1", 6000))

Not sure how important it is to people to send empty udp messages. A "quick and dirty" fix would be changing the empty msg in the test to a non-empty one. Since macOS is listed as tier 1 support in https://julialang.org/downloads/, meaning all test should pass, so leaning towards fixing this for 1.3 -- just my 2 cents :-)

[StefanKarpinski]

Sending empty UDP messages should work. I wonder if this is a libuv bug...

[JeffBezanson]

This seems to be due to a change in mac os 10.15 and not a change on our end, so possibly not release blocking.

[Keno]

I've submitted a bug report to Apple as FB7503750.

[Keno]

@staticfloat reports this is fixed upstream in 10.15.4.