Tokio: UnixDatagram: no way to signal EOF to the peer

[jean-airoldie]

Calling recv on a tokio_net::UnixDatagram socket after calling Shutdown::Read will block forever.

use tokio_net::UnixDatagram;
use std::net::Shutdown;

#[tokio::test]
async fn shutdown_unblocks_recv() -> io::Result<()> {
    let (mut dgram1, _dgram2) = UnixDatagram::pair()?;

    dgram1.shutdown(Shutdown::Read)?;
    // This blocks forever.
    let err = dgram1.recv(&mut vec![]).await.unwrap_err();
    Ok(())
}

From my understanding, this is caused by the fact that calling recv on a shutdown read socket returns WouldBlock, so it cannot be differentiated with an empty buffer.

Platform

linux 5.0.0-31-generic

Version

master cfc15617a5247ea780c32c85b7134b88b6de5845

[jean-airoldie]

Moreover, Shutdown::Write on the peer causes the same behavior.

use tokio_net::UnixDatagram;
use std::net::Shutdown;

#[tokio::test]
async fn shutdown_write_unblocks_recv() -> io::Result<()> {
    let (mut dgram1, dgram2) = UnixDatagram::pair()?;

    dgram2.shutdown(Shutdown::Write)?;
    let mut buf = vec![];
    // This blocks forever.
    let err = dgram1.recv(&mut buf).await.unwrap_err();
    Ok(())
}

[jean-airoldie]

So essentially there is no sane way to signal EOF to the peer. We can't even use our own custom EOF message because its possible that the queue is full and the message cannot be sent.

[DoumanAsh]

This is actually strange because it should read with 0.

I'm not on unix, but are you sure it is just would blocking? I would suspect it is not getting ready instead
Under hood it uses mio-uds which in turn uses std::net::Unix* types

I would suggest to strace syscalls I guess?

[jean-airoldie]

I'll test out the std types but i think this might be the expected behavior since datagrams sockets are essentially connectionless.

From the recv manpage:

When a stream socket peer has performed an orderly shutdown, the return value will be 0 (the traditional "end-of-file" return).

Datagram sockets in various domains (e.g., the UNIX and Internet domains) permit zero-length datagrams.  When such a datagram is received, the return value is 0.

This is kinda unexpected for unix datagram sockets generated via the pair method since conceptually its an exclusive connection. But that might be expected behavior.

In that case a application level EOF would be needed to indicate a gracefull shutdown.

[DoumanAsh]

Well, it is still technically supposed to return 0 instead of blocking, if I read it correctly, so it is strange for it to block

But yeah, zero length is not necessary indication of EOF in UDS case

[jean-airoldie]

Using the std UnixDatagram return Ok(0) in blocking mode, but Err(WouldBlock) in nonblocking mode. Since mio_uds uses the nonblocking mode, this is likely what is causing the issue.

[jean-airoldie]

In my case I think that making the socket blocking after shutdown would make sense since im calling shutdown Write. Indeed in that case a read operations are guaranteed to be not block because if they did it would return Ok(0) because of EOF. I would also have prevent zero sized datagrams. But its probably not a good general solution.

[DoumanAsh]

@jean-airoldie could you link std code where you found it?

I'm seem to be unable to find anything like that

also having strace output with recv return value would be nice

[jean-airoldie]

As for the code:

use std::{os::unix::net::UnixDatagram, net::Shutdown, io};

#[test]
fn shutdown_unblocks_blocking_recv() -> io::Result<()> {
    let (dgram1, dgram2) = UnixDatagram::pair()?;
    let mut buf = vec![0u8; 8];

    dgram1.shutdown(Shutdown::Read)?;
    let read = dgram1.recv(&mut buf)?;
    assert!(read == 0);

    Ok(())
}

#[test]
fn shutdown_unblocks_nonblocking_recv() -> io::Result<()> {
    let (dgram1, dgram2) = UnixDatagram::pair()?;
    let mut buf = vec![0u8; 8];
    dgram1.set_nonblocking(true)?;

    dgram1.shutdown(Shutdown::Read)?;
    // This errors with `Error: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }`
    let read = dgram1.recv(&mut buf)?;
    assert!(read == 0);

    Ok(())
}

The strace will follow shortly.

[jean-airoldie]

This is the abbreviated strace for the shutdown_unblocks_nonblocking_recv function.

[...]
write(1, "\nrunning 1 test\n", 16
running 1 test
)      = 16
futex(0x7f133738b0c8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f13361af000
mprotect(0x7f13361b0000, 2097152, PROT_READ|PROT_WRITE) = 0
clone(child_stack=0x7f13363aeef0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f13363af9d0, tls=0x7f13363af700, child_tidptr=0x7f13363af9d0) = 4991
futex(0x5591966e4a98, FUTEX_WAIT_PRIVATE, 0, {tv_sec=59, tv_nsec=999703287}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x5591966e4a20, FUTEX_WAKE_PRIVATE, 1) = 0
write(1, "test shutdown_unblocks_nonblocki"..., 44test shutdown_unblocks_nonblocking_recv ... ) = 44
write(1, "\33[31mFAILED\33[m\17", 15FAILED)   = 15
write(1, "\n", 1
[...]

[DoumanAsh]

@jean-airoldie sorry I kinda forgot about it...
For code, I meant std lib code where you see that 0 return would get treated as would block, not your examples

[jean-airoldie]

I'm not sure I'm following, my example is std lib code and if we check the strace we see that WouldBlock gets returned instead of 0.

[DoumanAsh]

Sorry for confusing, I think we might have some misunderstanding. You mentioned:

Using the std UnixDatagram return Ok(0) in blocking mode, but Err(WouldBlock) in nonblocking mode.

And I thought you meant that some code in std lib implementation of UnixDatagram is responsible for it.

[jean-airoldie]

Oh sorry, I meant that the posix socket is actually returning -1 with EAGAIN instead of 0, which is equivalent to Err(WouldBlock) instead of Ok(0).

So I don't think there is an ez fix to this one.

[DoumanAsh]

Ah ok, well it is actually funny, because I'd expect shutdown to result in immediate return 0 (because there can be no more).

But now that I think about it, it might be possible to see shutdown for TCP, but not of UDS or UDP, because of their connectionless nature :(

[Leo1003]

Here is a piece of codes in Linux kernel
net/core/datagram.c:291

struct sk_buff *__skb_recv_datagram(struct sock *sk, unsigned int flags,
                    void (*destructor)(struct sock *sk,
                               struct sk_buff *skb),
                    int *off, int *err)
{
    struct sk_buff *skb, *last;
    long timeo;

    timeo = sock_rcvtimeo(sk, flags & MSG_DONTWAIT);

    do {
        skb = __skb_try_recv_datagram(sk, flags, destructor, off, err,
                          &last);
        if (skb)
            return skb;

        if (*err != -EAGAIN)
            break;
    } while (timeo &&
        !__skb_wait_for_more_packets(sk, err, &timeo, last));

    return NULL;
}
EXPORT_SYMBOL(__skb_recv_datagram);

As you can see, if the non-blocking flag is set, then it would hit if (*err != -EAGAIN) and break earlier. But in blocking mode, it would run into __skb_wait_for_more_packets() finally. This function check the shutdown flags and we can get 0 in the return code of recv().

I'm not sure if this is a bug or an intended behavior. And other unix system's behavior.

[Leo1003]

As for peer shutdown, it's a no-op in Datagram type...

net/unix/af_unix.c:2445

    unix_state_lock(sk);
    sk->sk_shutdown |= mode;
    other = unix_peer(sk);
    if (other)
        sock_hold(other);
    unix_state_unlock(sk);
    sk->sk_state_change(sk);

    if (other &&
        (sk->sk_type == SOCK_STREAM || sk->sk_type == SOCK_SEQPACKET)) {

        int peer_mode = 0;

        if (mode&RCV_SHUTDOWN)
            peer_mode |= SEND_SHUTDOWN;
        if (mode&SEND_SHUTDOWN)
            peer_mode |= RCV_SHUTDOWN;
        unix_state_lock(other);
        other->sk_shutdown |= peer_mode;
        unix_state_unlock(other);
        other->sk_state_change(other);
        if (peer_mode == SHUTDOWN_MASK)
            sk_wake_async(other, SOCK_WAKE_WAITD, POLL_HUP);
        else if (peer_mode & RCV_SHUTDOWN)
            sk_wake_async(other, SOCK_WAKE_WAITD, POLL_IN);
    }

[DoumanAsh]

@Leo1003 yeah it makes sense that shutdown is noop for uds, but it is really counter intuitive that it EGAIN in non-blocking mode...

I think there is not much to do in this case, if it is how kernel designed uds

[Leo1003]

Tested with FreeBSD using the following codes.
It returns Ok(0), which is in line with our expectations!

FreeBSD: 12.0-RELEASE r341666 GENERIC  amd64
Rustc: 1.38.0 (625451e37 2019-09-23) & 1.39.0 (4560ea788 2019-11-04)

use std::{os::unix::net::UnixDatagram, net::Shutdown, io};

fn main() -> io::Result<()> {
    let (dgram1, dgram2) = UnixDatagram::pair()?;
    let mut buf = vec![0u8; 8];
    dgram1.set_nonblocking(true)?;

    dgram1.shutdown(Shutdown::Read)?;
    // This errors with `Error: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }`
    let read = dgram1.recv(&mut buf)?;
    assert!(read == 0);
    println!("read = {}", read);

    Ok(())
}

I was unable to find the documentation defining the behavior of non-blocking uds. We need someone to determine if the behavior on Linux is a bug.

[Darksonn]

What's the status of this issue?

[Leo1003]

I think this is Linux kernel's problem. But I am not familiar with kernel development. Maybe someone can help us to report this issue.