Go: syscall: allow registration of new socket types for package net

[mdlayher]

Overview

At this time, there is no mechanism for socket types outside of the standard library to access the runtime network poller. This proposal, if accepted, would enable a resolution to issue #10565. This would enable packages outside of the standard library to take advantage of the runtime network poller, instead of implementing their own network polling mechanism.

Proposed Change

I propose adding a new API to package net which enables registration of arbitrary sockets for use with the runtime network poller. The design of this API is based upon a comment from @rsc found here: https://github.com/golang/go/issues/11492#issuecomment-117855313.

It seems to me that the net package should just keep using (and providing) only FileConn but perhaps we can put a registration mechanism in package syscall to let clients register converters between sockaddrs and net.Addr for non-standard sockaddr types.

This is what I was able to come up with after a little bit of experimentation. Parameter list is to be determined, but this is what I was able to get working with my prototype on a Linux system. Efforts will be made to make this mechanism as generic and cross-platform friendly as possible, but it may not be implemented immediately on non-UNIX platforms. From what I can tell, syscall.Sockaddr does appear to be available on all platforms.

package net

// Registration mechanism, perhaps called in init() or main() when a
// socket is first initalized.
func RegisterSocket(
    family int,
    sockaddr syscall.Sockaddr,
    addr Addr,
    convertSockaddr func(syscall.Sockaddr) Addr,
    convertNetAddr func(Addr) syscall.Sockaddr,
)

// Generic net.Conn and net.PacketConn implementation which embeds the
// internal net.conn type.  Checks for registered socket hooks to determine
// validity of sent and received net.Addr implementations.
type SocketConn struct {
    conn
}

Example

Using a modified version of package net, I was able to gain access to the runtime network poller and
simplify my raw sockets package code to something like the following:

// Called in init() in package raw
net.RegisterSocket(
    syscall.AF_PACKET,
    &syscall.SockaddrLinklayer{},
    &Addr{},
    // internal conversion functions for syscall.SockaddrLinklayer <-> raw.Addr
    convertSockaddr,
    convertNetAddr,
)

sock, _ := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, proto)
_ = syscall.Bind(sock, &syscall.SockaddrLinklayer{
    Protocol: pbe,
    Ifindex:  ifi.Index,
})
f := os.NewFile(uintptr(sock), "linklayer")
// c is type net.SocketConn, backed by raw socket (uses raw.Addr for addressing)
c := net.FilePacketConn(f)

Summary

The runtime network poller is an excellent mechanism, and enabling access to it will allow the future development of packages for raw ethernet sockets, netlink sockets, and other platform-specific socket types.

If this proposal is accepted, I'd happily seek guidance from @mikioh regarding creating the best possible API for this feature. In addition, this would enable me to contribute code from my raw ethernet socket package as a resolution to https://github.com/golang/go/issues/8432.

Questions and comments appreciated, and thanks for your time.

/cc @rsc @mikioh

[rsc]

Hi,

Thanks for the work you've put in to figuring out what's possible. Can you
say more about why these are in package net instead of package syscall?
It's a bit of a non-starter for package net's API to expose so much (really
anything) from package syscall. If this were in syscall it would be more
palatable, and at first glance I don't see why it can't be. But I haven't
tried, so probably I'm missing something.

Thanks.
Russ

On Tue, Mar 29, 2016 at 2:18 PM Matt Layher [email protected]
wrote:

_Overview_

At this time, there is no mechanism for socket types outside of the
standard library to access the runtime network poller. This proposal, if
accepted, would enable a resolution to issue #10565
https://github.com/golang/go/issues/10565. This would enable packages
outside of the standard library to take advantage of the runtime network
poller, instead of implementing their own network polling mechanism.

_Proposed Change_

I propose adding a new API to package net which enables registration of
arbitrary sockets for use with the runtime network poller. The design of
this API is based upon a comment from @rsc https://github.com/rsc found
here: #11492 (comment)
https://github.com/golang/go/issues/11492#issuecomment-117855313.

It seems to me that the net package should just keep using (and providing)
only FileConn but perhaps we can put a registration mechanism in package
syscall to let clients register converters between sockaddrs and net.Addr
for non-standard sockaddr types.

This is what I was able to come up with after a little bit of
experimentation. Parameter list is to be determined, but this is what I was
able to get working with my prototype on a Linux system. Efforts will be
made to make this mechanism as generic and cross-platform friendly as
possible, but it may not be implemented immediately on non-UNIX platforms.
From what I can tell, syscall.Sockaddr does appear to be available on all
platforms.

package net
// Registration mechanism, perhaps called in init() or main() when a// socket is first initalized.func RegisterSocket(
family int,
sockaddr syscall.Sockaddr,
addr Addr,
convertSockaddr func(syscall.Sockaddr) Addr,
convertNetAddr func(Addr) syscall.Sockaddr,
)
// Generic net.Conn and net.PacketConn implementation which embeds the// internal net.conn type. Checks for registered socket hooks to determine// validity of sent and received net.Addr implementations.type SocketConn struct {
conn
}

_Example_

Using a modified version of package net, I was able to gain access to the
runtime network poller and
simplify my raw sockets package https://github.com/mdlayher/raw code to
something like the following:

// Called in init() in package raw
net.RegisterSocket(
syscall.AF_PACKET,
&syscall.SockaddrLinklayer{},
&Addr{},
// internal conversion functions for syscall.SockaddrLinklayer <-> raw.Addr
convertSockaddr,
convertNetAddr,
)
sock, _ := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, proto)
_ = syscall.Bind(sock, &syscall.SockaddrLinklayer{
Protocol: pbe,
Ifindex: ifi.Index,
})f := os.NewFile(uintptr(sock), "linklayer")// c is type net.SocketConn, backed by raw socket (uses raw.Addr for addressing)c := net.FilePacketConn(f)

_Summary_

The runtime network poller is an excellent mechanism, and enabling access
to it will allow the future development of packages for raw ethernet
sockets, netlink sockets, and other platform-specific socket types.

If this proposal is accepted, I'd happily seek guidance from @mikioh
https://github.com/mikioh regarding creating the best possible API for
this feature.

Questions and comments appreciated, and thanks for your team.

/cc @rsc https://github.com/rsc @mikioh https://github.com/mikioh

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/15021

[mdlayher]

Hi Russ,

It appears to me that the runtime network poller functionality is exposed directly using the net.pollDesc type, which is then wrapped by net.netFD and net.conn. From what I can tell, it appears I'd have to duplicate or possibly move some code from package net to package syscall in order to enable accessing the runtime network poller there.

Specifically, I'm looking at net/fd_unix.go, and net/fd_poll_runtime.go. Is there an easier answer to this problem that I am not yet aware of?

Thanks for the quick reply, and I look forward to hearing back.

[rsc]

Your Example looks exactly as I think it should except for the fact that
the call is net.RegisterSocket instead of syscall.RegisterSocket. Can
RegisterSocket be moved into syscall? If not, why not? Thanks.

On Tue, Mar 29, 2016 at 3:08 PM Matt Layher [email protected]
wrote:

Hi Russ,

It appears to me that the runtime network poller functionality is exposed
directly using the net.pollDesc type, which is then wrapped by net.netFD
and net.conn. From what I can tell, it appears I'd have to duplicate or
possibly move some code from package net to package syscall in order to
enable accessing the runtime network poller there.

Specifically, I'm looking at net/fd_unix.go, and net/fd_poll_runtime.go.
Is there an easier answer to this problem that I am not yet aware of?

Thanks for the quick reply, and I look forward to hearing back.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/15021#issuecomment-203054121

[mdlayher]

I am in favor of moving it to package syscall instead, but because much of the low-level functionality to access the runtime network poller already exists in package net, I am not sure how to best access the network poller from package syscall without duplicating code or moving it to syscall entirely.

Can you suggest a viable approach to solving this problem? Perhaps the low level runtime network poller code in net could be moved to an internal package that both net and syscall use to access it?

In summary, yes, I am in favor of creating syscall.RegisterSocket instead, but I am simply not sure how to access the runtime network poller from package syscall instead of package net, where it resides now, without code duplication.

EDIT: in addition, since net imports syscall, but I would need the net.Addr type in syscall, this would create a circular dependency. Any thoughts on how to resolve this issue as well?

[rsc]

Can you point me at the code you changed? Maybe that will help me see why
RegisterSocket cannot easily move into syscall. Thanks.

On Tue, Mar 29, 2016 at 3:45 PM Matt Layher [email protected]
wrote:

I am in favor of moving it to package syscall instead, but because much
of the low-level functionality to access the runtime network poller already
exists in package net, I am not sure how to best access the network
poller from package syscall without duplicating code or moving it to
syscall entirely.

Can you suggest a viable approach to solving this problem? Perhaps the low
level runtime network poller code in net could be moved to an internal
package that both net and syscall use to access it?

In summary, yes, I am in favor of creating syscall.RegisterSocket
instead, but I am simply not sure how to access the runtime network poller
from package syscall instead of package net, where it resides now,
without code duplication.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/15021#issuecomment-203068706

[mdlayher]

Here's a link to the prototype I made to make this work. Obviously, it would need cleanup, proper locking, and probably should be wrapped in a type.

https://github.com/mdlayher/go/commit/2e7d77c17dbc830135ca35eb5617e073c8fc717b

[mdlayher]

Hey @rsc , did you get a chance to look at the prototype I created above? I'd love to hear your thoughts on it. Thanks for your time.

[mikioh]

@mdlayher,

I didn't have a look at your proposal/prototype carefully, and probably have no spare time for this feature (including #10565 opened by me) during Go 1.7 development cycle, but a bit. I think that the basic requirements for this feature look like the following:

1. Need to use the net package
   
   
   
   • As a glue to crypto/tls and net/http packages
   
   
2. Need to have a replacement for the syscall package
   
   
   
   • No use of types in the syscall package is mandatory because the syscall package is locked down and will be deprecated
   
   
   • The new package should provide a type that represents some sort of tuple known as a socket
   
   
   • The new package should provide flexibility for system adaptation not only for the existing kernels but for upcoming kernels that have fancy new network sockets
   
   
3. Need to have a replacement for the runtime package
   
   
   
   • This requirement might be optional, but having a way to use runtime-integrated network poller would be better
   
   
   • The new package should provide basic event notification mechanism on the runtime-integrated network poller
   
   

10565 just covers R1, and perhaps your proposal too.

I'd like to see a design that fulfils the above requirements, at least R1 and R2. Otherwise, the design would be the source of trouble in future, like "I cannot achieve my goal because of the lack of R2 (or both R2 and R3)." I suppose that a typical use case would be either R1+R2 or R2+R3; the former is for people who want to use the existing read/write methods in the net package with new socket types and the latter is for people who want to use new read/write methods with new socket types.

Thoughts?

[mdlayher]

@mikioh Thanks for replying. Your requirements seem sound, but I wouldn't even know where to begin covering R2 and R3 (mine only covers R1).

Happy to help out if you have any sketches of a proposed API or something similar, but I don't have the knowledge of the internals of package runtime to create a public interface to the runtime network poller, etc.

[minux]

FTR, no one says the syscall package will be deprecated. It's just frozen.
It should work fine for existing features and we even fixes bugs for
existing features. We just won't add new features or make significant
bugfixes (like BSD ABI broken changes.)

[mikioh]

FTR, https://golang.org/s/go1.4-syscall says that:

Note that we cannot clean up the existing syscall package to any meaningful extent because of the compatibility guarantee. We can freeze and, in effect, deprecate it, however.

not literally, but in effect.

[mikioh]

@mdlayher,

I think x/net repository is a good place for the new package implementing R2, because we can copy the package into src/vendor/golang.org/x/net/ and plumb it in the net package later.

[bradfitz]

I will investigate options a bit here.

[mdlayher]

Thanks, @bradfitz ! Looking forward to seeing what you come up with. Happy to help if there's anything I could do to assist you.

[bradfitz]

(Still on my plate to look at.)

[bradfitz]

The runtime already exposes a portable netpoll API to the net package, using uintptr as the portable FD type (like https://golang.org/pkg/os/#File.Fd), since that's what works between Windows and everything else.

I'd prefer to see a new package exposing a more Go-like (and less net/syscall-like) interface to that poller. I'd ideally like to see it done without any syscall.Sockaddr crud.

At the heart of epoll & friends is a simple API letting people register FDs they care about (and how: readability vs writability) and wait on a bunch of them at once.

We could have an API with uintptrs to watch/unwatch, a Go type for readable/writeable, and func callbacks when the edge triggers.

Does anybody want to explore prototyping that?

[mikioh]

I don't understand what you mean by "Go-like (and less net/syscall-like)." Can you please elaborate on that? What's your plan to accommodate user-specified socket descriptors with net package API access?

My guess is that your plan is just to provide a package like libuv for Node.js or mio for Rust. The package neither provide new IO methods such as {recv,send}mmsg nor replacements for broken APIs in frozen syscall and x/sys packages such as Sendfile for Darwin. Also the package doesn't help to inject user-specified socket descriptors with the net package via File{Conn,Listener,PacketConn} APIs because it doesn't address #10565. Is my understanding correct?

[elliotmr]

There is already an interface for the network poller defined, it is just that it is linked directly into the net package using go:linkname on private functions in runtime/netpoll.go. Is there some reason that the methods there (ServerInit, Open, Close, Reset, Wait, WaitCanceled, SetDeadline, and Unblock) can't be just exposed as public APIs?

This is exactly what I need for implementing an API over an AF_CAN socket type (https://groups.google.com/forum/#!topic/golang-nuts/l-1JD02_o0Q). Doing some digging this morning I found out that my options 2 and 3 are not possible because of this special link between the runtime and net packages.

[mdlayher]

FYI: mikioh recently submitted a series of three CLs which appear to be designed to implement a version of the solution proposed here.

https://go-review.googlesource.com/c/35995/
https://go-review.googlesource.com/c/35996/
https://go-review.googlesource.com/c/35994/

They are marked "DO NOT REVIEW" but I wanted to make sure they were mentioned for anyone keeping an eye on this issue.

[gopherbot]

CL https://golang.org/cl/36799 mentions this issue.

[gopherbot]

CL https://golang.org/cl/36800 mentions this issue.

[gopherbot]

CL https://golang.org/cl/45790 mentions this issue.

[rsc]

The original post says:

At this time, there is no mechanism for socket types outside of the standard library to access the runtime network poller.

This is no longer true. Other packages can now construct socket fds directly and then call os.NewFile to get a *os.File. Operations on that *os.File will use the network poller. Taking the example from the top:

sock, _ := syscall.Socket(syscall.AF_PACKET, syscall.SOCK_RAW, proto)
_ = syscall.Bind(sock, &syscall.SockaddrLinklayer{
    Protocol: pbe,
    Ifindex:  ifi.Index,
})
f := os.NewFile(uintptr(sock), "linklayer")
// c is type net.SocketConn, backed by raw socket (uses raw.Addr for addressing)
c := net.FilePacketConn(f)

If you just don't do the last line and use f in blocking i/o operations, those operations will use the network poller.

However, it's still not possible to turn a socket fd into a net.Conn/net.PacketConn/net.Listener (as in the last line), which you might want for interoperation with other software expecting a net.Conn or for timeout manipulation. That's what would be served by a new registry. The registry would need to maintain the association between (family int, sotype int) <-> (net string) and then also for each such association the corresponding (addr string)<->(sa sockaddr) conversion functions.

Given those, package net can introduce a new unexported net.Conn and net.Addr implementation that uses that functionality.

The CL I just posted (https://golang.org/cl/45790) compiles but has parts missing. Still, it should be concrete enough to make clear what I mean. I'm not 100% sure about the hooking into Dial as shown - maybe that's a bridge too far. If we did that we'd of course want to do Listen as well, but there's no DialPacket and I'm disinclined to add one just for this. I'm also a little scared of making the new types seem too much like standard types. It might be better to save Dial for standard types only.

Hooking into Dial in the example CL was really just a way to exercise the string->sockaddr conversion signature. It's possible we can drop it if we don't do Dial.

[rsc]

If someone wants to write a CL along those lines for Go 1.10 and see if it can be done fairly cleanly, I think we agree it's worthwhile. Leaving Dial out, at least for the first version, should make it unobjectionable.

I'm going to mark this approved assuming a sketch like in the CL.

[mdlayher]

I'm happy to give this a try. Thanks for the example CL!

[jstarks]

@rsc, it looks to me like os.NewFile will never enable the poller on Linux (it always passes pollable = false to poll.FD.Init()), so there is still no way to use the network poller for foreign fds. Am I missing something?

[ianlancetaylor]

@jstarks I think you're right; that too would have to change.

[mdlayher]

I'm not sure I'll be able to take this on. @mikioh, are you interested?

[acln0]

Hello. Is anyone working on this? If not, I am willing to do the work, be it for 1.11 or 1.12.

EDIT: In light of https://github.com/golang/go/commit/ea5825b0b64e1a017a76eac0ad734e11ff557c8e, which I only now discovered, perhaps this is less relevant now. But the issue of creating a net.Conn from such an *os.File still remains, for what it's worth. If there is nevertheless interest in the socket registry, I can try to take this on.

[mdlayher]

Please feel free to take this on!

[acln0]

I have thought about this some more. If I understand things correctly, I believe the socket type registry is only worth implementing if there are also going to be hooks into net.Dial (and net.Listen etc.) for it.

Since a non-blocking *os.File can now be created from a raw file descriptor, a hypothetical bluetooth package living outside the standard library can, schematically, do something like this:

package bluetooth

import (
    "net"
    "os"
    "time"

    "golang.org/x/sys/unix"
)

type Conn struct {
    laddr *addr
    raddr *addr
    fd    *os.File
}

func DialHCI(device uint16) (*Conn, error) {
    const (
        family  = unix.AF_BLUETOOTH
        sotype  = unix.SOCK_RAW | unix.SOCK_CLOEXEC | unix.SOCK_NONBLOCK
        proto   = unix.BTPROTO_HCI
        channel = unix.HCI_CHANNEL_CONTROL
    )
    rawFD, err := unix.Socket(family, sotype, proto)
    if err != nil {
        return nil, err
    }
    sa := unix.SockaddrHCI{
        Dev:     device,
        Channel: channel,
    }
    if err := unix.Bind(rawFD, &sa); err != nil {
        return nil, err
    }
    return &Conn{
        laddr: &addr{
            // put fields from sa here
        },
        fd: os.NewFile(uintptr(rawFD), "hci"),
    }, nil
}

func (c *Conn) LocalAddr() net.Addr {
    return c.laddr
}

func (c *Conn) RemoteAddr() net.Addr {
    return c.raddr
}

// Remaining net.Conn method implementations are elided for brevity, since all the
// calls are forwarded to their respective calls on c.fd, which work as expected.

type addr struct {
    // bluetooth-specific things in here
}

func (a *addr) Network() string { return "bluetooth" }

func (a *addr) String() string {
    // use fields on addr to construct this
    return ""
}

It's not difficult to imagine a similar construction for packet-oriented connections, with a straight-forward type assertion inside the library for methods like WriteTo. (EDIT: on second thought, perhaps this is not so easy after all...)

With this approach, users of the hypothetical bluetooth package would now have a usable, non-blocking net.Conn, without too much effort from either them or the package authors.

On the other hand, the nice property of being able to pass the strings returned by net.Addr.Network and net.Addr.String into net.Dial or net.Listen would probably be lost.

The existence of a socket type registry in the syscall package would potentially hide some of the details from end-users (and preserve the aforementioned property), so long as the generic net.Dial would be able to express specific requirements for the new socket. That being said, if users end up having to encode specific details about the socket in the address string passed to net.Dial (akin to the DSN passed to sql.Open), I believe the socket registry would not be an overall gain, and would lead to poorly-specified and obscure interfaces.

I hope I have not misunderstood the issues at hand. Moving forward, if the registry is still something we would like to implement, I can do the work for it. Otherwise, perhaps this issue could be closed in light of https://github.com/golang/go/commit/ea5825b0b64e1a017a76eac0ad734e11ff557c8e.

Thank you.

[acln0]

Sorry for talking to myself here. I have thought about this some more, and I have a more concrete proposal.

For 1.12, I plan to send a CL that adds the registration mechanism to package syscall, and changes package net to make FileConn and FilePacketConn work, when the *os.File argument to these functions refers to a socket, and the type of the socket was registered with syscall.

I think hooking into Dial (or other such intrusive things) can be discussed at a later time, if and when people use the new interface and provide feedback.

Does that sound good?

[gopherbot]

Change https://golang.org/cl/136595 mentions this issue: net, syscall: implement socket registration mechanism

[mikioh]

@mdlayher,

Do you think this issue is still necessary to implement minor protocols? If not, it might be better to close this issue or re-title for the x/sys repository because we can use os.File.SyscallConn on Go 1.12 and above (see #24331).

[mdlayher]

I believe @acln0 is investigating that option for the use cases I have! I don't have any further knowledge at this point.

[mdlayher]

Because of the ability to call *os.File.SyscallConn on a non-blocking file descriptor registered with the runtime poller via os.NewFile, it appears this functionality is no longer needed!

See https://github.com/mdlayher/netlink/issues/119 for details on how this has been implemented in my netlink sockets package.

At this point I am going to close this issue as unneeded, but I'll be sure to report back if we find any further limitations.

[ianlancetaylor]

Thanks for following up.