Go: net/http: add built-in graceful shutdown support to Server
by patrick.allen.higgins:
http.Server only offers flavors of Serve() without a way to shut them down. Closing the listener should make the server stop, but there seems to be a race in TLS servers where this does not always work. Further, in-progress requests cannot complete before exiting without some kind of wrapper for net.Listener and net.Conn which includes synchronization between Listener.Accept() and Conn.Close(). See https://groups.google.com/d/topic/golang-nuts/i4_kPK-rVxI/discussion for some details. A use-case for this is zero-downtime restarts: a master process opens a listener and then spawns a slave, passing it the listener. When the master receives a SIGHUP, it spawns a new slave. When the slave starts servicing requests, it signals the master and the master signals the old slave to shutdown. The old slave must shutdown gracefully to prevent service interruption.
gopherbot
All 45 comments
Sounds familiar... http://tomayko.com/writings/unicorn-is-unix
alberts
on 30 Jan 2013
You mention a few separate issues. 1. net/http doesn't offer a convenience function for net.Listen followed by http.Serve. 2. closing the underlying net.Listener doesn't shut down the http.Serve loop (because of a race condition?) 3. there's no mechanism to gracefully shut down an http.Serve loop, waiting for any in-flight requests to complete I think the second point is a priority. The first doesn't seem very interesting to me. The third point is a feature that we should explore further. This issue should be about the third point only. Can you please file issues for the other two points? I would especially like as much information as you have about the second point. Thanks
_Status changed to Accepted._
adg
on 30 Jan 2013
_Labels changed: added priority-later, go1.1maybe, packagechange, removed priority-triage, go1.1._
adg
on 30 Jan 2013
Comment 4 by patrick.allen.higgins:
I will try to boil down a test case for the second point. I didn't file an issue for it because fixing the third point would likely fix the second as well.
gopherbot
on 31 Jan 2013
Comment 5 by patrick.allen.higgins:
I created issue #4737, which I believe is related to the second point.
gopherbot
on 31 Jan 2013
Comment 6 by patrick.allen.higgins:
My proposal for this issue is to put an optional *sync.WaitGroup into http.Server and modify (*Server).Serve to call Add(1) and Done() around the creation of servicing goroutines. That would accomplish what my net.Listener wrappers are doing.
gopherbot
on 2 Feb 2013
robpike
It would be nice to have a Server.CloseIdleConnections (to kill keep-alive connections) much like we have Transport.CloseIdleConnections for the Client. There isn't a way for an external library to do this since the state is not exposed.
daaku
on 9 Sep 2013
Comment 16 by [email protected]:
After making some headway on this problem outside the standard library I decided to take a stab at it in net/http itself. https://golang.org/cl/67730046/
gopherbot
on 24 Feb 2014
Sent out https://golang.org/cl/69260044/ for discussion. It should be sufficient to implement shutdown outside of net/http, with varying levels of gracefulness.
_Labels changed: added release-go1.3maybe, removed priority-later, release-none._
bradfitz
on 27 Feb 2014
Also sent out https://golang.org/cl/69670043 for disabling keep-alives (sending "Connection: close" in Handlers)
bradfitz
on 28 Feb 2014
This issue was updated by revision 67b8bf3e32ec0bcc79453caeea9595a3ca03692.
This allows for all sorts of graceful shutdown policies, without picking a policy (e.g. lameduck period) and without adding lots of locking to the server core. That policy and locking can be implemented outside of net/http now. LGTM=adg R=golang-codereviews, josharian, r, adg, dvyukov CC=golang-codereviews https://golang.org/cl/69260044
bradfitz
on 28 Feb 2014
This issue was updated by revision 916682ea367a290da9392ef38016af5b8fd6a3b.
LGTM=adg, josharian R=adg, josharian, r CC=golang-codereviews https://golang.org/cl/69670043
bradfitz
on 28 Feb 2014
This issue was updated by revision 7124ee59d18fabe5494227b19250b4040a4aa8b.
LGTM=bradfitz R=bradfitz CC=golang-codereviews https://golang.org/cl/70410044 Committer: Brad Fitzpatrick
bradfitz
on 2 Mar 2014
Status update: Go 1.3 will have all the necessary net/http changes (Server.ConnState and Server.SetKeepAlivesEnabled) for other people to implement graceful shutdown properly, outside the net/http package. But because there are various policy questions of how quickly and how politely to shut down a server, we're going to refrain from unilaterally making that policy decision now (in the form of e.g. a Server.Close method) and instead make people pick their own policy for now. It's possible in the future (after some experience using different shutdown strategies in Go 1.3) we might see the common patterns and add a convenience shutdown/close method, perhaps with a timeout or options, once we know what the set of options are. So punting this to Go 1.4, even though most of the work will already be in Go 1.3.
_Labels changed: added release-go1.4, removed release-go1.3maybe._
bradfitz
on 5 Mar 2014
This didn't happen for 1.4. Maybe 1.5.
_Labels changed: added release-go1.5, removed release-go1.4._
rsc
on 16 Sep 2014
There have been no proposals during the Go 1.5 dev cycle. I trust people are still happy with the ConnState hooks added in Go 1.3, as evidenced by its use in:
I'm changing this to Milestone "Unplanned", but if somebody wants to see this in Go 1.6, please make a proposal before ~Feb 2016.
bradfitz
on 30 Apr 2015
How do I make a proposal? I'd love to have the ability to call Close() on an http server object. It's weird that I can't do it already and have to use a third party package to do that.
ghost
on 17 Jul 2015
bradfitz
on 17 Jul 2015
Until #9478 is fixed or something like CloseIdleConnections is added, the ConnState hook is of little use, since a graceful restart will still have to wait for all idle connections to timeout or be used once (and be shut by SetKeepAlivesEnabled). Correct?
FiloSottile
on 4 Aug 2016
Why do you have to wait for idle connections? If they're idle, just close them or restart your process.
bradfitz
on 4 Aug 2016
Oh, I was failing to see the obvious answer: once you only have Idle connections, exit.
Thanks, nevermind.
FiloSottile
on 4 Aug 2016
(Deleted some +1 comments per our comment policy)
Please click the emoji thumbs-up at the top of threads to express your desires. Or write a proposal & prototype.
bradfitz
on 22 Aug 2016
Any progress on this issue?
leakingtapan
on 29 Oct 2016
No, but I think a plan has started to form. It'll be a blocking:
func (s *Server) Shutdown(context.Context) error {}
func (s *Server) Close() error {}
Where the first one is graceful and takes a context (so you can abort early if it's taking too long), and the second one is forced and immediate. It would only return an error to fit the normal io.Closer signature, but would basically always return nil, even if it broke a bunch of in-use connections.
The Go 1.8 dev tree closes Sunday night, so I'll mark this as Go 1.9.
bradfitz
on 29 Oct 2016
Is there actually any point in implementing io.Closer? I don't think that an HTTP server really falls into that category, or that one would want to use it interchangeably with other io.Closers
dominikh
on 29 Oct 2016
@dominikh Given that Listener.Close returns an error, wouldn't you want to pass that on?
cespare
on 29 Oct 2016
@cespare Good point. I was thrown off by Brad saying that it would always return nil.
dominikh
on 29 Oct 2016
"but would basically always return nil" :-)
Yeah, passing on the net.Listener(s)' return value makes sense.
bradfitz
on 29 Oct 2016
CL https://golang.org/cl/32329 mentions this issue.
gopherbot
on 30 Oct 2016
I managed to implement this in a few hours before the freeze, and people have been wanting this forever, so Go 1.8 it is.
bradfitz
on 30 Oct 2016
CL https://golang.org/cl/32412 mentions this issue.
gopherbot
on 30 Oct 2016
I may be missing something, but it looks like hijacked connections are just removed from the tracked connections and if I'm not mistaken WebSockets hijack the connection; so my question is, with this change how would one ensure graceful shutdown of connections including WebSockets?
It appears that they would be interrupted/closed after all normal http idle connections are completed and the process ends.
I've only been able to do it using a custom listener that keeps track of the connections created and closed, ConnState for keeping track of idle KeepAlive connections all bundled in a library, kms, that exposes a ShutdownInitialized channel that WebSockets, and anything else, can use to gracefully shut themselves down with.
deankarn
on 1 Nov 2016
@joeybloggs, filed https://github.com/golang/go/issues/17721
bradfitz
on 1 Nov 2016
@bradfitz out of curiosity along with the current implementation, why not provide a ShutdownInitialized channel that is accessible via a function from the *Server(or some other means like http.Request which would be more convenient) and then keep track of all of the connections via a WaitGroup, by wrapping the Server's listener and then hijacked connections would have the option of gracefully shutting down by using the ShutDownInitialized.
then server.Shutdown() can wait using the WaitGroup(), context and Timeout.
It just seems like having graceful shutdown built into net/http that doesn't support some pretty common cases seems incomplete/wrong somehow.
deankarn
on 1 Nov 2016
Let's comment on new issue(s) instead of a closed issue. The new issue(s) can reference this one, and then they'll be cross-linked. Also, it's more constructive (at least for me) if you explain the problematic or desired behavior, and not the implementation. The implementation is generally straight-forward given the requirements.
bradfitz
on 1 Nov 2016
this issue has been dragging along since 1.1
rami-dabain
on 27 Apr 2017
@rami-dabain, this is a closed issue. This was added to Go 1.8: https://golang.org/doc/go1.8#http_shutdown
bradfitz
on 27 Apr 2017
Related issues
rsc
路
3Comments
OneOfOne
路
3Comments
Miserlou
路
3Comments
ashb
路
3Comments
michaelsafyan
路
3Comments
Most helpful comment
I managed to implement this in a few hours before the freeze, and people have been wanting this forever, so Go 1.8 it is.