Caddy: Bugreport: "fatal error: concurrent map writes"

Created on 28 Jul 2016  路  10Comments  路  Source: caddyserver/caddy

1. What version of Caddy are you running (caddy -version)?

  • Version: 0.9.0 (caddy_linux_amd64)

    2. What are you trying to do?

Running Caddy as a reverse proxy inside a Docker container.

3. What is your entire Caddyfile?

http://jira.{$TECHNICAL_DOMAIN} {
  redir https://jira.{$TECHNICAL_DOMAIN}{uri} 301
}

http://wiki.{$TECHNICAL_DOMAIN} {
  redir https://wiki.{$TECHNICAL_DOMAIN}{uri} 301
}

jira.{$TECHNICAL_DOMAIN} {

  gzip

  errors stdout
  log stdout

  proxy / http://jira:8080 {
    fail_timeout 60s
    transparent
  }

  tls self_signed

}

wiki.{$TECHNICAL_DOMAIN} {

  gzip

  errors stdout
  log stdout

  proxy / http://confluence:8090 {
    fail_timeout 60s
    transparent
  }

  tls self_signed

}

4. How did you run Caddy (give the full command and describe the execution environment)?

  • Container OS: Alpine Linux v3.4"
  • Host OS: CoreOS 1109.1.0 (MoreOS)
  • Command: /usr/local/bin/caddy -agree -email ...

    5. What did you expect to see?

No errors.

6. What did you see instead (give full error messages and/or log)?

Relatively slow performance and the following error after a while: 

fatal error: concurrent map writes

goroutine 25635 [running]:
runtime.throw(0xc976f0, 0x15)
    /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc8203d0ff0 sp=0xc8203d0fd8
runtime.mapassign1(0xb930e0, 0xc82019d6b0, 0xc8203d10b8, 0xc8203d10c8)
    /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc8203d1098 sp=0xc8203d0ff0
net/textproto.MIMEHeader.Set(0xc82019d6b0, 0xc1cb20, 0x4, 0xc8200e82c0, 0x10)
    /usr/local/go/src/net/textproto/header.go:22 +0xc2 fp=0xc8203d10e8 sp=0xc8203d1098
net/http.Header.Set(0xc82019d6b0, 0xc1cb20, 0x4, 0xc8200e82c0, 0x10)
    /usr/local/go/src/net/http/header.go:31 +0x49 fp=0xc8203d1118 sp=0xc8203d10e8
github.com/mholt/caddy/caddyhttp/proxy.Proxy.ServeHTTP(0x7fd3899938c8, 0xc82019db00, 0xc8200f1b40, 0x1, 0x1, 0x7fd3899d00d0, 0xc8206e4720, 0xc8202ea1c0, 0x7c131e, 0x0, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/proxy/proxy.go:132 +0x703 fp=0xc8203d1340 sp=0xc8203d1118
github.com/mholt/caddy/caddyhttp/proxy.(*Proxy).ServeHTTP(0xc82019db30, 0x7fd3899d00d0, 0xc8206e4720, 0xc8202ea1c0, 0xc82019d650, 0x0, 0x0)
    <autogenerated>:6 +0xc8 fp=0xc8203d13a0 sp=0xc8203d1340
github.com/mholt/caddy/caddyhttp/errors.ErrorHandler.ServeHTTP(0x7fd389993958, 0xc82019db30, 0xc82019d650, 0xc8200f03a0, 0x6, 0xc82019b1d0, 0x0, 0x0, 0x0, 0x7fd3899d00d0, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/errors/errors.go:39 +0x118 fp=0xc8203d14f8 sp=0xc8203d13a0
github.com/mholt/caddy/caddyhttp/errors.(*ErrorHandler).ServeHTTP(0xc82019af50, 0x7fd3899d00d0, 0xc8206e4720, 0xc8202ea1c0, 0xc82000abe0, 0x0, 0x0)
    <autogenerated>:1 +0xda fp=0xc8203d1578 sp=0xc8203d14f8
github.com/mholt/caddy/caddyhttp/gzip.Gzip.ServeHTTP(0x7fd389993980, 0xc82019af50, 0xc8200eb340, 0x1, 0x1, 0x7fd389994f90, 0xc82000abe0, 0xc8202ea1c0, 0x0, 0x0, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/gzip/gzip.go:83 +0x65c fp=0xc8203d1778 sp=0xc8203d1578
github.com/mholt/caddy/caddyhttp/gzip.(*Gzip).ServeHTTP(0xc82019db60, 0x7fd389994f90, 0xc82000abe0, 0xc8202ea1c0, 0x7fd389994f60, 0x0, 0x0)
    <autogenerated>:13 +0xc8 fp=0xc8203d17d8 sp=0xc8203d1778
github.com/mholt/caddy/caddyhttp/log.Logger.ServeHTTP(0x7fd3899939a8, 0xc82019db60, 0xc82019aeb0, 0x1, 0x1, 0xd5d518, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0, 0x20, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/log/log.go:40 +0x231 fp=0xc8203d19e0 sp=0xc8203d17d8
github.com/mholt/caddy/caddyhttp/log.(*Logger).ServeHTTP(0xc82019db90, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0, 0xc8202ea101, 0x0, 0x0)
    <autogenerated>:1 +0xc9 fp=0xc8203d1a48 sp=0xc8203d19e0
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).serveHTTP(0xc820273ab0, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0, 0x5, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:263 +0x738 fp=0xc8203d1c30 sp=0xc8203d1a48
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).ServeHTTP(0xc820273ab0, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:210 +0xdc fp=0xc8203d1c70 sp=0xc8203d1c30
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).wrapWithSvcHeaders.func1(0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:112 +0x97 fp=0xc8203d1cb0 sp=0xc8203d1c70
net/http.HandlerFunc.ServeHTTP(0xc820274e60, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0)
    /usr/local/go/src/net/http/server.go:1618 +0x3a fp=0xc8203d1cd0 sp=0xc8203d1cb0
net/http.serverHandler.ServeHTTP(0xc82019ec80, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0)
    /usr/local/go/src/net/http/server.go:2081 +0x19e fp=0xc8203d1d30 sp=0xc8203d1cd0
net/http.initNPNRequest.ServeHTTP(0xc8203d4000, 0xc82019ec80, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0)
    /usr/local/go/src/net/http/server.go:2489 +0x221 fp=0xc8203d1ec0 sp=0xc8203d1d30
net/http.(*initNPNRequest).ServeHTTP(0xc8203d9df0, 0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0)
    <autogenerated>:253 +0xb6 fp=0xc8203d1ef8 sp=0xc8203d1ec0
net/http.(Handler).ServeHTTP-fm(0x7fd389994ea0, 0xc820022020, 0xc8202ea1c0)
    /usr/local/go/src/net/http/h2_bundle.go:3847 +0x50 fp=0xc8203d1f30 sp=0xc8203d1ef8
net/http.(*http2serverConn).runHandler(0xc82007a780, 0xc820022020, 0xc8202ea1c0, 0xc8204220c0)
    /usr/local/go/src/net/http/h2_bundle.go:4060 +0x9f fp=0xc8203d1f80 sp=0xc8203d1f30
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8203d1f88 sp=0xc8203d1f80
created by net/http.(*http2serverConn).processHeaderBlockFragment
    /usr/local/go/src/net/http/h2_bundle.go:3853 +0x55e

goroutine 1 [semacquire, 15 minutes]:
sync.runtime_Semacquire(0xc8200f01ec)
    /usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc8200f01e0)
    /usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/mholt/caddy.(*Instance).Wait(0xc8201a0680)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy.go:373 +0x25
github.com/mholt/caddy/caddy/caddymain.Run()
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy/caddymain/run.go:114 +0xa09
main.main()
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy/main.go:13 +0x1b

goroutine 5 [syscall, 15 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [select, 15 minutes]:
github.com/mholt/caddy/caddytls.maintainAssets(0xc82000e600)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddytls/maintain.go:42 +0x67d
created by github.com/mholt/caddy/caddytls.init.1
    /Users/matt/Dev/src/github.com/mholt/caddy/caddytls/maintain.go:13 +0x4e

goroutine 8 [select, 15 minutes, locked to thread]:
runtime.gopark(0xd5e950, 0xc820020f28, 0xc23c40, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc820020f28, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc820020f28)
    /usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 18 [chan receive, 15 minutes]:
github.com/mholt/caddy.trapSignalsCrossPlatform.func1()
    /Users/matt/Dev/src/github.com/mholt/caddy/sigtrap.go:29 +0x10d
created by github.com/mholt/caddy.trapSignalsCrossPlatform
    /Users/matt/Dev/src/github.com/mholt/caddy/sigtrap.go:47 +0x2b

goroutine 19 [chan receive, 15 minutes]:
github.com/mholt/caddy.trapSignalsPosix.func1()
    /Users/matt/Dev/src/github.com/mholt/caddy/sigtrap_posix.go:18 +0x36c
created by github.com/mholt/caddy.trapSignalsPosix
    /Users/matt/Dev/src/github.com/mholt/caddy/sigtrap_posix.go:83 +0x2b

goroutine 10 [chan receive, 15 minutes]:
github.com/mholt/caddy/caddyhttp/httpserver.newGracefulListener.func1(0xc82011e8a0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/graceful.go:18 +0x3e
created by github.com/mholt/caddy/caddyhttp/httpserver.newGracefulListener
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/graceful.go:23 +0xe3

goroutine 13 [chan receive, 15 minutes]:
github.com/mholt/caddy/caddyhttp/httpserver.newGracefulListener.func1(0xc82011e9f0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/graceful.go:18 +0x3e
created by github.com/mholt/caddy/caddyhttp/httpserver.newGracefulListener
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/graceful.go:23 +0xe3

goroutine 22 [chan receive, 15 minutes]:
github.com/mholt/caddy.startServers.func2(0xc8200e66c0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy.go:643 +0x64
created by github.com/mholt/caddy.startServers
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy.go:653 +0x6bf

goroutine 9 [IO wait, 15 minutes]:
net.runtime_pollWait(0x7fd3899949c8, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820273b80, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820273b80, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820273b20, 0x0, 0x7fd3899cfaa0, 0xc820113200)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc8200ec090, 0x525e98, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
github.com/mholt/caddy/caddyhttp/httpserver.tcpKeepAliveListener.Accept(0xc8200ec090, 0x0, 0x0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:376 +0x41
github.com/mholt/caddy/caddyhttp/httpserver.(*gracefulListener).Accept(0xc82011e8a0, 0x0, 0x0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/graceful.go:40 +0x63
net/http.(*Server).Serve(0xc82019ec00, 0x7fd3899cfa68, 0xc82011e8a0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).Serve(0xc820273a40, 0x7fd3899cfa68, 0xc82011e8a0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:185 +0x2e9
github.com/mholt/caddy.startServers.func1.1(0xc8200e66c0, 0x7fd389993918, 0xc820273a40, 0x7fd389993a48, 0xc8200ec090, 0xc8201a0680)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy.go:631 +0x3f
created by github.com/mholt/caddy.startServers.func1
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy.go:633 +0xa0

goroutine 11 [IO wait, 15 minutes]:
net.runtime_pollWait(0x7fd389994848, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82011ae60, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82011ae60, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).readFrom(0xc82011ae00, 0xc8200beb00, 0x546, 0x546, 0x0, 0x0, 0x0, 0x7fd3899cb050, 0xc820014140)
    /usr/local/go/src/net/fd_unix.go:277 +0x2a5
net.(*UDPConn).ReadFromUDP(0xc820022078, 0xc8200beb00, 0x546, 0x546, 0x546, 0x546, 0x0, 0x0)
    /usr/local/go/src/net/udpsock_posix.go:61 +0x117
github.com/lucas-clemente/quic-go.(*Server).Serve(0xc82000eba0, 0xc820022078, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/server.go:90 +0x112
github.com/lucas-clemente/quic-go.(*Server).ListenAndServe(0xc82000eba0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/server.go:79 +0x8a
github.com/lucas-clemente/quic-go/h2quic.(*Server).serveImpl(0xc820274ce0, 0xc820079400, 0x0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/h2quic/server.go:86 +0x377
github.com/lucas-clemente/quic-go/h2quic.(*Server).ListenAndServe(0xc820274ce0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/h2quic/server.go:44 +0x114
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).Serve.func1(0xc820273a40)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:178 +0x29
created by github.com/mholt/caddy/caddyhttp/httpserver.(*Server).Serve
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:182 +0x2bb

goroutine 12 [IO wait, 10 minutes]:
net.runtime_pollWait(0x7fd389994908, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820273bf0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820273bf0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820273b90, 0x0, 0x7fd3899cfaa0, 0xc82041d000)
    /usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc8200ec098, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
github.com/mholt/caddy/caddyhttp/httpserver.tcpKeepAliveListener.Accept(0xc8200ec098, 0x0, 0x0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:376 +0x41
github.com/mholt/caddy/caddyhttp/httpserver.(*gracefulListener).Accept(0xc82011e9f0, 0x0, 0x0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/graceful.go:40 +0x63
crypto/tls.(*listener).Accept(0xc820113320, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/tls.go:52 +0x60
net/http.(*Server).Serve(0xc82019ec80, 0x7fd3899cfcb0, 0xc820113320, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2117 +0x129
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).Serve(0xc820273ab0, 0x7fd3899cfcb0, 0xc820113320, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:185 +0x2e9
github.com/mholt/caddy.startServers.func1.1(0xc8200e66c0, 0x7fd389993918, 0xc820273ab0, 0x7fd389993a48, 0xc8200ec098, 0xc8201a0680)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy.go:631 +0x3f
created by github.com/mholt/caddy.startServers.func1
    /Users/matt/Dev/src/github.com/mholt/caddy/caddy.go:633 +0xa0

goroutine 14 [select, 15 minutes]:
github.com/mholt/caddy/caddytls.standaloneTLSTicketKeyRotation(0xc8201aaa00, 0xc820012bc0, 0xc82000ec60)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddytls/crypto.go:209 +0x589
created by github.com/mholt/caddy/caddytls.RotateSessionTicketKeys
    /Users/matt/Dev/src/github.com/mholt/caddy/caddytls/crypto.go:171 +0x7f

goroutine 15 [IO wait, 15 minutes]:
net.runtime_pollWait(0x7fd389994788, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82011af40, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82011af40, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).readFrom(0xc82011aee0, 0xc8200bf080, 0x546, 0x546, 0x0, 0x0, 0x0, 0x7fd3899cb050, 0xc820014140)
    /usr/local/go/src/net/fd_unix.go:277 +0x2a5
net.(*UDPConn).ReadFromUDP(0xc820022088, 0xc8200bf080, 0x546, 0x546, 0x546, 0x546, 0x0, 0x0)
    /usr/local/go/src/net/udpsock_posix.go:61 +0x117
github.com/lucas-clemente/quic-go.(*Server).Serve(0xc82000eea0, 0xc820022088, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/server.go:90 +0x112
github.com/lucas-clemente/quic-go.(*Server).ListenAndServe(0xc82000eea0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/server.go:79 +0x8a
github.com/lucas-clemente/quic-go/h2quic.(*Server).serveImpl(0xc820274e40, 0xc8201aaa00, 0x0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/h2quic/server.go:86 +0x377
github.com/lucas-clemente/quic-go/h2quic.(*Server).ListenAndServe(0xc820274e40, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/lucas-clemente/quic-go/h2quic/server.go:44 +0x114
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).Serve.func1(0xc820273ab0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:178 +0x29
created by github.com/mholt/caddy/caddyhttp/httpserver.(*Server).Serve
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:182 +0x2bb

goroutine 48 [select, 11 minutes]:
net/http.(*http2serverConn).serve(0xc8202b8c80)
    /usr/local/go/src/net/http/h2_bundle.go:3163 +0xedc
net/http.(*http2Server).ServeConn(0xc820113380, 0x7fd3899cfdf8, 0xc8200bb800, 0xc82035fbb0)
    /usr/local/go/src/net/http/h2_bundle.go:2711 +0xa45
net/http.http2ConfigureServer.func1(0xc82019ec80, 0xc8200bb800, 0x7fd389994cc8, 0xc82029ab70)
    /usr/local/go/src/net/http/h2_bundle.go:2603 +0xae
net/http.(*conn).serve(0xc8202d0400)
    /usr/local/go/src/net/http/server.go:1414 +0x777
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

goroutine 65 [IO wait, 11 minutes]:
net.runtime_pollWait(0x7fd3899946c8, 0x72, 0xc8202ae400)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8202abd40, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8202abd40, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8202abce0, 0xc8202ae400, 0x400, 0x400, 0x0, 0x7fd3899cb050, 0xc820014140)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200220c0, 0xc8202ae400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
github.com/mholt/caddy/caddyhttp/httpserver.(*gracefulConn).Read(0xc8202b23c0, 0xc8202ae400, 0x400, 0x400, 0xd, 0x0, 0x0)
    <autogenerated>:35 +0x82
crypto/tls.(*block).readFromUntil(0xc8202c5c20, 0x7fd3899cfe58, 0xc8202b23c0, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc8200bb800, 0xd5ea17, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc8200bb800, 0xc8202d2878, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:939 +0x167
io.ReadAtLeast(0x7fd389994d40, 0xc8200bb800, 0xc8202d2878, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd389994d40, 0xc8200bb800, 0xc8202d2878, 0x9, 0x9, 0xc8203cdf00, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
net/http.http2readFrameHeader(0xc8202d2878, 0x9, 0x9, 0x7fd389994d40, 0xc8200bb800, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:715 +0xa5
net/http.(*http2Framer).ReadFrame(0xc8202d2840, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:906 +0xe9
net/http.(*http2serverConn).readFrames(0xc8202b8c80)
    /usr/local/go/src/net/http/h2_bundle.go:3067 +0x5a
created by net/http.(*http2serverConn).serve
    /usr/local/go/src/net/http/h2_bundle.go:3157 +0x720

goroutine 25150 [IO wait]:
net.runtime_pollWait(0x7fd389993d08, 0x72, 0xc8202bc000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82011a300, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82011a300, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82011a2a0, 0xc8202bc000, 0x1000, 0x1000, 0x0, 0x7fd3899cb050, 0xc820014140)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200ec030, 0xc8202bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fd3899951a0, 0xc8200ec030, 0xc820052138, 0xc8202bc000, 0x1000, 0x1000, 0x403c63, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc82025a700, 0xc8202bc000, 0x1000, 0x1000, 0xc8206dfd1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc82041a4e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82041a4e0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200520d0)
    /usr/local/go/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:860 +0x10a6

goroutine 14365 [select]:
net/http.(*http2serverConn).serve(0xc8202b8280)
    /usr/local/go/src/net/http/h2_bundle.go:3163 +0xedc
net/http.(*http2Server).ServeConn(0xc820113380, 0x7fd3899cfdf8, 0xc8203d4600, 0xc820363bb0)
    /usr/local/go/src/net/http/h2_bundle.go:2711 +0xa45
net/http.http2ConfigureServer.func1(0xc82019ec80, 0xc8203d4600, 0x7fd389994cc8, 0xc8235c7a70)
    /usr/local/go/src/net/http/h2_bundle.go:2603 +0xae
net/http.(*conn).serve(0xc8202d1180)
    /usr/local/go/src/net/http/server.go:1414 +0x777
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

goroutine 25151 [runnable]:
net/http.newTransferWriter(0xbfa340, 0xc820260540, 0xc8202aa4d0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:60 +0x8b2
net/http.(*Request).write(0xc820260540, 0x7fd389995218, 0xc820700840, 0xc820700700, 0xc820442210, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:455 +0x8bc
net/http.(*persistConn).writeLoop(0xc8200520d0)
    /usr/local/go/src/net/http/transport.go:1286 +0x2dc
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:861 +0x10cb

goroutine 8403 [IO wait]:
net.runtime_pollWait(0x7fd389994248, 0x72, 0xc8202d4000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82011a220, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82011a220, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82011a1c0, 0xc8202d4000, 0x1000, 0x1000, 0x0, 0x7fd3899cb050, 0xc820014140)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200ec080, 0xc8202d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
github.com/mholt/caddy/caddyhttp/httpserver.(*gracefulConn).Read(0xc820113ac0, 0xc8202d4000, 0x1000, 0x1000, 0xd, 0x0, 0x0)
    <autogenerated>:35 +0x82
crypto/tls.(*block).readFromUntil(0xc820290930, 0x7fd3899cfe58, 0xc820113ac0, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc8203d4000, 0xd5ea17, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc8203d4000, 0xc82014e248, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:939 +0x167
io.ReadAtLeast(0x7fd389994d40, 0xc8203d4000, 0xc82014e248, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd389994d40, 0xc8203d4000, 0xc82014e248, 0x9, 0x9, 0xc820195f00, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
net/http.http2readFrameHeader(0xc82014e248, 0x9, 0x9, 0x7fd389994d40, 0xc8203d4000, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:715 +0xa5
net/http.(*http2Framer).ReadFrame(0xc82014e210, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:906 +0xe9
net/http.(*http2serverConn).readFrames(0xc82007a780)
    /usr/local/go/src/net/http/h2_bundle.go:3067 +0x5a
created by net/http.(*http2serverConn).serve
    /usr/local/go/src/net/http/h2_bundle.go:3157 +0x720

goroutine 8394 [select]:
net/http.(*http2serverConn).serve(0xc82007a780)
    /usr/local/go/src/net/http/h2_bundle.go:3163 +0xedc
net/http.(*http2Server).ServeConn(0xc820113380, 0x7fd3899cfdf8, 0xc8203d4000, 0xc820103bb0)
    /usr/local/go/src/net/http/h2_bundle.go:2711 +0xa45
net/http.http2ConfigureServer.func1(0xc82019ec80, 0xc8203d4000, 0x7fd389994cc8, 0xc8203d9df0)
    /usr/local/go/src/net/http/h2_bundle.go:2603 +0xae
net/http.(*conn).serve(0xc8202d0080)
    /usr/local/go/src/net/http/server.go:1414 +0x777
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

goroutine 14378 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7fd389993c48, 0x72, 0xc820429000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc820272bc0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820272bc0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc820272b60, 0xc820429000, 0x800, 0x800, 0x0, 0x7fd3899cb050, 0xc820014140)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200220b0, 0xc820429000, 0x800, 0x800, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
github.com/mholt/caddy/caddyhttp/httpserver.(*gracefulConn).Read(0xc82041cfe0, 0xc820429000, 0x800, 0x800, 0xd, 0x0, 0x0)
    <autogenerated>:35 +0x82
crypto/tls.(*block).readFromUntil(0xc8202c5aa0, 0x7fd3899cfe58, 0xc82041cfe0, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:460 +0xcc
crypto/tls.(*Conn).readRecord(0xc8203d4600, 0xd5ea17, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:562 +0x2d1
crypto/tls.(*Conn).Read(0xc8203d4600, 0xc8202d3d18, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:939 +0x167
io.ReadAtLeast(0x7fd389994d40, 0xc8203d4600, 0xc8202d3d18, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7fd389994d40, 0xc8203d4600, 0xc8202d3d18, 0x9, 0x9, 0xc8203cff00, 0x0, 0x0)
    /usr/local/go/src/io/io.go:315 +0x62
net/http.http2readFrameHeader(0xc8202d3d18, 0x9, 0x9, 0x7fd389994d40, 0xc8203d4600, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:715 +0xa5
net/http.(*http2Framer).ReadFrame(0xc8202d3ce0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/h2_bundle.go:906 +0xe9
net/http.(*http2serverConn).readFrames(0xc8202b8280)
    /usr/local/go/src/net/http/h2_bundle.go:3067 +0x5a
created by net/http.(*http2serverConn).serve
    /usr/local/go/src/net/http/h2_bundle.go:3157 +0x720

goroutine 25144 [IO wait]:
net.runtime_pollWait(0x7fd389993b88, 0x72, 0xc82014c000)
    /usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc82011a140, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82011a140, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82011a0e0, 0xc82014c000, 0x1000, 0x1000, 0x0, 0x7fd3899cb050, 0xc820014140)
    /usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc8200ec018, 0xc82014c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fd3899951a0, 0xc8200ec018, 0xc820052208, 0xc82014c000, 0x1000, 0x1000, 0x403c63, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1690 +0x67
net/http.(*noteEOFReader).Read(0xc82025a5a0, 0xc82014c000, 0x1000, 0x1000, 0xc820240d1d, 0x0, 0x0)
    <autogenerated>:284 +0xd0
bufio.(*Reader).fill(0xc82041b440)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82041b440, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200521a0)
    /usr/local/go/src/net/http/transport.go:1076 +0x177
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:860 +0x10a6

goroutine 25145 [select]:
net/http.(*persistConn).writeLoop(0xc8200521a0)
    /usr/local/go/src/net/http/transport.go:1280 +0x472
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:861 +0x10cb

goroutine 25634 [select]:
net/http.(*persistConn).roundTrip(0xc8200520d0, 0xc820f5a520, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1476 +0xf1f
net/http.(*Transport).RoundTrip(0xc8200b4000, 0xc820260540, 0xc800000000, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:327 +0x9bb
github.com/mholt/caddy/caddyhttp/proxy.(*ReverseProxy).ServeHTTP(0xc820274bc0, 0x7fd3899d00d0, 0xc820442120, 0xc820260540, 0xc8203685c0, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/proxy/reverseproxy.go:171 +0xd4
github.com/mholt/caddy/caddyhttp/proxy.Proxy.ServeHTTP(0x7fd3899938c8, 0xc82019db00, 0xc8200f1b40, 0x1, 0x1, 0x7fd3899d00d0, 0xc820442120, 0xc8202ea000, 0x7c131e, 0x0, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/proxy/proxy.go:153 +0x987
github.com/mholt/caddy/caddyhttp/proxy.(*Proxy).ServeHTTP(0xc82019db30, 0x7fd3899d00d0, 0xc820442120, 0xc8202ea000, 0xc82019d650, 0x0, 0x0)
    <autogenerated>:6 +0xc8
github.com/mholt/caddy/caddyhttp/errors.ErrorHandler.ServeHTTP(0x7fd389993958, 0xc82019db30, 0xc82019d650, 0xc8200f03a0, 0x6, 0xc82019b1d0, 0x0, 0x0, 0x0, 0x7fd3899d00d0, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/errors/errors.go:39 +0x118
github.com/mholt/caddy/caddyhttp/errors.(*ErrorHandler).ServeHTTP(0xc82019af50, 0x7fd3899d00d0, 0xc820442120, 0xc8202ea000, 0xc8202f0050, 0x0, 0x0)
    <autogenerated>:1 +0xda
github.com/mholt/caddy/caddyhttp/gzip.Gzip.ServeHTTP(0x7fd389993980, 0xc82019af50, 0xc8200eb340, 0x1, 0x1, 0x7fd389994f90, 0xc8202f0050, 0xc8202ea000, 0x0, 0x0, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/gzip/gzip.go:83 +0x65c
github.com/mholt/caddy/caddyhttp/gzip.(*Gzip).ServeHTTP(0xc82019db60, 0x7fd389994f90, 0xc8202f0050, 0xc8202ea000, 0x7fd389994f60, 0x0, 0x0)
    <autogenerated>:13 +0xc8
github.com/mholt/caddy/caddyhttp/log.Logger.ServeHTTP(0x7fd3899939a8, 0xc82019db60, 0xc82019aeb0, 0x1, 0x1, 0xd5d518, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000, 0x34, ...)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/log/log.go:40 +0x231
github.com/mholt/caddy/caddyhttp/log.(*Logger).ServeHTTP(0xc82019db90, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000, 0xc8202ea001, 0x0, 0x0)
    <autogenerated>:1 +0xc9
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).serveHTTP(0xc820273ab0, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000, 0x5, 0x0, 0x0)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:263 +0x738
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).ServeHTTP(0xc820273ab0, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:210 +0xdc
github.com/mholt/caddy/caddyhttp/httpserver.(*Server).wrapWithSvcHeaders.func1(0x7fd389994ea0, 0xc820022000, 0xc8202ea000)
    /Users/matt/Dev/src/github.com/mholt/caddy/caddyhttp/httpserver/server.go:112 +0x97
net/http.HandlerFunc.ServeHTTP(0xc820274e60, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000)
    /usr/local/go/src/net/http/server.go:1618 +0x3a
net/http.serverHandler.ServeHTTP(0xc82019ec80, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000)
    /usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.initNPNRequest.ServeHTTP(0xc8203d4000, 0xc82019ec80, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000)
    /usr/local/go/src/net/http/server.go:2489 +0x221
net/http.(*initNPNRequest).ServeHTTP(0xc8203d9df0, 0x7fd389994ea0, 0xc820022000, 0xc8202ea000)
    <autogenerated>:253 +0xb6
net/http.(Handler).ServeHTTP-fm(0x7fd389994ea0, 0xc820022000, 0xc8202ea000)
    /usr/local/go/src/net/http/h2_bundle.go:3847 +0x50
net/http.(*http2serverConn).runHandler(0xc82007a780, 0xc820022000, 0xc8202ea000, 0xc820422060)
    /usr/local/go/src/net/http/h2_bundle.go:4060 +0x9f
created by net/http.(*http2serverConn).processHeaderBlockFragment
    /usr/local/go/src/net/http/h2_bundle.go:3853 +0x55e
bug
Source
picture yawn

All 10 comments

Yeesh. Thanks for reporting this!

We've actually made pretty significant changes to proxy headers since the 0.9 release. (The concurrent map access appears to happen in some code related to proxying headers that no longer exists.) Do you have the ability to build from source (as easy as go build with Go installed)? If so, could you run the latest on master and see if this panic still occurs?

mholt picture mholt on 28 Jul 2016

Had the same issue previously. A fresh build from master appears to work now.

aboukirev picture aboukirev on 29 Jul 2016

Okay, cool. Going to close this since we have probably already fixed it, given that the offending line was deleted in a recent change to fix another issue.

mholt picture mholt on 30 Jul 2016

Issue didn't reappear in master.

yawn picture yawn on 1 Aug 2016
👍1

How to get a master build ? there is no --HEAD for brew install caddy, download from caddy is 0.9.0 release.

wenerme picture wenerme on 2 Aug 2016

@wenerme With Go installed, just go get -u github.com/mholt/caddy/caddy should do the trick.

mholt picture mholt on 2 Aug 2016

@mholt Anyway to include all plugin ?

wenerme picture wenerme on 2 Aug 2016

@wenerme Just have to add all their imports to run.go.

mholt picture mholt on 2 Aug 2016

@mholt Could you please tell me all import path, thanks.All plugin list in download page.

wenerme picture wenerme on 2 Aug 2016

@wenerme They're all listed here: https://github.com/caddyserver/buildsrv/blob/master/features/registry.go

mholt picture mholt on 2 Aug 2016
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mholt picture mholt  路  3Comments
ericmdantas picture ericmdantas  路  3Comments
la0wei picture la0wei  路  3Comments
mikolysz picture mikolysz  路  3Comments
billop picture billop  路  3Comments