Caddy: Fatal error: concurrent map read and map write

[tomasdeml]

1. What version of Caddy are you running (caddy -version)?

v0.9.1

2. What are you trying to do?

Refresh heavily a web site served by caddy reverse proxy, i.e. hold F5 key for a while in Chrome / Firefox.

3. What is your entire Caddyfile?

https://:443 {
 tls self_signed
 proxy / 127.0.0.1:580 {
    transparent
    max_fails 0
  }
  bind 127.0.0.1
}

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy.exe -conf Caddyfile

5. What did you expect to see?

I expected the web site to appear once the heavy refreshing is done.

6. What did you see instead (give full error messages and/or log)?

The web site kept loading indefinitely (spinning wheel). Refresh or restart of browser did not help. Then Caddy failed with fatal error (see attachment for complete output):

Activating privacy features... done.
https:// (only accessible on this machine)
fatal error: concurrent map read and map write

goroutine 19217 [running]:
runtime.throw(0xa3a054, 0x21)
    /usr/local/go/src/runtime/panic.go:566 +0x9c fp=0xc0458ffc38 sp=0xc0458ffc18
runtime.mapaccess1_faststr(0x9ca040, 0xc04217d9b0, 0xa2aff8, 0xa, 0x94ce9aa0244a93f8)
    /usr/local/go/src/runtime/hashmap_fast.go:201 +0x4fa fp=0xc0458ffc98 sp=0xc0458ffc38
net/textproto.MIMEHeader.Get(0xc04217d9b0, 0xa2aff8, 0xa, 0xc04232eb00, 0x0)
    /usr/local/go/src/net/textproto/header.go:33 +0x7c fp=0xc0458ffcd0 sp=0xc0458ffc98
net/http.Header.Get(0xc04217d9b0, 0xa2aff8, 0xa, 0xa, 0xc04232eb88)
    /usr/local/go/src/net/http/header.go:39 +0x46 fp=0xc0458ffd08 sp=0xc0458ffcd0
net/http.(*Request).write(0xc042324d20, 0xcbbd20, 0xc04472c6c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/request.go:507 +0xabf fp=0xc0458ffe48 sp=0xc0458ffd08
net/http.(*persistConn).writeLoop(0xc042c71700)
    /usr/local/go/src/net/http/transport.go:1644 +0x1b3 fp=0xc0458fff98 sp=0xc0458ffe48
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc0458fffa0 sp=0xc0458fff98
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1058 +0x515
...

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

Unfortunately, I could not reproduce the issue by proxying to another Caddy instance or a public Internet site. I can only reproduce the issue with our application as upstream host running on localhost. It seems to be dependent on timing and number of requests from browser.

The issue does not occur when running caddy with HTTP2 disabled, when running without proxy or when running with proxy but without max_fails directive. I also could not reproduce the issue with boom benchmark tool (tried w/ and w/o -h2 switch).

[mholt]

Can you post the full panic output? The stack trace is cut off...

(I also don't recommend saying max_fails 0 for localhost backends, because it'll cause your CPU to spin. You're telling Caddy that the backend is never down even when it is.)

[tomasdeml]

Sure, hope this one is complete. Captured by redirecting stdout and stderr to file. Caddy fails very early once the refreshing starts.

[mholt]

Got it, thanks. That's quite a busy server you've got there.

Weird. According to that stack trace, this map access within the std lib is apparently concurrent with something else. So maybe there's something in our proxy.go or reverseproxy.go files that is accessing the same map at the same time? But how is that possible...?

Will look into this, would like some help as well.

[mholt]

Based on what you said:

The issue does not occur when running caddy with HTTP2 disabled, when running without proxy or when running with proxy but without max_fails directive.

It seems like the greatest common denominator is max_fails 0 + HTTP/2. Remove either one of these and the race disappears, correct? What if max_fails is set to 1, I wonder? Meanwhile I'll try reproducing this with wrk...

[tomasdeml]

Remove either one of these and the race disappears, correct?

Yep.

I will be out of Office for a while so I cannot try it right now with max_fails 1 :(

[mholt]

Dang. Well I would really love to be able to reproduce this. In the meantime, if anyone can examine the proxy.go and reverseproxy.go files and see if there is shared map state somewhere in there, that would be greatly appreciated.

[zacheryph]

https://gist.github.com/anonymous/146fe3bcaa53505978954a412324f911

fatal error: concurrent map writes

I received this on Caddy 0.9.0 running within docker on Ubuntu 16.04. Self made container downloading caddy from caddyserver.com with zero additional features enabled.

[mholt]

[zacheryph]

[tw4452852]

[mholt]

[tw4452852]

[mholt]

[tw4452852]

[mholt]

[tomasdeml]

[tomasdeml]

[mholt]

[tomasdeml]

[mholt]

[faradaytrs]

[mholt]