Ambassador: Regression: Segmentation fault in v1.9.0

[danielmittelman]

Describe the bug
I recently upgraded my AES installation from v1.8.1 to v1.9.0. The configuration worked well in the older version, but as we launched v1.9.0 we saw our pods beginning to crash and restart with the following trace:

2020-11-18 09:23:04 diagd 1.9.0 [P98TThreadPoolExecutor-0_0] INFO: CA9076E8-B718-43D7-900B-145C344897EE: 127.0.0.1 "GET /ambassador/v0/diag/" 344ms 200 success
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x103958a]
goroutine 2156 [running]:
net/url.(*URL).String(0x0, 0xbfe557de1cef922a, 0xe41751f5d)
  net/url/url.go:813 +0x4a
github.com/datawire/apro/cmd/amb-sidecar/devportal/server.(*Fetcher).retrieve(0xc000d90000, 0x2a08691, 0x5)
  github.com/datawire/apro/cmd/amb-sidecar/devportal/server/fetcher.go:402 +0x1daa
github.com/datawire/apro/cmd/amb-sidecar/devportal/server.(*Fetcher).Run(0xc000d90000, 0x2ce5a00, 0xc000e5fec0)
  github.com/datawire/apro/cmd/amb-sidecar/devportal/server/fetcher.go:99 +0x23d
github.com/datawire/apro/cmd/amb-sidecar/runner.runE.func8(0x2ce5a00, 0xc000e5fec0, 0x2a1d333, 0x11)
  github.com/datawire/apro/cmd/amb-sidecar/runner/main.go:394 +0x3e
github.com/datawire/ambassador/pkg/dgroup.(*Group).Go.func1(0xc00091fb60, 0x0)
  github.com/datawire/[email protected]/pkg/dgroup/group.go:139 +0x66
github.com/datawire/ambassador/pkg/derrgroup.(*Group).Go.func2(0xc000e5f800, 0xc00064a120, 0x2a1d333, 0x11)
  github.com/datawire/[email protected]/pkg/derrgroup/errgroup.go:131 +0x2b
created by github.com/datawire/ambassador/pkg/derrgroup.(*Group).Go
  github.com/datawire/[email protected]/pkg/derrgroup/errgroup.go:129 +0x12d
/buildroot/ambassador/python/entrypoint.sh: line 26: warning: run_pending_traps: bad value in trap_list[17]: 0x556d6d536c69
/buildroot/ambassador/python/entrypoint.sh: line 23: warning: run_pending_traps: bad value in trap_list[17]: 0x556d6d536c69
2020-11-18 09:23:04 AMBASSADOR INFO amb-sidecar exited with status 2
/buildroot/ambassador/python/entrypoint.sh: line 23: warning: run_pending_traps: bad value in trap_list[17]: 0x556d6d536c69
2020-11-18 09:23:04 AMBASSADOR INFO killing extant processes
kill: can't kill pid 115: No such process
/buildroot/ambassador/python/entrypoint.sh: line 215: warning: run_pending_traps: bad value in trap_list[17]: 0x556d6d536c69
[2020-11-18 09:23:04 +0000] [82] [INFO] Handling signal: term
/buildroot/ambassador/python/entrypoint.sh: line 23: warning: run_pending_traps: bad value in trap_list[17]: 0x556d6d536c69
2020-11-18 09:23:04 AMBASSADOR INFO shutting down (1)

To Reproduce
Steps to reproduce the behavior:

1. Create a fresh install of AES v1.9.0 using Helm 3 on Kubernetes, and expose using a load balancer
2. Send a simple HTTP request to the load balancer, request works. Wait a few minutes and see that the pods remain consistently up
3. Apply the same mappings that worked in v1.8.1
4. Wait a few seconds, the liveness/readiness check will contact the URL /ambassador/v0/diag/ which causes a segmentation fault and the pod to eventually terminate and restart

Expected behavior
Ambassador should not crash intermittently when contacting the URL /ambassador/v0/diag/

Versions (please complete the following information):

• Ambassador: v1.9.0
• Kubernetes environment: EKS v1.18
• Version: v1.9.0

[inercia]

This is a known issue on Mappings where the host cannot be parsed as a valid URL. A fix will be released on 1.9.1.

You can workaround the issue by setting:

docs:
  ignored: true

on every Mapping that uses host_regex: true.

[kflynn]

(Worth noting that we'll be releasing a fix for this shortly -- it is not intended behavior. 😐 )

[khussey]

This issue is fixed in Ambassador 1.9.1, which is now available.