Datadog-agent: dogstatsd:7.19.0 panics on containerd

[thepwagner]

Output of the info page (if this is a bug)

N/A, the container crashes.

Describe what happened:

• We run dogstatsd inside containerd instead of Docker.
• We're happily running datadog/dogstatsd:7.18.1, thanks!
• When we attempted to upgrade to yesterday's datadog/dogstatsd:7.19.0 release, our containers now crash on start up:

2020-05-01 12:16:56 UTC | DSD | INFO | (pkg/forwarder/forwarder.go:231 in Start) | Forwarder started, sending to 1 endpoint(s) with 1 worker(s) each: "https://7-19-0-app.agent.datadoghq.com" (1 api key(s))
panic: Trying to get nil ContainerInterface

goroutine 1 [running]:
github.com/DataDog/datadog-agent/pkg/util/containers/providers.ContainerImpl(...)
        /go/src/github.com/DataDog/datadog-agent/pkg/util/containers/providers/provider.go:20
github.com/DataDog/datadog-agent/pkg/util/docker.GetAgentContainerUTSMode(0xed1ccd, 0x1d, 0x0, 0x0)
        /go/src/github.com/DataDog/datadog-agent/pkg/util/docker/uts.go:23 +0xa4
github.com/DataDog/datadog-agent/pkg/util.GetAgentUTSMode(0xec33e4, 0xf, 0xc000038030, 0x4)
        /go/src/github.com/DataDog/datadog-agent/pkg/util/uts_docker.go:22 +0x1e7
github.com/DataDog/datadog-agent/pkg/util.isOSHostnameUsable(0xc00011c6e0)
        /go/src/github.com/DataDog/datadog-agent/pkg/util/hostname.go:81 +0x4b
github.com/DataDog/datadog-agent/pkg/util.GetHostnameData(0x1013f60, 0xc0001f0a20, 0xc00011c5a0, 0x0, 0x0, 0x1e)
        /go/src/github.com/DataDog/datadog-agent/pkg/util/hostname.go:182 +0x468
github.com/DataDog/datadog-agent/pkg/util.GetHostname(...)
        /go/src/github.com/DataDog/datadog-agent/pkg/util/hostname.go:101
main.runAgent(0xc0004d7d30, 0xc0004d7d30, 0x786b84, 0xc00001e200, 0xeb934e)
        /go/src/github.com/DataDog/datadog-agent/cmd/dogstatsd/main.go:181 +0x575
main.start(0x1846aa0, 0xc0001a49c0, 0x0, 0x2, 0x0, 0x0)
        /go/src/github.com/DataDog/datadog-agent/cmd/dogstatsd/main.go:92 +0x26
github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra.(*Command).execute(0x1846aa0, 0xc0001a4980, 0x2, 0x2, 0x1846aa0, 0xc0001a4980)
        /go/src/github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra/command.go:826 +0x460
github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1846820, 0xeff9a8, 0xc0002fae60, 0xe)
        /go/src/github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra/command.go:864
main.main()
        /go/src/github.com/DataDog/datadog-agent/cmd/dogstatsd/main_nix.go:28 +0x103

Describe what you expected:

datadog/dogstatsd:7.19.0 to continue working in environments where datadog/dogstatsd:7.18.1 did.

Steps to reproduce the issue:
Previously:

# ctr i pull docker.io/datadog/dogstatsd:7.18.1
docker.io/datadog/dogstatsd:7.18.1:                                               resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:b510acb316eeb00bf77b6034a101547e0a5ec56c56f5969edf8222c1249239a8: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:18c0c6bb97a0746928b32553c7d3ae5451274404aec38a5ad6b450714cd59717:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:bf767fefff94c2517e9b88cfe8838a5c6a3cfd29ba45db6e4d585f0874a9f7a2:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:fdef8c744116f80c7968014624c66910cea4242276a76313491cd92a0b85062a:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:31dbdc2a3be518384a0540ad576d6cf083388d023bd70b7a53cff9552829e55f:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:607f556995bc29eb0ce9268dc4114f3c1b919a8b7c4ad16e12bb758d6544ed00:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4167d3e149762ea326c26fc2fd4e36fdeb7d4e639408ad30f37b8f25ac285a98:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.3 s                                                                    total:  7.5 Mi (5.8 MiB/s)                                       
unpacking linux/amd64 sha256:b510acb316eeb00bf77b6034a101547e0a5ec56c56f5969edf8222c1249239a8...
done

# ctr c create --env DD_API_KEY=dummy docker.io/datadog/dogstatsd:7.18.1 dd-old
# ctr t start dd-old
2020-05-01 12:16:19 UTC | DSD | INFO | (pkg/forwarder/forwarder.go:171 in Start) | Forwarder started, sending to 1 endpoint(s) with 1 worker(s) each: "https://7-18-1-app.agent.datadoghq.com" (1 api key(s))
2020-05-01 12:16:20 UTC | DSD | INFO | (pkg/metadata/host/host.go:161 in getNetworkMeta) | could not get network metadata: could not detect network ID
2020-05-01 12:16:20 UTC | DSD | INFO | (pkg/serializer/serializer.go:346 in SendMetadata) | Sent metadata payload, size (raw/compressed): 643/394 bytes.
2020-05-01 12:16:20 UTC | DSD | INFO | (pkg/dogstatsd/listeners/udp.go:94 in Listen) | dogstatsd-udp: starting to listen on 0.0.0.0:8125
2020-05-01 12:16:20 UTC | DSD | ERROR | (pkg/forwarder/worker.go:157 in process) | Error while processing transaction: error while sending transaction, rescheduling it: Post https://7-18-1-app.agent.datadoghq.com/intake/?api_key=dummy: dial tcp: lookup 7-18-1-app.agent.datadoghq.com on [::1]:53: dial udp [::1]:53: socket: address family not supported by protocol
^C2020-05-01 12:16:23 UTC | DSD | WARN | (cmd/dogstatsd/main.go:226 in stopAgent) | Some components were unhealthy: [forwarder metadata-host metadata-inventories aggregator dogstatsd-main]
2020-05-01 12:16:23 UTC | DSD | INFO | (cmd/dogstatsd/main.go:234 in stopAgent) | See ya!

Now:

# ctr i pull docker.io/datadog/dogstatsd:7.19.0
docker.io/datadog/dogstatsd:7.19.0:                                               resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:0a859436e77bf8062e69af9e9aff1654b489abd9c7ff073f52bf7031c3e949da: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:aa9775ed976c7be170c5f03d4ad2fee652447a40334c15d97ad05965d334a8c4:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:2eb9baa6756936e3def21517e40f64437502d5b00e6b474128f2904762efb1b7:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:c260c77f4c67150d1c41d90c52ccfa2e63052696a43c366b1b3d7d92e27801a4:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:3db2f0183f5a80176359d8525add1f072dc8aae6b69e3b344889a01f13eda30f:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:49744fbfba3442cccc793455fca4b83a77376b1d6e7b29a3f21084eabeb6d2a1:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:21c83c5242199776c232920ddb58cfa2a46b17e42ed831ca9001c8dbc532d22d:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.1 s                                                                    total:  7.4 Mi (6.7 MiB/s)                                       
unpacking linux/amd64 sha256:0a859436e77bf8062e69af9e9aff1654b489abd9c7ff073f52bf7031c3e949da...
done

# ctr c create --env DD_API_KEY=dummy docker.io/datadog/dogstatsd:7.19.0 dd-new
# ctr t start dd-new
2020-05-01 12:16:56 UTC | DSD | INFO | (pkg/forwarder/forwarder.go:231 in Start) | Forwarder started, sending to 1 endpoint(s) with 1 worker(s) each: "https://7-19-0-app.agent.datadoghq.com" (1 api key(s))
panic: Trying to get nil ContainerInterface

goroutine 1 [running]:
github.com/DataDog/datadog-agent/pkg/util/containers/providers.ContainerImpl(...)
        ... (duplicate trace removed, see above)

Additional environment details (Operating System, Cloud provider, etc):

# uname -a
Linux 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux

# ctr version
Client:
  Version:  1.2.13
  Revision: 7ad184331fa3e55e52b890ea95e65ba581ae3429

Server:
  Version:  1.2.13
  Revision: 7ad184331fa3e55e52b890ea95e65ba581ae3429

[kyleian]

Seeing this on my end as well having had latest set as my target tag, where my service deployments are failing waiting for dogstatsd sidecar to stand up. Will pin back to 7.18.1 for my builds for the time being, thanks for that.

[CharlyF]

Hey - Thank you for the report.
I can reproduce indeed. We are looking into it with high priority and will update you as soon as we can.
Apologies for the headache.

Best,
.C

[CharlyF]

Following up on this, we have identified the issue and are working on a fix.
While we review and QA the fix we have updated the docker images datadog/dogstatsd:latest as well as datadog/dogstatsd:7 to point to the previous stable image, datadog/dogstatsd:7.18.1 image id fdef8c744116.
Thanks again for your patience, we will keep you posted as we release 7.19.1

Best,
.C

[thepwagner]

I've confirmed 7.19.1 fixes this issue, thanks!