Go: cmd/pprof: add HTTPS support with client certificates

[sorenmat]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.8.3 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/smo/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.8.3/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.8.3/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/dh/gykpvks552v4h7yfms86x3r80000gn/T/go-build191348200=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

Tried to connect to a HTTPS service in order to pull out pprof data

What did you expect to see?

The pprof data

What did you see instead?

http fetch https://192.168.99.100:32473/debug/pprof/profile?seconds=5: Get https://192.168.99.100:32473/debug/pprof/profile?seconds=5: remote error: tls: bad certificate

I think it would sort of nice to be able to connect via pprof to a server that runs SSL/TLS with client certificates. There
is a work-around this of cause, just put up and proxy that does SSL termination, but that might not always be feasible.
I assume the command would look something like.
go tool pprof -seconds 5 -caert cert.pem -key key.pem -ca.cert ca.pem https://192.168.99.100:32473/debug/pprof/profile

[mvdan]

Why not download the profile using a program that likely has these features already, like wget or curl, then use the file in go tool pprof instead?

[OneOfOne]

It is already supported:

$ go tool pprof -seconds 5 https+insecure://192.168.99.100:32473/debug/pprof/profile

[odeke-em]

@OneOfOne do you know if this is documented? If not, could we document this and then we can close the issue, as you've already provided the fix.

[sorenmat]

@OneOfOne not sure that works if you have mutual TLS enabled. But @mvdan had a point, I can download it using curl and then load it with proof.
We can mark this is resolved IMO, just me not thinking 🙄

[OneOfOne]

It's documented in the go1.8 release notes but I think it should be added to the tool's documentation.

https://tip.golang.org/doc/go1.8

The pprof tool can now profile TLS servers and skip certificate validation by using the “https+insecure” URL scheme.

[odeke-em]

@OneOfOne wanna send a documentation CL? /cc @bradfitz

[bradfitz]

Keep in mind that pprof is vendored from an upstream repo, so you can't change some parts of it in Go's repo directly. If you did, it'd just get deleted the next time we imported from the upstream pprof repo.

But I'm not sure which docs you're proposing editing.

https://golang.org/cmd/pprof/ doesn't have much as-is.

[odeke-em]

Ahh I see, thanks for the clarification @bradfitz. I've godoc'd https://github.com/google/pprof and also grep'd for insecure and found that 'https+insecure' is in an internal package for the default fetcher so I no longer believe that documenting this trivial.

[ggaurav10]

Hi,
I am trying to profile swarm using "go tool pprof"
Below are error messages with different options that I have tried:

# go tool pprof http://127.0.0.1:2376/debug/pprof/trace
Fetching profile over HTTP from http://127.0.0.1:2376/debug/pprof/trace
http://127.0.0.1:2376/debug/pprof/trace: Get http://127.0.0.1:2376/debug/pprof/trace: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
failed to fetch any profiles

# go tool pprof https+insecure://127.0.0.1:2376/debug/pprof/trace
Fetching profile over HTTP from https+insecure://127.0.0.1:2376/debug/pprof/trace
https+insecure://127.0.0.1:2376/debug/pprof/trace: Get https://127.0.0.1:2376/debug/pprof/trace: remote error: tls: bad certificate
failed to fetch any profiles

Below are the config details:

# go version
go version go1.9 linux/amd64

# /var/vcap/packages/swarm/bin/swarm --version
swarm version 1.2.8 (HEAD)

Swarm is running with debug option enabled, however, the curl ouput at "info" endpoint has the debug set to false:

./bin/swarm --debug=true --log-level=info manage --cluster-driver=swarm --host unix:///var/vcap/sys/run/swarm_manager/swarm_manager.sock --engine-failure-retry 10 --host=0.0.0.0:2376 --strategy=spread --filter=health --filter=port --filter=dependency --filter=affinity --filter=constraint --replication=false --replication-ttl=30s --advertise=xx.xx.xx.xx2376 --heartbeat=20s --api-enable-cors=false --cluster-opt=swarm.overcommit=0.5 --tls=true --tlscacert=/var/vcap/jobs/swarm_manager/config/docker.cacert --tlscert=/var/vcap/jobs/swarm_manager/config/docker.cert --tlskey=/var/vcap/jobs/swarm_manager/config/docker.key --tlsverify=true nodes://yy.yy.yy.yy:4243

#curl -k --key /var/vcap/jobs/swarm_manager/config/docker.key --cert /var/vcap/jobs/swarm_manager/config/docker.cert https://127.0.0.1:2376/info

The output of above cmd has "Debug":false in its output.

Is there anything that I missing here?

[sjug]

I've pushed a PR to pprof to add this functionality. https://github.com/google/pprof/pull/261

[ianlancetaylor]

This needs to be fixed upstream, so closing the Go issue.