Athens: Memory leak with Minio storage

[jbub]

Describe the bug
I noticed a huge memory leak after upgrading to v0.3.0. After building a project the memory usage jumped to 2.32G, before the upgrade the memory usage was flat at around 70MB. It seems like its not related to the v0.3.0 release, but to minio storage in general.

To Reproduce
Build a project which is using Athens as a GOPROXY.

Expected behavior
Flat memory usage.

Environment (please complete the following information):

• OS: Docker on Ubuntu 18.10
• Go version : 1.12
• Proxy version : gomods/athens:v0.3.0
• Storage (fs/mongodb/s3 etc.) : minio

[arschles]

@jbub thanks for filing this. we'll check it out as soon as we can

[jbub]

Related go.mod:

module my.super.module

go 1.12

require (
    github.com/FZambia/sentinel v1.0.0
    github.com/Shopify/sarama v1.21.0
    github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd
    github.com/coreos/bbolt v1.3.2 // indirect
    github.com/coreos/etcd v3.3.2+incompatible
    github.com/coreos/go-semver v0.2.0 // indirect
    github.com/coreos/go-systemd v0.0.0-20190212144455-93d5ec2c7f76 // indirect
    github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
    github.com/davecgh/go-spew v1.1.1
    github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect
    github.com/dustin/go-humanize v1.0.0 // indirect
    github.com/globalsign/mgo v0.0.0-20180220100256-baa28fcb8e7d
    github.com/gogo/protobuf v1.2.1
    github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef // indirect
    github.com/golang/protobuf v1.3.0
    github.com/gomodule/redigo v1.7.0
    github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c // indirect
    github.com/gorilla/websocket v1.4.0 // indirect
    github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
    github.com/jonboulle/clockwork v0.1.0 // indirect
    github.com/kr/pretty v0.1.0 // indirect
    github.com/pmezard/go-difflib v1.0.0
    github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829
    github.com/satori/go.uuid v1.2.0
    github.com/soheilhy/cmux v0.1.4 // indirect
    github.com/stretchr/testify v1.3.0
    github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 // indirect
    github.com/ugorji/go/codec v0.0.0-20190204201341-e444a5086c43 // indirect
    github.com/vrischmann/envconfig v1.1.0
    github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
    go.etcd.io/bbolt v1.3.2 // indirect
    go.opencensus.io v0.19.1
    golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3
    golang.org/x/time v0.0.0-20181108054448-85acf8d2951c // indirect
    google.golang.org/grpc v1.19.0
)

[marpio]

Possible cause: https://github.com/minio/minio-go/issues/848
and potentially a fix: https://github.com/minio/minio-go/issues/848#issuecomment-338082471

[marwan-at-work]

@jbub can you run Athens with this env var: GODEBUG=gctrace=1 and copy the final gc and scvg lines to see that there is in fact a memory leak and not just a memory bottleneck?

[marwan-at-work]

@jbub also can you please post your Athens config file or env vars if you're not using the default ones?

[jbub]

ATHENS_PORT: ":3009"
ATHENS_STORAGE_TYPE: "minio"
ATHENS_MINIO_ENDPOINT: "mydomain.xxx"
ATHENS_MINIO_ACCESS_KEY_ID: "${MINIO_ACCESS_KEY}"
ATHENS_MINIO_SECRET_ACCESS_KEY: "${MINIO_SECRET_KEY}"
ATHENS_MINIO_BUCKET_NAME: "athens"
ATHENS_NETRC_PATH: "/config/.netrc"

[jbub]

scvg-1: 576 MB released
scvg-1: inuse: 585, idle: 3505, sys: 4091, released: 3505, consumed: 585 (MB)
gc 236 @241.959s 3%: 286+5.0+0.10 ms clock, 572+0.30/9.2/0+0.20 ms cpu, 1159->1159->1159 MB, 1166 MB goal, 2 P
gc 237 @242.260s 3%: 266+9.0+0.007 ms clock, 532+0/14/0+0.015 ms cpu, 1736->1736->1159 MB, 2318 MB goal, 2 P (forced)
gc 238 @242.572s 3%: 0.083+31+1.6 ms clock, 0.16+3.7/15/0+3.3 ms cpu, 1160->1161->1159 MB, 2318 MB goal, 2 P (forced)
scvg-1: inuse: 1739, idle: 2351, sys: 4091, released: 2351, consumed: 1739 (MB)
gc 239 @242.635s 8%: 12680+6.5+0.005 ms clock, 25361+0.22/9.4/0+0.010 ms cpu, 1737->1737->1159 MB, 2319 MB goal, 2 P
scvg1: inuse: 2327, idle: 1763, sys: 4091, released: 1763, consumed: 2327 (MB)
gc 240 @255.606s 25%: 61297+43+0.65 ms clock, 122594+0.11/69/0+1.3 ms cpu, 2325->2325->1734 MB, 2326 MB goal, 2 P

[marwan-at-work]

This is most likely the issue @marpio recorded as I cannot reproduce any memory leaks in gctrace or the go tracer tool. I have a fork with the suggested solution above. @jbub feel free to test it out and see if it solves the issue: https://github.com/marwan-at-work/athens/tree/miniomemfix

In any case, the fork I have would solve the massive heap allocation to begin with so it's something that Athens should merge in soon anyway.