golangci-lint 🚀 - Lower or control memory usage

Related to #342 and pointers usage on latest commit.

We could dig more, sure, but it's a start.

lopezator on 3 Jan 2019

Hi!
A current solution may be to configure GOGC: it can trade memory to time and decrease memory usage in ~2 times. I've shown some results here.

jirfag on 31 Mar 2019

mem.pdf
Also, I've profiled memory of golangci-lint: PDF with results is attached. We can see that type checking consumes the most of memory. I have some ideas on how to fix it but they are all too costly to implement.

jirfag on 31 Mar 2019

We have regular intermittent build failures due to high memory usage. It would be good to reduce usage.

Currently: we're using...

GOGC=20 golangci-lint run --fix --verbose --concurrency 1 --deadline 3m0s

See https://github.com/golangci/golangci-lint#memory-usage-of-golangci-lint

alexec on 10 Jul 2019

istio/istio repo uses 32gb to lint: INFO Memory: 218 samples, avg is 18237.6MB, max is 31795.7MB

howardjohn on 17 Sep 2019

👀8 😄2

👍
Looking forward to testing out master

daxmc99 on 23 Sep 2019

I've merged a fix reducing memory usage of go/analysis linters (staticcheck, gosimple, stylecheck, bodyclose, govet) 5x. Please, check it in a master branch.

jirfag on 23 Sep 2019

🎉6

😍

bodyclose:   14.4 GB / 1m 20s -> 1.2 GB / 41s
govet:       15.5 GB / 1m 25s -> 1.8 GB / 30s
gosimple      5.9 GB / 49 s   -> 1.5 GB / 40s
staticcheck:  7.2 GB / 1m 1s  -> 1.7 GB / 1m 52s
govet:       15.5 GB / 1m 25s -> 1.8 GB / 30s
stylecheck: didn't run because statickcheck failed, but I'd expect similar numbers so I didn't test it

staticcheck seems to have slowed down, but it caught new errors, so maybe it was upgraded on master.

gracenoah on 24 Sep 2019

👍1

It's still using 29 GB of memory when running with all linters enabled rather than enabling 1 linter at a time. Is that expected? Is it just a side effect of Go's GC and it wouldn't use that much memory if it wasn't available?

gracenoah on 24 Sep 2019

@gracenoah thank you for comparison,
I'am continuously reducing memory consumption: it's my top priority now. Is the repo rhere are you testing open-source?

jirfag on 24 Sep 2019

It's not, sorry. It's an internal monorepo. If you want me to test some proposed performance changes, you can email me at [email protected] but that's all I can do.

gracenoah on 24 Sep 2019

I've investigated more high memory consumption.

We can dramatically reduce memory consumption of all linters except unused, interfacer, unparam. To do it we need to run them by our go/analysis runner. Most of them require loader.Program or AST: we can mimic this interface from go/analysis's *packages.Package. It can take 1-3 weeks and reduce memory usage in the same way staticchecks memory dropped in v1.19.0.
The biggest question is what to do with unused, interfacer, unparam. I will research it later.

jirfag on 25 Sep 2019

Hi @jirfag 👋 We continually have memory issues with https://github.com/terraform-providers/terraform-provider-aws if you're looking for an open source project to test. v1.19.1 still throws TravisCI out of memory errors for us with GOGC=30, which was working okay in v1.17.1. Our issue seems to have began in v1.18.0, similar to what was noted here: https://github.com/golangci/golangci-lint/issues/731#issuecomment-534801956

bflad on 25 Sep 2019

Hi @jirfag - continuing the conversation here from #731!

A few hours ago, you wrote:

and reduce memory usage in the same way staticcheck's memory dropped in v1.19.0

But that is not the reality we are facing; in fact, we are experiencing the opposite (a huge increase in memory consumption that starts exactly at the commit claimed to improve things), as I demonstrated in #731.

Unfortunately, our codebase is not open-source, but I am happy to run experiments for you and report results back, as 6a979fb and beyond block our ability to upgrade golangci-lint.

bmhatfield on 25 Sep 2019

One additional note for folks struggling with this. For us, a large portion of memory consumption was coming from concurrent use of go's linker. We were previously able to reduce this significantly by setting GOGC=50, by limiting concurrency (to 4), and by running go vet separately from golangci-lint, but as of 6a979fb, that strategy is no longer effective.

bmhatfield on 25 Sep 2019

@bmhatfield I guess it was caused by large update of staticcheck. I will compare it in the end with 1.17

jirfag on 25 Sep 2019

I've made a prototype. It's results (time/peak memory usage):

|repo|active linters|v1.17.1|v1.18.0|v1.19.1|prototype|
|----|-------------|------|-------|-------|---------|
|rclone|[deadcode errcheck goimports golint govet ineffassign structcheck unconvert varcheck]|29s/5.3GB|25s/5.3GB|17s/1.9GB|13s/0.67GB|
|terraform-provider-aws|[deadcode errcheck gofmt gosimple govet ineffassign misspell structcheck unconvert unused varcheck]|-|-|44s/5.8GB|42s/2.8GB|
|traefik|[deadcode depguard dogsled errcheck funlen goconst gocritic godox gofmt goimports golint gosimple govet ineffassign interfacer misspell nakedret staticcheck structcheck stylecheck typecheck unconvert unused varcheck whitespace]|41s/6.6GB|49s/6.7GB|7m48s/18GB|1m12s/2.1GB|
|golangci-lint|[bodyclose deadcode depguard dogsled dupl errcheck funlen gochecknoinits goconst gocritic gocyclo godox gofmt goimports golint gosec gosimple govet ineffassign interfacer lll misspell nakedret scopelint staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck whitespace]|17s/2.1GB|14s/2.5GB|24s/3.6GB|12.6s/0.5GB|

jirfag on 28 Sep 2019

❤2 🎉1

@jirfag I happen to be at my computer just as you're posting this. I'll run against my repo with your branch right now!

bmhatfield on 28 Sep 2019

Sounds fantastic, I'll try it against istio/istio today

howardjohn on 28 Sep 2019

[bhatfield go-services]% GOGC=50; golangci-lint --concurrency 4 -v run --disable=govet
INFO [config_reader] Config search paths: [./ /Users/bhatfield/Documents/digits/go-services /Users/bhatfield/Documents/digits /Users/bhatfield/Documents /Users/bhatfield /Users /]
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 18 linters: [deadcode errcheck goconst gocritic gocyclo goimports gosimple ineffassign nakedret prealloc scopelint staticcheck structcheck stylecheck typecheck unconvert unused varcheck]
INFO [loader] Go packages loading at mode 575 (deps|exports_file|files|imports|name|compiled_files|types_sizes) took 3.741453781s
INFO [loader] Loaded 886 AST files in 187.128374ms
WARN [runner] Can't run linter goanalysis_metalinter: fact_deprecated: failed prerequisites: fact_deprecated@[REDACTED]/api [[REDACTED]/api/userapi.test]
INFO [runner] processing took 4.944µs with stages: nolint: 2.192µs, max_same_issues: 815ns, skip_dirs: 366ns, max_from_linter: 241ns, identifier_marker: 175ns, cgo: 154ns, autogenerated_exclude: 138ns, filename_unadjuster: 136ns, skip_files: 132ns, diff: 131ns, path_prettifier: 127ns, source_code: 61ns, path_shortener: 60ns, exclude: 55ns, uniq_by_line: 55ns, max_per_file_from_linter: 54ns, exclude-rules: 52ns
INFO [runner] linters took 10.144776527s with stages: goanalysis_metalinter: 5.966777269s, unused: 4.177935139s
INFO File cache stats: 0 entries of total size 0B
INFO Memory: 111 samples, avg is 1409.0MB, max is 2758.4MB
INFO Execution took 14.09073157s

Seems as though we have a winner!

Current master by comparison:

[bhatfield go-services]% GOGC=50; golangci-lint --concurrency 4 -v run --disable=govet
INFO [config_reader] Config search paths: [./ /Users/bhatfield/Documents/digits/go-services /Users/bhatfield/Documents/digits /Users/bhatfield/Documents /Users/bhatfield /Users /]
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 18 linters: [deadcode errcheck goconst gocritic gocyclo goimports gosimple ineffassign nakedret prealloc scopelint staticcheck structcheck stylecheck typecheck unconvert unused varcheck]
INFO [lintersdb] Optimized sublinters [staticcheck gosimple unused stylecheck] into metalinter megacheck
INFO [loader] Go packages loading at mode 991 (imports|types_sizes|compiled_files|files|name|syntax|types|types_info|deps) took 4.275296035s
INFO [runner] worker.3 took 2.108474523s with stages: gocritic: 1.644872104s, deadcode: 152.749573ms, varcheck: 130.423249ms, errcheck: 58.359607ms, unconvert: 49.271473ms, structcheck: 33.866387ms, goconst: 20.107534ms, nakedret: 18.733497ms, typecheck: 3.026µs
INFO [runner] worker.1 took 6.519319968s with stages: ineffassign: 6.365728743s, gocyclo: 55.793279ms, scopelint: 54.41237ms, prealloc: 43.334344ms
INFO [runner] worker.4 took 8.181789092s with stages: goimports: 8.18176335s
INFO [runner] worker.2 took 19.884139475s with stages: megacheck: 19.884118886s
INFO [runner] Workers idle times: #1: 13.364830687s, #3: 17.774407932s, #4: 11.702474352s
INFO [runner/skip dirs] Skipped 19 issues from dir services/internal-apis/accounting/models/vendor by pattern (^|/)vendor($|/)
INFO [runner/skip dirs] Skipped 1 issues from dir services/internal-apis/file-relay/models/vendor by pattern (^|/)vendor($|/)
INFO [runner] Issues before processing: 952, after processing: 0
INFO [runner] Processors filtering stat (out/in): identifier_marker: 799/799, exclude-rules: 0/77, path_prettifier: 952/952, skip_files: 952/952, exclude: 77/799, cgo: 952/952, skip_dirs: 932/952, autogenerated_exclude: 799/932, filename_unadjuster: 952/952
INFO [runner] processing took 121.644512ms with stages: autogenerated_exclude: 60.206428ms, exclude: 21.734738ms, skip_dirs: 21.180425ms, identifier_marker: 12.700814ms, path_prettifier: 3.165726ms, filename_unadjuster: 1.316085ms, cgo: 1.252663ms, exclude-rules: 72.081µs, nolint: 4.232µs, max_same_issues: 2.18µs, diff: 1.897µs, source_code: 1.595µs, max_from_linter: 1.322µs, path_shortener: 1.316µs, skip_files: 1.13µs, max_per_file_from_linter: 972ns, uniq_by_line: 908ns
INFO File cache stats: 0 entries of total size 0B
INFO Memory: 106 samples, avg is 3354.2MB, max is 7537.1MB
INFO Execution took 24.269353126s

bmhatfield on 28 Sep 2019

One concern is that the new branch has a warning that doesn't exist on master:

WARN [runner] Can't run linter goanalysis_metalinter: fact_deprecated: failed prerequisites: fact_deprecated@[REDACTED]/api [[REDACTED]/api/userapi.test]

I don't know what fact_deprecated means in this context, and userapi.test is an entire packages worth of tests, so it's hard to track down what might be tripping up the linter.

Is it possible I'm seeing false memory decreases as a result of this warning?

bmhatfield on 28 Sep 2019

@bmhatfield thank you! It's unstable yet and will try to debug this warning. Memory decrease is because of huge rework of analysis and shouldn't be affected by the warning.

jirfag on 28 Sep 2019

Totally fair that it's unstable; completely acknowledged that I'm running from a PR branch. One other small note: the runner INFO messages that are currently on master also are not displaying here. So I am slightly uncertain if all the linters that were previously running are in fact still running.

bmhatfield on 28 Sep 2019

@bmhatfield it's ok because I've changed the linting algorithm and didn't implement timings for a new one yet. In previous versions of golangci-lint, each linter analyzed all packages serially and linters were running in parallel. In the prototype, each package is processed in parallel by all linters and after processing of each package it's unloaded from the memory (which gives memory usage decrease).

jirfag on 28 Sep 2019

🚀1

repo | active linters | v1.17.1 | v1.18.0 | v1.19.1 | prototype
| -- | -- | -- | -- | -- | -- |
digits/go-services (closed source, 200+ package monorepo) | [deadcode errcheck goconst gocritic gocyclo goimports gosimple ineffassign misspell nakedret prealloc scopelint staticcheck structcheck stylecheck typecheck unconvert unused varcheck] | n/a | 10s/2.23GB | 25s/8GB | 14s/2.8GB

bmhatfield on 28 Sep 2019

@bmhatfield I'm sorry it didn't become better than v1.17.1. It will be difficult without the source code but I will try to find out how to decrease memory even more

jirfag on 28 Sep 2019

@bmhatfield you can help if you will find out which linter eats a lot of memory

jirfag on 28 Sep 2019

By any chance, do you have suggestions for establishing that beyond enabling/disabling them one at a time?

bmhatfield on 28 Sep 2019

there is no debugging mechanism for it yet because of parallel running. Disable them by half using binary search :)

jirfag on 28 Sep 2019

Config: (on the prototype I disabled dogsled godox whitespace, since they aren't in 1.18)

golangci v0.18

$ golangci-lint run -v --print-resources-usage -c ./common/config/.golangci.yml ./...
INFO [config_reader] Used config file common/config/.golangci.yml
INFO [lintersdb] Active 21 linters: [deadcode errcheck gocritic gofmt goimports golint gosimple govet ineffassign interfacer lll maligned misspell staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck]
INFO [lintersdb] Optimized sublinters [staticcheck gosimple unused stylecheck] into metalinter megacheck
INFO [loader] Go packages loading at mode 991 (types|types_info|types_sizes|files|imports|syntax|compiled_files|deps|name) took 9.187964444s
INFO [loader] SSA repr building timing: packages building 111.880528ms, total 1.909969436s
INFO [runner] worker.5 took 4m38.440351149s with stages: ineffassign: 2m20.467720233s, gofmt: 2m16.782892549s, deadcode: 872.436116ms, maligned: 316.741329ms, typecheck: 4.333µs
INFO [runner] worker.3 took 4m38.679659319s with stages: misspell: 2m27.510904232s, govet: 2m3.740101627s, unconvert: 4.005739384s, varcheck: 1.667645077s, lll: 1.115568345s, structcheck: 393.13492ms, errcheck: 245.954732ms
INFO [runner] worker.8 took 4m39.076042771s with stages: gocritic: 4m39.076035022s
INFO [runner] worker.7 took 4m40.46926617s with stages: unparam: 4m40.46924975s
INFO [runner] worker.4 took 4m42.689282482s with stages: interfacer: 4m42.689271092s
INFO [runner] worker.2 took 4m43.375669785s with stages: goimports: 4m43.375660588s
INFO [runner] worker.6 took 6m57.377418315s with stages: golint: 6m57.374595204s
INFO [runner] worker.1 took 10m16.16298366s with stages: megacheck: 10m16.16216944s
INFO [runner] Workers idle times: #2: 5m32.783212721s, #3: 5m37.479014962s, #4: 5m33.451251964s, #5: 5m37.687893141s, #6: 3m18.778642464s, #7: 5m35.689466104s, #8: 5m37.072765204s
INFO [runner/skip dirs] Skipped dir pkg/test/istio.io/examples by pattern (^|/)examples($|/)
INFO [runner/skip dirs] Skipped dir galley/pkg/source/kube/builtin by pattern (^|/)builtin($|/)
INFO [runner] Issues before processing: 10635, after processing: 0
INFO [runner] processing took 1.937716329s with stages: autogenerated_exclude: 1.057425815s, exclude: 272.12384ms, identifier_marker: 206.856003ms, nolint: 120.622718ms, skip_dirs: 77.341161ms, path_prettifier: 73.721491ms, skip_files: 62.220775ms, cgo: 48.808345ms, filename_unadjuster: 11.378063ms, exclude-rules: 7.058967ms, diff: 128.295µs, max_from_linter: 8.99µs, path_shortener: 6.075µs, uniq_by_line: 4.518µs, source_code: 4.263µs, max_same_issues: 3.762µs, max_per_file_from_linter: 3.248µs
INFO File cache stats: 1896 entries of total size 14.6MiB
INFO Memory: 264 samples, avg is 20763.8MB, max is 33607.2MB
INFO Execution took 10m27.397462073s

golangci prototype

$ ~/git/golangci-lint/golangci-lint run -v --print-resources-usage -c ./common/config/.golangci.yml ./...
INFO [config_reader] Used config file common/config/.golangci.yml
INFO [lintersdb] Active 21 linters: [deadcode errcheck gocritic gofmt goimports golint gosimple govet ineffassign interfacer lll maligned misspell staticcheck structcheck stylecheck typecheck unconvert unparam unused varcheck]
INFO [loader] Go packages loading at mode 575 (exports_file|files|imports|types_sizes|deps|name|compiled_files) took 33.603773142s
INFO [loader] Loaded 1896 AST files in 576.034167ms
INFO [runner/skip dirs] Skipped 8 issues from dir galley/pkg/source/kube/builtin by pattern (^|/)builtin($|/)
INFO [runner/skip dirs] Skipped 9 issues from dir pkg/test/istio.io/examples by pattern (^|/)examples($|/)
INFO [runner] Issues before processing: 11769, after processing: 19
INFO [runner] Processors filtering stat (out/in): exclude: 298/2720, path_shortener: 19/19, cgo: 11769/11769, identifier_marker: 2720/2720, max_same_issues: 19/19, source_code: 19/19, exclude-rules: 145/298, diff: 19/19, autogenerated_exclude: 2720/2771, nolint: 19/145, uniq_by_line: 19/19, max_from_linter: 19/19, path_prettifier: 11769/11769, skip_dirs: 2771/2788, max_per_file_from_linter: 19/19, filename_unadjuster: 11769/11769, skip_files: 2788/11769
INFO [runner] processing took 413.328827ms with stages: autogenerated_exclude: 157.171943ms, identifier_marker: 75.638564ms, exclude: 65.633371ms, nolint: 37.759243ms, skip_files: 35.168167ms, path_prettifier: 25.537573ms, skip_dirs: 8.384812ms, source_code: 3.578241ms, cgo: 2.229089ms, filename_unadjuster: 1.672089ms, exclude-rules: 524.256µs, uniq_by_line: 12.514µs, path_shortener: 8.833µs, max_from_linter: 5.98µs, max_per_file_from_linter: 2.409µs, max_same_issues: 1.27µs, diff: 473ns
INFO [runner] linters took 3m10.248080062s with stages: unused: 1m41.68963915s, goanalysis_metalinter: 1m28.144648699s
istioctl/pkg/install/pre-check.go:62:23: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
                fmt.Fprintf(writer, fmt.Sprintf("Failed to initialize the Kubernetes client: %v.\n", err))
                                    ^
istioctl/pkg/install/pre-check.go:69:23: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
                fmt.Fprintf(writer, fmt.Sprintf("Failed to query the Kubernetes API Server: %v.\n", err))
                                    ^
pkg/envoy/instance.go:136:5: SA4003: no value of type uint32 is less than 0 (staticcheck)
        if adminPort <= 0 {
           ^
pkg/test/env/istio.go:40:2: ST1003: should not use ALL_CAPS in Go names; use CamelCase instead (stylecheck)
        ISTIO_GO Variable = "ISTIO_GO"
        ^
pkg/test/env/istio.go:44:2: ST1003: should not use ALL_CAPS in Go names; use CamelCase instead (stylecheck)
        ISTIO_BIN Variable = "ISTIO_BIN"
        ^
pkg/test/env/istio.go:48:2: ST1003: should not use ALL_CAPS in Go names; use CamelCase instead (stylecheck)
        ISTIO_OUT Variable = "ISTIO_OUT"
        ^
pkg/test/env/istio.go:60:2: ST1003: should not use ALL_CAPS in Go names; use CamelCase instead (stylecheck)
        PULL_POLICY Variable = "PULL_POLICY"
        ^
pkg/test/env/istio.go:65:2: ST1003: should not use ALL_CAPS in Go names; use CamelCase instead (stylecheck)
        ISTIO_TEST_KUBE_CONFIG Variable = "ISTIO_TEST_KUBE_CONFIG"
        ^
istioctl/pkg/kubernetes/client.go:133:23: ST1016: methods on the same type should have the same receiver name (seen 3x "c", 10x "client") (stylecheck)
func (client *Client) AllPilotsDiscoveryDo(pilotNamespace, method, path string, body []byte) (map[string][]byte, error) {
                      ^
galley/pkg/crd/validation/webhook.go:139:29: ST1016: methods on the same type should have the same receiver name (seen 1x "p", 1x "args") (stylecheck)
func (p *WebhookParameters) String() string {
                            ^
mixer/pkg/il/interpreter/interpreter.go:63:23: ST1016: methods on the same type should have the same receiver name (seen 2x "i", 1x "in") (stylecheck)
func (i *Interpreter) Eval(fnName string, bag attribute.Bag) (Result, error) {
                      ^
pilot/pkg/model/service.go:272:28: ST1016: methods on the same type should have the same receiver name (seen 1x "si", 1x "instance") (stylecheck)
func (si *ServiceInstance) GetLocality() string {
                           ^
pkg/mcp/source/source_test.go:239:116: verifySentResourcesMultipleTypes - result 0 (map[string]*istio.io/api/mcp/v1alpha1.Resources) is never used (unparam)
func verifySentResourcesMultipleTypes(t *testing.T, h *sourceTestHarness, wantResources map[string]*mcp.Resources) map[string]*mcp.Resources {
                                                                                                                   ^
pkg/mcp/snapshot/snapshot_test.go:91:138: createTestWatch - result 0 (*istio.io/istio/pkg/mcp/source.WatchResponse) is never used (unparam)
func createTestWatch(c source.Watcher, collection, version string, responseC chan *source.WatchResponse, wantResponse, wantCancel bool) (*source.WatchResponse, source.CancelWatchFunc, error) { // nolint: lll
                                                                                                                                         ^
pilot/pkg/model/config.go:364:49: sortConfigByCreationTime - result 0 ([]istio.io/istio/pilot/pkg/model.Config) is never used (unparam)
func sortConfigByCreationTime(configs []Config) []Config {
                                                ^
pilot/pkg/networking/core/v1alpha3/route/route.go:948:63: getEnvoyRouteTypeAndVal - result 1 (string) is never used (unparam)
func getEnvoyRouteTypeAndVal(r *route.Route) (envoyRouteType, string) {
                                                              ^
pilot/pkg/networking/core/v1alpha3/networkfilter.go:58:90: setAccessLog - result 0 (*github.com/envoyproxy/go-control-plane/envoy/config/filter/network/tcp_proxy/v2.TcpProxy) is never used (unparam)
func setAccessLog(env *model.Environment, node *model.Proxy, config *tcp_proxy.TcpProxy) *tcp_proxy.TcpProxy {
                                                                                         ^
galley/pkg/config/source/kube/apiserver/status/status.go:74:69: (*status).setDesired - result 0 (bool) is never used (unparam)
func (r *status) setDesired(v resource.Version, status interface{}) bool {
                                                                    ^
tests/e2e/tests/bookinfo/main_test.go:296:58: checkHTTPResponse - result 0 (int) is never used (unparam)
func checkHTTPResponse(gateway, expr string, count int) (int, error) {
                                                         ^
INFO File cache stats: 1910 entries of total size 14.8MiB
INFO Memory: 686 samples, avg is 6489.4MB, max is 15436.9MB
INFO Execution took 3m44.487537853s

Note that with dogsled godox whitespace it was actually slower, so one of those may be fairly slow?

tl;dr
before: 20763.8MB, max is 33607.2MB took 10m27.397462073s
after: 6489.4MB, max is 15436.9MB took 3m44.487537853s

Note this is on a laptop with 16gb of memory so it was probably thrashing the GC or something before, 10m is a bit artificially high.

On a subset of the repo (pilot/), v18 gives me 3.5gb/7.4gb/15s, prototype 1.5gb/2gb/8s. So looks fantastic.

If you want a giant repo to test of istio/istio is a good fit. For whatever reason our config is at ./common/config/.golangci.yml instead of at the top of the repo though

Right now in our CI we run v18 with 24gb of memory and GOGC=20

howardjohn on 28 Sep 2019

PS: For us, the real boundary line is 4GB memory consumption. Below that, assuming a reasonable running time of less than 30s, we're able to run it in our CircleCI environment. Above that, and their runners OOM the job.

So at 14s, 2.8GB, we're in great shape compared with 25s/8gb, which won't complete in our environment.

bmhatfield on 28 Sep 2019

@jirfag Found the linter consuming memory/time on the prototype: unused.

Without unused: 6s/407.4MB

deadcode errcheck goconst gocritic gocyclo goimports gosimple ineffassign misspell nakedret prealloc scopelint staticcheck structcheck stylecheck typecheck unconvert varcheck

With unused: 10s/2.2GB

deadcode errcheck goconst gocritic gocyclo goimports gosimple ineffassign misspell nakedret prealloc scopelint staticcheck structcheck stylecheck typecheck unconvert unused varcheck

(note: with unused enabled there seems to be a fairly significant variance (±1.2GB) in memory consumed from run to run; I saw as low as 2.2GB and as high as 3.4GB; with unused disabled the variance reduced to ±50MB)

Edit: For completeness, I also performed the same comparison (unused enabled/disabled) on current master; it had no meaningful impact on memory consumed or running time. I believe that's expected due to the change in running strategy, but wanted to double-check anyways.

bmhatfield on 28 Sep 2019

#758 was merged.
I've made a bit more optimizations in #764
Unused is the only linter that eats a ton of memory because it can't work incrementally. You can disable it. You can make an issue in staticcheck GitHub repository about it: I think its optimization is possible.
I'm going to close this issue after merging #764 Please, write your comments if you think we need to fix something with memory before that.

jirfag on 30 Sep 2019

🎉3

Do you plan to release another tag ?

pierrre on 1 Oct 2019

@jirfag thanks a lot for the recent optimizations!
Golangci-lint (df4f6766baff8f2ce10ae7a6a4d81fe37b729989) now only uses 68MB RAM on my machine.
Before it used all my 16GB and freezed my machine. :-)