Mattermost-server: Image causes Mattermost server to crash: fatal error: runtime: out of memory

[wezm]

Per Mattermost guidelines, GitHub issues are for bug reports: http://www.mattermost.org/filing-issues/

For troubleshooting see: http://forum.mattermost.org/.
For feature proposals see: http://www.mattermost.org/feature-requests/

If you've found a bug--something appears unintentional--please report using the following format:

Summary

Image causes Mattermost to crash

Steps to reproduce

Reply to a thread with this image:

The image was generated as follows:

• Take screenshot
• Crop in gThumb
• Add arrow in Pinta

Expected behavior

The image is uploaded and can be posted.

Observed behavior (that appears unintentional)

Mattermost crashes with, "fatal error: runtime: out of memory", error. There's nothing to indicate the server is actually out of memory. Here is a graph of memory use on it:

Mar 12 22:18:22 gareth platform[668]: fatal error: runtime: out of memory
Mar 12 22:18:22 gareth platform[668]: runtime stack:
Mar 12 22:18:22 gareth platform[668]: runtime.throw(0x13c10ad, 0x16)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/panic.go:605 +0x95
Mar 12 22:18:22 gareth platform[668]: runtime.sysMap(0xc4471c0000, 0x24000000, 0x100, 0x241b058)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
Mar 12 22:18:22 gareth platform[668]: runtime.(*mheap).sysAlloc(0x24016a0, 0x24000000, 0x1)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/malloc.go:470 +0xd7
Mar 12 22:18:22 gareth platform[668]: runtime.(*mheap).grow(0x24016a0, 0x12000, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/mheap.go:887 +0x60
Mar 12 22:18:22 gareth platform[668]: runtime.(*mheap).allocSpanLocked(0x24016a0, 0x12000, 0x241b068, 0x8000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/mheap.go:800 +0x334
Mar 12 22:18:22 gareth platform[668]: runtime.(*mheap).alloc_m(0x24016a0, 0x12000, 0xffffffffffff0101, 0x7fca96cf0df8)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/mheap.go:666 +0x118
Mar 12 22:18:22 gareth platform[668]: runtime.(*mheap).alloc.func1()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/mheap.go:733 +0x4d
Mar 12 22:18:22 gareth platform[668]: runtime.systemstack(0x7fca96cf0e10)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/asm_amd64.s:360 +0xab
Mar 12 22:18:22 gareth platform[668]: runtime.(*mheap).alloc(0x24016a0, 0x12000, 0x2010101, 0x7fca98cd9f00)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/mheap.go:732 +0xa1
Mar 12 22:18:22 gareth platform[668]: runtime.largeAlloc(0x24000000, 0x101, 0x23f8ed0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/malloc.go:827 +0x98
Mar 12 22:18:22 gareth platform[668]: runtime.mallocgc.func1()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/malloc.go:722 +0x46
Mar 12 22:18:22 gareth platform[668]: runtime.systemstack(0xc42001c000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/asm_amd64.s:344 +0x79
Mar 12 22:18:22 gareth platform[668]: runtime.mstart()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/proc.go:1135
Mar 12 22:18:22 gareth platform[668]: goroutine 2972 [running]:
Mar 12 22:18:22 gareth platform[668]: runtime.systemstack_switch()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc422c42ea8 sp=0xc422c42ea0 pc=0x45be30
Mar 12 22:18:22 gareth platform[668]: runtime.mallocgc(0x24000000, 0x11ab700, 0xc422c43301, 0xc422c431c8)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/malloc.go:721 +0x7ae fp=0xc422c42f50 sp=0xc422c42ea8 pc=0x412a1e
Mar 12 22:18:22 gareth platform[668]: runtime.makeslice(0x11ab700, 0x24000000, 0x24000000, 0xc422c433d0, 0xc422c431b0, 0xc420014028)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/slice.go:54 +0x77 fp=0xc422c42f80 sp=0xc422c42f50 pc=0x4455f7
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectMultipartStreamNoLength(0xc4224be200, 0xc421680300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4224bd980, 0xc421e56320, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:202 +0x374 fp=0xc422c435f0 sp=0xc422c42f80 pc=0x952c64
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectCommon(0xc4224be200, 0xc421680300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4224bd980, 0xc421e56320, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:152 +0x5fc fp=0xc422c43870 sp=0xc422c435f0 pc=0x9526cc
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObjectWithContext(0xc4224be200, 0xc421680300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4224bd980, 0xc421e56320, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-context.go:38 +0x2e6 fp=0xc422c439f0 sp=0xc422c43870 pc=0x949856
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObject(0xc4224be200, 0xc421680300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4224bd980, 0xc421e56320, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:130 +0x106 fp=0xc422c43b70 sp=0xc422c439f0 pc=0x952086
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/utils.(*S3FileBackend).WriteFile(0xc4216802a0, 0xc422d7a000, 0x35d5, 0x65d5, 0xc422c12370, 0xa3, 0xc421e68180)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/file_backend_s3.go:152 +0x225 fp=0xc422c43e10 sp=0xc422c43b70 pc=0x9a22d5
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).WriteFile(0xc42012c000, 0xc422d7a000, 0x35d5, 0x65d5, 0xc422c12370, 0xa3, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:85 +0x8e fp=0xc422c43e58 sp=0xc422c43e10 pc=0xc978ee
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).generateThumbnailImage(0xc42012c000, 0x1ccfd60, 0xc422686100, 0xc422c12370, 0xa3, 0x34c, 0xd1)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:477 +0x163 fp=0xc422c43f60 sp=0xc422c43e58 pc=0xc9d5a3
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).HandleImages.func1(0xc422924eb0, 0xc42012c000, 0xc4215941a0, 0xc422c12370, 0xa3, 0x34c, 0xd1)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:380 +0x93 fp=0xc422c43fa8 sp=0xc422c43f60 pc=0xd26443
Mar 12 22:18:22 gareth platform[668]: runtime.goexit()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc422c43fb0 sp=0xc422c43fa8 pc=0x45ea61
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/app.(*App).HandleImages
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:378 +0x154
Mar 12 22:18:22 gareth platform[668]: goroutine 1 [chan receive, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: main.runServer(0x13aca74, 0xb, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/cmd/platform/server.go:165 +0xda5
Mar 12 22:18:22 gareth platform[668]: main.runServerCmd(0x23e62a0, 0x24191a8, 0x0, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/cmd/platform/server.go:45 +0xde
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(*Command).execute(0x23e62a0, 0xc4200101a0, 0x0, 0x0, 0x23e62a0, 0xc4200101a0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:746 +0x475
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x23e62a0, 0x23e2160, 0x23e23a0, 0x23e25e0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:831 +0x30e
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(*Command).Execute(0x23e62a0, 0xc42004bf70, 0x10cf762)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:784 +0x2b
Mar 12 22:18:22 gareth platform[668]: main.main()
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/cmd/platform/mattermost.go:31 +0x2d
Mar 12 22:18:22 gareth platform[668]: goroutine 2504 [chan receive, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.(*ConsoleLogWriter).run(0xc4213dc5e0, 0x1cc7320, 0xc42000e018)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/termlog.go:33 +0x116
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.NewConsoleLogWriter
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/termlog.go:26 +0xcf
Mar 12 22:18:22 gareth platform[668]: goroutine 8 [syscall, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: os/signal.signal_recv(0x70ff20)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/sigqueue.go:131 +0xa6
Mar 12 22:18:22 gareth platform[668]: os/signal.loop()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Mar 12 22:18:22 gareth platform[668]: created by os/signal.init.0
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Mar 12 22:18:22 gareth platform[668]: goroutine 2975 [sleep]:
Mar 12 22:18:22 gareth platform[668]: time.Sleep(0x1ef1deb2)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/time.go:65 +0x130
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc42244f980, 0xa, 0xc42244f800, 0xc42149a780)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x15c
Mar 12 22:18:22 gareth platform[668]: goroutine 2228 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/utils.NewHTMLTemplateWatcher.func1(0xc42202ffe0, 0xc421fab770, 0xc42201efe0, 0x1a)
Mar 12 22:18:22 gareth systemd[1]: mattermost.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/html.go:53 +0x1f8
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/utils.NewHTMLTemplateWatcher
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/html.go:48 +0x322
Mar 12 22:18:22 gareth platform[668]: goroutine 2229 [chan receive, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: database/sql.(*DB).connectionOpener(0xc4201725a0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/database/sql/sql.go:871 +0x53
Mar 12 22:18:22 gareth platform[668]: created by database/sql.Open
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/database/sql/sql.go:609 +0x1ee
Mar 12 22:18:22 gareth platform[668]: goroutine 2523 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go.(*Client).loop(0xc421f71100)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go/analytics.go:338 +0x1b5
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go.(*Client).startLoop
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go/analytics.go:232 +0x3f
Mar 12 22:18:22 gareth platform[668]: goroutine 2520 [select, 1 minutes, locked to thread]:
Mar 12 22:18:22 gareth platform[668]: runtime.gopark(0x14334e8, 0x0, 0x13a51d9, 0x6, 0x18, 0x1)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/proc.go:287 +0x12c
Mar 12 22:18:22 gareth platform[668]: runtime.selectgo(0xc42139df50, 0xc4215b7560)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/select.go:395 +0x1149
Mar 12 22:18:22 gareth platform[668]: runtime.ensureSigM.func1()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/signal_unix.go:511 +0x220
Mar 12 22:18:22 gareth platform[668]: runtime.goexit()
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1
Mar 12 22:18:22 gareth platform[668]: goroutine 2519 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/jobs.(*Schedulers).Start.func1.1()
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/schedulers.go:79 +0x4ad
Mar 12 22:18:22 gareth platform[668]: sync.(*Once).Do(0xc4200f6728, 0xc42136ffb8)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/sync/once.go:44 +0xbe
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/jobs.(*Schedulers).Start.func1(0xc4200f6700)
Mar 12 22:18:22 gareth systemd[1]: mattermost.service: Unit entered failed state.
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/schedulers.go:61 +0x4c
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/jobs.(*Schedulers).Start
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/schedulers.go:60 +0xb9
Mar 12 22:18:22 gareth platform[668]: goroutine 2518 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/jobs.(*Watcher).Start(0xc42130c990)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/jobs_watcher.go:51 +0x294
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/jobs.(*Workers).Start.func1
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/workers.go:80 +0x88
Mar 12 22:18:22 gareth platform[668]: goroutine 2885 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc422d907e0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53
Mar 12 22:18:22 gareth platform[668]: goroutine 2491 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*Hub).Start.func1()
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:370 +0x301
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*Hub).Start.func2()
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:458 +0x4c
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/app.(*Hub).Start
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:475 +0x128
Mar 12 22:18:22 gareth platform[668]: goroutine 2893 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc422d90c60)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53
Mar 12 22:18:22 gareth platform[668]: goroutine 2207 [syscall, 1 minutes]:
Mar 12 22:18:22 gareth systemd[1]: mattermost.service: Failed with result 'exit-code'.
Mar 12 22:18:22 gareth platform[668]: syscall.Syscall6(0xe8, 0xa, 0xc422545c24, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc422545c24, 0x7, 0x7, 0xffffffffffffffff, 0xc4213a2a80, 0xc421fa9c80, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1544 +0x77
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc42201f000, 0x1431200, 0xc42201f000, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x91
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc421fab770)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x194
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.NewWatcher
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x23a
Mar 12 22:18:22 gareth platform[668]: goroutine 2179 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/utils.NewConfigWatcher.func1(0xc421c43ea0, 0xc421faaa50, 0xc421e53950, 0x22, 0xc421e53950, 0x22, 0xc422026bc0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/config.go:201 +0x1e5
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/utils.NewConfigWatcher
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/config.go:196 +0x1bb
Mar 12 22:18:22 gareth platform[668]: goroutine 2178 [syscall, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: syscall.Syscall6(0xe8, 0x6, 0xc422565c24, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix.EpollWait(0x6, 0xc422565c24, 0x7, 0x7, 0xffffffffffffffff, 0xc4213a2d80, 0xc421fa9e00, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1544 +0x77
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(*fdPoller).wait(0xc42201e960, 0x1431200, 0xc42201e960, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x91
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc421faaa50)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x194
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.NewWatcher
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x23a
Mar 12 22:18:22 gareth platform[668]: goroutine 2882 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc421f6a120)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53
Mar 12 22:18:22 gareth platform[668]: goroutine 2889 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*WebConn).writePump(0xc4214c8480)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:164 +0x243
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*WebConn).Pump.func1(0xc4214c8480, 0xc421cce600)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:110 +0x2b
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/app.(*WebConn).Pump
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:109 +0x6b
Mar 12 22:18:22 gareth platform[668]: goroutine 2969 [sleep]:
Mar 12 22:18:22 gareth platform[668]: time.Sleep(0x32f28b9f)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/time.go:65 +0x130
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc42244e600, 0xa, 0xc42244e000, 0xc4224bca80)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x15c
Mar 12 22:18:22 gareth platform[668]: goroutine 2872 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc422d90a20)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53
Mar 12 22:18:22 gareth platform[668]: goroutine 2974 [sleep]:
Mar 12 22:18:22 gareth platform[668]: time.Sleep(0x33c80b51)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/time.go:65 +0x130
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc42244eb40, 0xa, 0xc42244eae0, 0xc4214814a0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x15c
Mar 12 22:18:22 gareth platform[668]: goroutine 2884 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8f370, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4215fdc98, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4215fdc98, 0xc422688000, 0x4000, 0x4000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4215fdc80, 0xc422688000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4215fdc80, 0xc422688000, 0x4000, 0x4000, 0xc421676380, 0x7fca98deb000, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b31a28, 0xc422688000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc421dc4360, 0x7fca98d4c9c0, 0xc422b31a28, 0x5, 0xc422b31a28, 0xc421dc54d0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc4216a0e00, 0x1433617, 0xc4216a0f20, 0x42f57b)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc4216a0e00, 0xc4216aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc422d907e0, 0xc4216aa000, 0x1000, 0x1000, 0xc422b874a0, 0xc421dbfdf8, 0x458ad0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc421b394a0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc421b394a0, 0x1, 0x0, 0x0, 0x1, 0xc421dbfc80, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc422d907e0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2886 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8f7f0, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc421ee4f98, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc421ee4f98, 0xc421f94000, 0x2000, 0x2000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc421ee4f80, 0xc421f94000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc421ee4f80, 0xc421f94000, 0x2000, 0x2000, 0x4cff9c, 0xc4216d15bd, 0x13a2a70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b31aa8, 0xc421f94000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc421b39560)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc421b39560, 0x2, 0x7622934a00, 0x229f1835a6, 0x18, 0x7ffec7d29c06, 0xc422915e00)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(*Conn).read(0xc4214a8780, 0x2, 0xc422c46980, 0x256a15092, 0x7fca98d8f880, 0xc421f6b440, 0x1)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/conn_read.go:12 +0x40
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(*Conn).advanceFrame(0xc4214a8780, 0x12ec440, 0x0, 0x1c0000000000)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/conn.go:780 +0x5c
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(*Conn).NextReader(0xc4214a8780, 0x44ded8, 0xb56a14d24, 0xc422c46b20, 0x7e4aa1, 0xc422c46b20)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/conn.go:940 +0xa3
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(*Conn).ReadJSON(0xc4214a8780, 0x1203180, 0xc4222e9ad0, 0x23f9100, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/json.go:50 +0x2f
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*WebConn).readPump(0xc4214c8480)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:138 +0x17e
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*WebConn).Pump(0xc4214c8480)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:113 +0x79
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/api4.connectWebSocket(0xc422a13e00, 0x1cd00e0, 0xc422dd8700, 0xc421252c00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/websocket.go:39 +0x320
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/api4.handler.ServeHTTP(0xc42012c000, 0x14307f8, 0x100, 0x1cd00e0, 0xc422dd8700, 0xc421252c00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/context.go:155 +0xb6e
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/api4.(*handler).ServeHTTP(0xc4227ec4a0, 0x1cd00e0, 0xc422dd8700, 0xc421252c00)
Mar 12 22:18:22 gareth platform[668]:         <autogenerated>:1 +0x63
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc420153f80, 0x1cd00e0, 0xc422dd8700, 0xc421252c00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux/mux.go:159 +0xed
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*CorsWrapper).ServeHTTP(0xc421595f10, 0x1cd00e0, 0xc422dd8700, 0xc421252600)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:81 +0xa2
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP(0x1cc50e0, 0xc421595f10, 0x1cc5120, 0x24191a8, 0x1, 0x1cd00e0, 0xc422dd8700, 0xc421252600)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers/recovery.go:78 +0xb9
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP(0xc4214b4420, 0x1cd00e0, 0xc422dd8700, 0xc421252600)
Mar 12 22:18:22 gareth platform[668]:         <autogenerated>:1 +0x76
Mar 12 22:18:22 gareth platform[668]: net/http.serverHandler.ServeHTTP(0xc422309a00, 0x1cd00e0, 0xc422dd8700, 0xc421252600)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:2619 +0xb4
Mar 12 22:18:22 gareth platform[668]: net/http.(*conn).serve(0xc422a82140, 0x1cd0e20, 0xc421759980)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:1801 +0x71d
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Server).Serve
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:2720 +0x288
Mar 12 22:18:22 gareth platform[668]: goroutine 2968 [sleep]:
Mar 12 22:18:22 gareth platform[668]: time.Sleep(0x22b26367)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/time.go:65 +0x130
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc421939380, 0xa, 0xc4219390e0, 0xc422fda2a0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x15c
Mar 12 22:18:22 gareth platform[668]: goroutine 2966 [IO wait]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8f8b0, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4200f4498, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4200f4498, 0xc422fd4000, 0x1000, 0x1000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4200f4480, 0xc422fd4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4200f4480, 0xc422fd4000, 0x1000, 0x1000, 0xc421f6f500, 0x0, 0x413576)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc421286ba0, 0xc422fd4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc422cc3860, 0x7fca98d4c9c0, 0xc421286ba0, 0x5, 0xc421286ba0, 0x413576)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc422dd0a80, 0x1433617, 0xc422dd0ba0, 0x42f57b)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc422dd0a80, 0xc422fd5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc422cc0d80, 0xc422fd5000, 0x1000, 0x1000, 0xc421289da0, 0xc421938cb8, 0x458ad0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc422fd0960)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc422fd0960, 0x1, 0x0, 0x0, 0x0, 0xc4215b7a40, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc422cc0d80)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2867 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8f130, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4215fc818, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4215fc818, 0xc4219b8000, 0x1000, 0x1000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4215fc800, 0xc4219b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4215fc800, 0xc4219b8000, 0x1000, 0x1000, 0x5, 0xc422ce6780, 0x4e8dd8)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b316b8, 0xc4219b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc4212b38f0, 0x7fca98d4c9c0, 0xc422b316b8, 0x5, 0xc422b316b8, 0xc42126a120)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc422d57180, 0x1433617, 0xc422d572a0, 0x42f57b)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc422d57180, 0xc42148e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc422d905a0, 0xc42148e000, 0x1000, 0x1000, 0xc42285d1e0, 0xc421c90118, 0x458ad0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc421b383c0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc421b383c0, 0x1, 0x0, 0x0, 0x1, 0xc421c2ff80, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc422d905a0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2490 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*Hub).Start.func1()
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:370 +0x301
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*Hub).Start.func2()
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:458 +0x4c
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/app.(*Hub).Start
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:475 +0x128
Mar 12 22:18:22 gareth platform[668]: goroutine 2516 [sleep, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: time.Sleep(0x34630b8a000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/time.go:65 +0x130
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/enterprise/compliance.(*ComplianceInterfaceImpl).StartComplianceDailyJob.func1(0xc421e11560)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/enterprise/compliance/compliance.go:327 +0xa7
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/enterprise/compliance.(*ComplianceInterfaceImpl).StartComplianceDailyJob
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/enterprise/compliance/compliance.go:273 +0x3f
Mar 12 22:18:22 gareth platform[668]: goroutine 2991 [chan receive]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*Transport).getConn.func2.1(0xc4222f0000, 0x23e09a0, 0x1432fe8)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:932 +0x50
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).getConn.func2
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:931 +0x7a
Mar 12 22:18:22 gareth platform[668]: goroutine 2817 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8feb0, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4213c3518, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4213c3518, 0xc4229e2000, 0x1000, 0x1000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4213c3500, 0xc4229e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4213c3500, 0xc4229e2000, 0x1000, 0x1000, 0xc421676380, 0x7fca98deb000, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b313e0, 0xc4229e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc422ba3830, 0x7fca98d4c9c0, 0xc422b313e0, 0x5, 0xc422b313e0, 0xc422afc240)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc422d56a80, 0x1433617, 0xc422d56ba0, 0x42f57b)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc422d56a80, 0xc421bfb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc421deb0e0, 0xc421bfb000, 0x1000, 0x1000, 0xc4228499c0, 0xc421a7afb8, 0x458ad0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc422c81e60)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc422c81e60, 0x1, 0x0, 0x0, 0x1, 0xc421a7aea0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc421deb0e0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2941 [semacquire]:
Mar 12 22:18:22 gareth platform[668]: sync.runtime_Semacquire(0xc422924ebc)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/sema.go:56 +0x39
Mar 12 22:18:22 gareth platform[668]: sync.(*WaitGroup).Wait(0xc422924eb0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/sync/waitgroup.go:131 +0x72
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).HandleImages(0xc42012c000, 0xc421594170, 0x1, 0x1, 0xc421594180, 0x1, 0x1, 0xc421e562a0, 0x1, 0x1)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:389 +0x1d7
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).UploadFiles(0xc42012c000, 0x13a4d23, 0x6, 0xc422dcaba0, 0x1a, 0xc4229f4e60, 0x1a, 0xc422b31d90, 0x1, 0x1, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:325 +0xd6b
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/api4.uploadFile(0xc422be4b40, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8f00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/file.go:91 +0x4bb
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/api4.handler.ServeHTTP(0xc42012c000, 0x1430e10, 0x10001, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8f00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/context.go:155 +0xb6e
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/api4.(*handler).ServeHTTP(0xc422695340, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8f00)
Mar 12 22:18:22 gareth platform[668]:         <autogenerated>:1 +0x63
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc420153f80, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8f00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux/mux.go:159 +0xed
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*CorsWrapper).ServeHTTP(0xc421595f10, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8d00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:81 +0xa2
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP(0x1cc50e0, 0xc421595f10, 0x1cc5120, 0x24191a8, 0x1, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8d00)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers/recovery.go:78 +0xb9
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP(0xc4214b4420, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8d00)
Mar 12 22:18:22 gareth platform[668]:         <autogenerated>:1 +0x76
Mar 12 22:18:22 gareth platform[668]: net/http.serverHandler.ServeHTTP(0xc422309a00, 0x1cd00e0, 0xc422dd9c00, 0xc422dc8d00)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:2619 +0xb4
Mar 12 22:18:22 gareth platform[668]: net/http.(*conn).serve(0xc422c70dc0, 0x1cd0e20, 0xc422d04680)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:1801 +0x71d
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Server).Serve
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:2720 +0x288
Mar 12 22:18:22 gareth platform[668]: goroutine 2949 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8fa30, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc421ea6798, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc421ea6798, 0xc422c10b00, 0x1, 0x1)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc421ea6780, 0xc422c10bb1, 0x1, 0x1, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc421ea6780, 0xc422c10bb1, 0x1, 0x1, 0xc420050c60, 0x0, 0x1cc6501)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b31d00, 0xc422c10bb1, 0x1, 0x1, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: net/http.(*connReader).backgroundRead(0xc422c10ba0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:660 +0x62
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*connReader).startBackgroundRead
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:656 +0xd8
Mar 12 22:18:22 gareth platform[668]: goroutine 2990 [select]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc4200765a0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53
Mar 12 22:18:22 gareth platform[668]: goroutine 2505 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.NewFileLogWriter.func1(0xc420172c80)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/filelog.go:87 +0x140
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.NewFileLogWriter
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/filelog.go:77 +0x26a
Mar 12 22:18:22 gareth platform[668]: goroutine 2992 [runnable]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8f730, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4224bff98, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4224bff98, 0xc420090c00, 0x400, 0x400)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4224bff80, 0xc420090c00, 0x400, 0x400, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4224bff80, 0xc420090c00, 0x400, 0x400, 0x0, 0xc420090c00, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b302a0, 0xc420090c00, 0x400, 0x400, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc4220d45a0, 0x7fca98d4c9c0, 0xc422b302a0, 0x5, 0xc422b302a0, 0xb9)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc422c9ae00, 0xc42001c016, 0x400, 0xb9)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readHandshake(0xc422c9ae00, 0xc422659316, 0xc422659380, 0xb4, 0xb4)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:943 +0x8d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).clientHandshake(0xc422c9ae00, 0x1433618, 0xc422c9af20)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/handshake_client.go:159 +0x9ca
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Handshake(0xc422c9ae00, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1307 +0x18f
Mar 12 22:18:22 gareth platform[668]: net/http.(*Transport).dialConn.func3(0x0, 0xc422c9ae00, 0xc4226870c0, 0xc421681800)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1151 +0x42
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1147 +0xdbe
Mar 12 22:18:22 gareth platform[668]: goroutine 2881 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8f5b0, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc421ee4d18, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc421ee4d18, 0xc4219b9000, 0x1000, 0x1000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc421ee4d00, 0xc4219b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc421ee4d00, 0xc4219b9000, 0x1000, 0x1000, 0xc421676380, 0x7fca98deb000, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b31a80, 0xc4219b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc42126ba70, 0x7fca98d4c9c0, 0xc422b31a80, 0x5, 0xc422b31a80, 0xc421dc5620)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc4216a1180, 0x1433617, 0xc4216a12a0, 0x42f57b)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc4216a1180, 0xc422dc0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc421f6a120, 0xc422dc0000, 0x1000, 0x1000, 0xc4228a5ac0, 0xc421c918b8, 0x458ad0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc421b39140)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc421b39140, 0x1, 0x0, 0x0, 0x0, 0xc421c91740, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc421f6a120)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2492 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).HubStart.func1(0xc42012c000)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:90 +0x5d6
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/app.(*App).HubStart
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:82 +0x5b7
Mar 12 22:18:22 gareth platform[668]: goroutine 2973 [select]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).roundTrip(0xc4200765a0, 0xc4221010e0, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1970 +0x608
Mar 12 22:18:22 gareth platform[668]: net/http.(*Transport).RoundTrip(0x23e09a0, 0xc421fcef00, 0x23e09a0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:413 +0x992
Mar 12 22:18:22 gareth platform[668]: net/http.send(0xc421fcef00, 0x1cc6c60, 0x23e09a0, 0x0, 0x0, 0x0, 0xc422b30280, 0xc4221be498, 0x41360d, 0xc422b30280)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/client.go:249 +0x1a9
Mar 12 22:18:22 gareth platform[668]: net/http.(*Client).send(0xc421481b00, 0xc421fcef00, 0x0, 0x0, 0x0, 0xc422b30280, 0x0, 0x1, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/client.go:173 +0xfd
Mar 12 22:18:22 gareth platform[668]: net/http.(*Client).Do(0xc421481b00, 0xc421fcef00, 0x0, 0x0, 0xc4221be6f0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/client.go:602 +0x28d
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.do(0xc4224be680, 0xc4216804e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421481b00, 0xc421e56480, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:465 +0x50
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.executeMethod(0xc4224be680, 0xc4216804e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421481b00, 0xc421e56480, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:570 +0x34a
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.uploadPart(0xc4224be680, 0xc4216804e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421481b00, 0xc421e56480, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-multipart.go:283 +0x5e1
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectMultipartStreamNoLength(0xc4224be680, 0xc4216804e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421481b00, 0xc421e56480, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:219 +0x6e7
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectCommon(0xc4224be680, 0xc4216804e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421481b00, 0xc421e56480, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:152 +0x5fc
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObjectWithContext(0xc4224be680, 0xc4216804e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421481b00, 0xc421e56480, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-context.go:38 +0x2e6
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObject(0xc4224be680, 0xc4216804e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421481b00, 0xc421e56480, ...)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:130 +0x106
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/utils.(*S3FileBackend).WriteFile(0xc421680480, 0xc4223be000, 0xa6d7, 0xf000, 0xc422c122c0, 0xa5, 0xc421e680c0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/file_backend_s3.go:152 +0x225
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).WriteFile(0xc42012c000, 0xc4223be000, 0xa6d7, 0xf000, 0xc422c122c0, 0xa5, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:85 +0x8e
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).generatePreviewImage(0xc42012c000, 0x1ccfd60, 0xc422686100, 0xc422c122c0, 0xa5, 0x34c)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:499 +0x133
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).HandleImages.func2(0xc422924eb0, 0xc42012c000, 0xc4215941a0, 0xc422c122c0, 0xa5, 0x34c)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:385 +0x89
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/app.(*App).HandleImages
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:383 +0x1c9
Mar 12 22:18:22 gareth platform[668]: goroutine 2489 [runnable]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8fdf0, 0x72, 0xffffffffffffffff)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4224be198, 0x72, 0xc4221c0f00, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4224be198, 0xffffffffffffff00, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Accept(0xc4224be180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).accept(0xc4224be180, 0xc4221c10f8, 0x401c27, 0xc422c70f80)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:238 +0x42
Mar 12 22:18:22 gareth platform[668]: net.(*TCPListener).accept(0xc422b30450, 0x69ea98, 0x45b1a0, 0xc4221c1140)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
Mar 12 22:18:22 gareth platform[668]: net.(*TCPListener).Accept(0xc422b30450, 0x1432e00, 0xc422c70f00, 0x1cd0ee0, 0xc421ef3470)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/tcpsock.go:247 +0x49
Mar 12 22:18:22 gareth platform[668]: net/http.(*Server).Serve(0xc422309a00, 0x1ccfe60, 0xc422b30450, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/server.go:2695 +0x1b2
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/app.(*App).StartServer.func2(0xc42012c000, 0xc422bce4d0, 0x1ccfe60, 0xc422b30450)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:209 +0x774
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/app.(*App).StartServer
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:192 +0xce5
Mar 12 22:18:22 gareth platform[668]: goroutine 2231 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: database/sql.(*DB).connectionCleaner(0xc4201725a0, 0x34630b8a000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/database/sql/sql.go:793 +0x446
Mar 12 22:18:22 gareth platform[668]: created by database/sql.(*DB).startCleanerLocked
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/database/sql/sql.go:780 +0xba
Mar 12 22:18:22 gareth platform[668]: goroutine 2983 [sleep]:
Mar 12 22:18:22 gareth platform[668]: time.Sleep(0x4dfcb2a)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/time.go:65 +0x130
Mar 12 22:18:22 gareth platform[668]: github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc4215e8060, 0xa, 0xc4215e8000, 0xc4221008a0)
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
Mar 12 22:18:22 gareth platform[668]: created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
Mar 12 22:18:22 gareth platform[668]:         /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x15c
Mar 12 22:18:22 gareth platform[668]: goroutine 2892 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98ce3268, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4215fd118, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4215fd118, 0xc42245a000, 0x2000, 0x2000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4215fd100, 0xc42245a000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4215fd100, 0xc42245a000, 0x2000, 0x2000, 0x4, 0xc42248f780, 0x4e8dd8)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b31aa0, 0xc42245a000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc421dc4ed0, 0x7fca98d4c9c0, 0xc422b31aa0, 0x5, 0xc422b31aa0, 0xc421dc5f20)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc4216a1c00, 0x1433617, 0xc4216a1d20, 0x42f57b)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc4216a1c00, 0xc422b0c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc422d90c60, 0xc422b0c000, 0x1000, 0x1000, 0xc422b87b40, 0xc421be08f8, 0x458ad0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc421b39a40)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc421b39a40, 0x1, 0x0, 0x0, 0x1, 0xc421be07e0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc422d90c60)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2868 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc422d905a0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53
Mar 12 22:18:22 gareth platform[668]: goroutine 2871 [IO wait, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98ce3328, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4215fdb18, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4215fdb18, 0xc422b6a000, 0x4000, 0x4000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4215fdb00, 0xc422b6a000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4215fdb00, 0xc422b6a000, 0x4000, 0x4000, 0xc421676380, 0x7fca98deb000, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b317c0, 0xc422b6a000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc4218f54a0, 0x7fca98d4c9c0, 0xc422b317c0, 0x5, 0xc422b317c0, 0xc422afc060)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc422d57880, 0x1433617, 0xc422d579a0, 0x42f57b)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc422d57880, 0xc421942000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc422d90a20, 0xc421942000, 0x1000, 0x1000, 0xc422b877a0, 0xc421c90bf8, 0x458ad0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc421b387e0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc421b387e0, 0x1, 0x0, 0x0, 0x1, 0xc421be0180, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc422d90a20)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2984 [chan receive]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*Transport).dialConn(0x23e09a0, 0x1cd0e60, 0xc420014028, 0x0, 0xc420045680, 0x5, 0xc4216ade60, 0x14, 0x1, 0x0, ...)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1157 +0xdf4
Mar 12 22:18:22 gareth platform[668]: net/http.(*Transport).getConn.func4(0x23e09a0, 0x1cd0e60, 0xc420014028, 0xc422101170, 0xc4222f0000)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:943 +0x78
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).getConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:942 +0x393
Mar 12 22:18:22 gareth platform[668]: goroutine 2989 [IO wait]:
Mar 12 22:18:22 gareth platform[668]: internal/poll.runtime_pollWait(0x7fca98d8f970, 0x72, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/runtime/netpoll.go:173 +0x57
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).wait(0xc4224bf198, 0x72, 0xffffffffffffff00, 0x1cc9f60, 0x1cc2b70)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*pollDesc).waitRead(0xc4224bf198, 0xc4228ab400, 0x400, 0x400)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Mar 12 22:18:22 gareth platform[668]: internal/poll.(*FD).Read(0xc4224bf180, 0xc4228ab400, 0x400, 0x400, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
Mar 12 22:18:22 gareth platform[668]: net.(*netFD).Read(0xc4224bf180, 0xc4228ab400, 0x400, 0x400, 0xc421f6f500, 0x0, 0x413576)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/fd_unix.go:202 +0x52
Mar 12 22:18:22 gareth platform[668]: net.(*conn).Read(0xc422b30230, 0xc4228ab400, 0x400, 0x400, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/net.go:176 +0x6d
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*block).readFromUntil(0xc4215703f0, 0x7fca98d4c9c0, 0xc422b30230, 0x5, 0xc422b30230, 0x7ffec7d29c06)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:488 +0x95
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).readRecord(0xc422c9aa80, 0x1433617, 0xc422c9aba0, 0x413576)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
Mar 12 22:18:22 gareth platform[668]: crypto/tls.(*Conn).Read(0xc422c9aa80, 0xc422a37000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).Read(0xc4200765a0, 0xc422a37000, 0x1000, 0x1000, 0xc42185d380, 0xc4213994a8, 0xc422d0eb68)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1391 +0x140
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).fill(0xc4216816e0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:97 +0x11a
Mar 12 22:18:22 gareth platform[668]: bufio.(*Reader).Peek(0xc4216816e0, 0x1, 0xc4222f0420, 0xc421aa2c80, 0x412960, 0x7fca98deb000, 0x6)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/bufio/bufio.go:129 +0x3a
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).readLoop(0xc4200765a0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1539 +0x185
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1186 +0xa2e
Mar 12 22:18:22 gareth platform[668]: goroutine 2818 [select, 1 minutes]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc421deb0e0)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53
Mar 12 22:18:22 gareth platform[668]: goroutine 2967 [select]:
Mar 12 22:18:22 gareth platform[668]: net/http.(*persistConn).writeLoop(0xc422cc0d80)
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1759 +0x165
Mar 12 22:18:22 gareth platform[668]: created by net/http.(*Transport).dialConn
Mar 12 22:18:22 gareth platform[668]:         /usr/local/go/src/net/http/transport.go:1187 +0xa53

Possible fixes

None known. I tried uploading the image several times. Each time the error happened.

[lindy65]

Hi @wezm,

Could you share which server version of Mattermost you have installed and which platform this is happening on (Win / iOS / Desktop app)?

[wezm]

Sure @lindy65,

• The image was being uploaded from the Mattermost desktop app on Linux, version 4.0.0
• Server version:
  Version: 4.7.0 Build Number: 4.7.0 Build Date: Thu Feb 15 22:03:15 UTC 2018 Build Hash: 9360131992064203f9ee46fa0504d20276c97339 Build Enterprise Ready: true DB Version: 4.7.0
• Running on Debian Linux 9.4

[lindy65]

Thanks @wezm,

I'll pass this info on to our engineers and ask for their help troubleshooting your issue 👍

[wezm]

Just adding some more details this. Another person on my Mattermost instance encountered the same issue uploading a file. They were using the web UI. I then got a copy of the image and tried uploading it from my phone (iOS) and it also failed. This time the Linux OOM killer killed it instead though.

Killed process 1880 (platform) total-vm:1597028kB, anon-rss:908932kB, file-rss:0kB, shmem-rss:0kB

The image was not large. I was able to upload an image earlier in the day successfully, so I don't think it's a misconfiguration.

Update: My server didn't have any swap. Adding some seems to have helped. In normal operation there is 800Mb of RAM free so not sure why these little images are causing Mattermost to consume all that RAM but adding swap seems to have helped.

[lindy65]

Thanks @wezm,

Hope to have some feedback for you today 👍

[lieut-data]

@wezm, what are the specs for the machine where you're running Mattermost? It's possible that, prior to the swap changes, you were just at the limit past which the increased memory required on image upload triggered the observed behaviour. I'd definitely recommend increasing the headroom here.

There's probably also some room for us to optimize the image upload process here.

[wezm]

@lieut-data It's a DigitalOcean droplet with 1Gb RAM and one CPU core. The only thing it runs is the Mattermost server (the webserver and database are on a different machine). In normal operation there are 100's of Mb of RAM free (at the time of writing 94Mb used, 753Mb free as reported by free -h). It doesn't seem right to me that a 38.4kB PNG file would cause the platform process to spike RAM so much as to run it out of RAM.

Here's 7 day graphs of RAM and CPU use on the machine:

[lieut-data]

Thanks for the specs, @wezm. I've setup a machine running the latest Debian Jesse supported by Mattermost with only 512Mb RAM and one CPU core. It's running 4.7.3, with MySQL also hosted on a different machine. So far, I'm unable to reproduce the behaviour you describe with the attached image.

That being said, I note that the memory characteristics of this simple setup are very different than yours, despite only Mattermost running:

> free -h
             total       used       free     shared    buffers     cached
Mem:          494M       486M       7.7M       4.6M        11M       358M
-/+ buffers/cache:       116M       378M
Swap:         461M         0B       461M

Which Linux distribution are you running? Is there anything else you can share about this setup that might help reproduce the problem?

[wezm]

It's Debian 9.4 (stretch). My instance is using PostgreSQL for the DB and S3 for image storage. I haven't customised it at all. I basically just started it in the DigitalOcean control panel and then installed Mattermost on it with Ansible.

# free -h
              total        used        free      shared  buff/cache   available
Mem:           996M         94M        751M         11M        150M        760M
Swap:          2.0G         48M        2.0G

# cat /etc/debian_version 
9.4

# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

I'm surprised you're seeing 480M used on a basically empty machine. Perhaps the way free reports things changed between Debian 8 and 9. I can see in your one that you have 11M + 358M being used for buffer and cache, this is reclaimable if some other process needed it. The issue originally occurred for me when I had no swap. Perhaps it's worth trying running without swap too, to see if that helps reproduce the issue. If it does, then the end result might just be guidance in the install instructions to ensure the host has swap available.

This is the list of processes on my server excluding kernel threads:

# ps -aux | grep -vF '['
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.4  56968  4752 ?        Ss   Mar12   0:02 /sbin/init
root       449  0.0  0.3  56848  3764 ?        Ss   Mar12   0:28 /lib/systemd/systemd-journald
root       470  0.0  0.0  45380   520 ?        Ss   Mar12   0:00 /lib/systemd/systemd-udevd
root       665  0.0  0.0 250116   680 ?        Ssl  Mar12   0:03 /usr/sbin/rsyslogd -n
root       667  0.0  0.1  38072  2012 ?        Ss   Mar12   0:00 /lib/systemd/systemd-logind
nobody     675  0.0  0.9 182860  9512 ?        Ssl  Mar12   0:18 /opt/digitalocean/bin/do-agent -log_syslog
unscd      677  0.0  0.0  14756   852 ?        Ss   Mar12   0:02 /usr/sbin/nscd -d
root       678  0.0  0.0  21160  1008 ?        Ss   Mar12   0:00 /usr/sbin/cron -f
message+   682  0.0  0.0  36552   948 ?        Ss   Mar12   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       715  0.0  0.1  63648  1784 ?        Ss   Mar12   0:03 /usr/sbin/sshd -D
root       716  0.0  0.0   4452     0 tty1     Ss+  Mar12   0:00 /sbin/agetty --noclear tty1 linux
root       720  0.0  0.0   5836     0 ttyS0    Ss+  Mar12   0:00 /sbin/agetty --keep-baud 115200,38400,9600 ttyS0 vt220
ntp        722  0.0  0.1  95760  1368 ?        Ssl  Mar12   0:09 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:109
matterm+ 14845  0.0  6.1 2205344 63164 ?       Ssl  Mar14   0:53 /opt/mattermost/bin/platform
root     25201  0.0  0.6  95168  6840 ?        Ss   00:07   0:00 sshd: root@pts/0
root     25204  0.0  0.5  56436  5384 ?        Ss   00:07   0:00 /lib/systemd/systemd --user
root     25205  0.0  0.1  82492  1156 ?        S    00:07   0:00 (sd-pam)
root     25212  0.0  0.4  12492  4548 pts/0    Ss   00:07   0:00 -bash
root     25328  0.0  0.1  29800  1572 pts/0    R+   00:14   0:00 ps -aux

Here I can see the mattermost process using about 60M of RAM.

[lieut-data]

I've setup a Debian 9.4 instance, disabled swap and uploaded a bunch of images of various sizes but have yet to see the memory spike as my own theory suggested. No issues uploading images yet.

vagrant@mattermost:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           492M        124M         17M        2.8M        350M        352M
Swap:            0B          0B          0B

vagrant@mattermost:~$ cat /etc/debian_version
9.4

vagrant@mattermost:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

vagrant@mattermost:~$ ps -aux | grep -vF '['
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  1.3 138984  6772 ?        Ss   Mar15   0:01 /sbin/init
root       176  0.0  0.9  47364  4620 ?        Ss   Mar15   0:00 /lib/systemd/systemd-journald
root       198  0.0  0.8  45972  4092 ?        Ss   Mar15   0:00 /lib/systemd/systemd-udevd
systemd+   308  0.0  0.8 127284  4156 ?        Ssl  Mar15   0:00 /lib/systemd/systemd-timesyncd
message+   317  0.0  0.7  45132  3700 ?        Ss   Mar15   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root       329  0.0  0.5  29636  2752 ?        Ss   Mar15   0:00 /usr/sbin/cron -f
root       330  0.0  0.9  46496  4728 ?        Ss   Mar15   0:00 /lib/systemd/systemd-logind
root       331  0.0  0.5 250116  3020 ?        Ssl  Mar15   0:00 /usr/sbin/rsyslogd -n
root       349  0.0  0.3  14536  1644 tty1     Ss+  Mar15   0:00 /sbin/agetty --noclear tty1 linux
root       352  0.0  1.0  69944  5456 ?        Ss   Mar15   0:00 /usr/sbin/sshd -D
root       464  0.0  0.5  20468  2944 ?        Ss   Mar15   0:00 /sbin/dhclient -4 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -I -df /var/lib/dhcp/dhclient6.eth0.leases eth0
matterm+  1421  0.0 23.7 520900 119820 ?       Ssl  Mar15   0:27 /opt/mattermost/bin/platform
matterm+  1430  0.0  1.5  40784  7980 ?        Sl   Mar15   0:00 plugins/jira/plugin.exe
vagrant   1454  0.0  1.2  64868  6264 ?        Ss   Mar15   0:00 /lib/systemd/systemd --user
vagrant   1455  0.0  0.3 164508  1708 ?        S    Mar15   0:00 (sd-pam)
vagrant   1461  0.0  1.0  95472  5084 ?        S    Mar15   0:00 sshd: vagrant@pts/0
vagrant   1462  0.0  0.9  20940  4884 pts/0    Ss   Mar15   0:00 -bash
vagrant   1782  0.0  0.6  38304  3160 pts/0    R+   14:29   0:00 ps -aux

Let us know how running with swap enabled goes and if you see any other oddities. We do generally suggest (or at least focus on) 2 vCPUs/cores, 4 GB RAM, and 45-90 GB storage aiming at 250-500 users, but nothing fundamentally ought to preclude running smaller teams on smaller boxes.

[wezm]

Ok, thanks for investigating. Since adding swap we've been able to upload images without any issues so far. There are only 15 people on my instance; as shown in the graphs above the machine is idle most of the time.

[lindy65]

Great to hear @wezm :)

I'll close this issue off here for now but if you have any issues in the future, please feel free to let us know!

[coderafei]

I have same issue when upload image, the server will crash
but i upgrade my ram from 1G to 2G, the problem disappear

[gyp5y]

We have the same issue on 8Gb server.

[lindy65]

Hi @gyp5y,

Thanks for your feedback,

Could you share a little more about your set-up as it will be easier to help troubleshoot. e.g. log errors, machine specs, etc?

[gyp5y]

There's nothing special in logs. It just restarts when someone tries to send multiple images.
Linux 4.13.13-6-pve, 8Gb ram, S3 backend for files, Elasticsearch

[lieut-data]

@gyp5y, would you be able to search the system logs for messages like @wezm, e.g.:

Mar 12 22:18:22 gareth platform[668]: fatal error: runtime: out of memory

There should be a trace, somewhere, explaining what's going on.

[gyp5y]

There's nothing similar. I've looked for it in system and mattermost's logs.
Ony strange thing in log I got before it restarts:
[2018/04/13 07:09:34 CDT] [EROR] /api/v4/channels/undefined/posts:Context code=400 rid=r6aaw5w8qpb75nptodb8mi1p8o uid=zyhqhjxd83datfgruxufop8q4o ip=xxx.xxx.xxx.xxx Invalid or missing channel_id parameter in request URL [details: ]
[2018/04/13 07:10:58 CDT] [EROR] /api/v4/files/undefined:Context code=400 rid=6isht7jtrifo7p76fmjbbn8pgo uid=c1sx3zdzp7nu8maxm7u5n1uego ip=xxx.xxx.xxx.xxx Invalid or missing file_id parameter in request URL [details: ]

[lieut-data]

@gyp5y, interesting -- thank you.

Can you elaborate on what you observe when you say it crashes? Is it that the binary stops running, and you have to start it manually? Or does your setup automatically restart the Mattermost platform service? Do you have logs showing the Mattermost server stopping and starting -- how often does it occur?

[gyp5y]

Our setup restarts it automatically. I didn't found any stack traces or dumps. After those error messages it just begins to start.

[lieut-data]

@gyp5y, do you have debug logs enabled? If not, can you enable them, and post everything between the last log that looks something like:

[2018/04/16 11:08:18 EDT] [DEBG] POST - /api/v4/files

and the Mattermost server restarting:

[2018/04/16 10:16:33 EDT] [INFO] Server is initializing...

Also, does this only happen when uploading multiple images? Does it happen for any two images, regardless of file size?

[gyp5y]

Here is most relevant part of the debug.log
[2018/04/17 02:07:53 CDT] [DEBG] websocket.read: closing websocket for userId=qmx1hx9au3fpux7k393to45bsh error=websocket: close 1001 (going away) [2018/04/17 02:07:53 CDT] [DEBG] websocket: user_typing [2018/04/17 02:07:53 CDT] [DEBG] POST - /api/v4/channels/members/me/view [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/websocket [2018/04/17 02:07:53 CDT] [DEBG] POST - /api/v4/users/status/ids [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/channels/gnbwd9cabt8um8q96mnxigk8qw/stats [2018/04/17 02:07:53 CDT] [DEBG] POST - /api/v4/users/ids [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/websocket [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/plugins/webapp [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/plugins/webapp [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/users/me/teams/krq1en963jfo3jk8sfz3bakxdh/channels [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/users/me/teams/krq1en963jfo3jk8sfz3bakxdh/channels/members [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/channels/m9wiku35ifgzujt7naxfgh7j3w/posts [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/users/me/teams/krq1en963jfo3jk8sfz3bakxdh/channels [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/users/me/teams/krq1en963jfo3jk8sfz3bakxdh/channels/members [2018/04/17 02:07:53 CDT] [DEBG] GET - /api/v4/channels/w4rm1zmukin4pj11s6ord6d3ur/posts [2018/04/17 02:07:54 CDT] [DEBG] POST - /api/v4/users/status/ids [2018/04/17 02:07:54 CDT] [DEBG] GET - /api/v4/users/me/teams/unread [2018/04/17 02:07:54 CDT] [DEBG] POST - /api/v4/users/status/ids [2018/04/17 02:07:54 CDT] [DEBG] GET - /api/v4/users/me/teams/unread [2018/04/17 02:07:54 CDT] [DEBG] GET - /api/v4/plugins/webapp [2018/04/17 02:07:54 CDT] [DEBG] POST - /api/v4/users/status/ids [2018/04/17 02:07:54 CDT] [DEBG] GET - /api/v4/users/me/teams/krq1en963jfo3jk8sfz3bakxdh/channels [2018/04/17 02:07:54 CDT] [DEBG] GET - /api/v4/users/me/teams/krq1en963jfo3jk8sfz3bakxdh/channels/members [2018/04/17 02:07:54 CDT] [DEBG] GET - /api/v4/channels/ursraa1z63ncuyb1gcned494ah/posts [2018/04/17 02:07:54 CDT] [DEBG] POST - /api/v4/users/status/ids [2018/04/17 02:07:54 CDT] [DEBG] GET - /api/v4/users/me/teams/unread [2018/04/17 02:07:54 CDT] [DEBG] POST - /api/v4/users/status/ids [2018/04/17 02:08:24 CDT] [INFO] Loaded system translations for 'en' from '/opt/mattermost/i18n/en.json' [2018/04/17 02:08:24 CDT] [INFO] Server is initializing...

I can't reproduce it for one or two images. It repeats if uploading 5 images (from first or second try) and it restarts before "POST" button is pressed.

[gyp5y]

I can confirm that on 4.9.0 this issue still exists.

[alecisec]

I'm also having this issue from 4.7.0 all the way up to 4.9.2. I'm running mattermost inside a docker container on AWS.

These are my logs

`
mattermost | [2018/05/07 01:46:51 UTC] [DEBG] POST - /api/v4/channels/members/me/view

mattermost | fatal error: runtime: out of memory

mattermost |

mattermost | runtime stack:

mattermost | runtime.throw(0x140f883, 0x16)

mattermost | /usr/local/go/src/runtime/panic.go:605 +0x95
mattermost | runtime.sysMap(0xc4902c0000, 0x24000000, 0x0, 0x24bbfb8)
mattermost | /usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
mattermost | runtime.(mheap).sysAlloc(0x24a2600, 0x24000000, 0x4)
mattermost | /usr/local/go/src/runtime/malloc.go:470 +0xd7
mattermost | runtime.(mheap).grow(0x24a2600, 0x12000, 0x0)
mattermost | /usr/local/go/src/runtime/mheap.go:887 +0x60
mattermost | runtime.(mheap).allocSpanLocked(0x24a2600, 0x12000, 0x24bbfc8, 0x7fcac8745e58)
mattermost | /usr/local/go/src/runtime/mheap.go:800 +0x334
mattermost | runtime.(mheap).alloc_m(0x24a2600, 0x12000, 0xc4232f0101, 0x7fcac5686e78)
mattermost | /usr/local/go/src/runtime/mheap.go:666 +0x118
mattermost | runtime.(mheap).alloc.func1()
mattermost | /usr/local/go/src/runtime/mheap.go:733 +0x4d
mattermost | runtime.systemstack(0x7fcac5686e10)
mattermost | /usr/local/go/src/runtime/asm_amd64.s:360 +0xab
mattermost | runtime.(mheap).alloc(0x24a2600, 0x12000, 0xc421010101, 0xc421403501)
mattermost | /usr/local/go/src/runtime/mheap.go:732 +0xa1
mattermost | runtime.largeAlloc(0x24000000, 0xc421550101, 0xc4217b1700)
mattermost | /usr/local/go/src/runtime/malloc.go:827 +0x98
mattermost | runtime.mallocgc.func1()
mattermost | /usr/local/go/src/runtime/malloc.go:722 +0x46
mattermost | runtime.systemstack(0xc420022000)
mattermost | /usr/local/go/src/runtime/asm_amd64.s:344 +0x79
mattermost | runtime.mstart()
mattermost | /usr/local/go/src/runtime/proc.go:1135
mattermost |
mattermost | goroutine 2543 [running]:
mattermost | runtime.systemstack_switch()
mattermost | /usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc42343e860 sp=0xc42343e858 pc=0x45be30
mattermost | runtime.mallocgc(0x24000000, 0x11e8ba0, 0xc42343ed01, 0x98)
mattermost | /usr/local/go/src/runtime/malloc.go:721 +0x7ae fp=0xc42343e908 sp=0xc42343e860 pc=0x412a1e
mattermost | runtime.makeslice(0x11e8ba0, 0x24000000, 0x24000000, 0xc42343ed98, 0xc42343eb78, 0xc42001c060)
mattermost | /usr/local/go/src/runtime/slice.go:54 +0x77 fp=0xc42343e938 sp=0xc42343e908 pc=0x4455f7
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectMultipartStreamNoLength(0xc42177cf80, 0xc421d64a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4220235c0, 0xc421ada5a0, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:202 +0x38a fp=0xc42343efb8 sp=0xc42343e938 pc=0x96b58a
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectCommon(0xc42177cf80, 0xc421d64a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4220235c0, 0xc421ada5a0, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:152 +0x642 fp=0xc42343f240 sp=0xc42343efb8 pc=0x96afd2
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObjectWithContext(0xc42177cf80, 0xc421d64a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4220235c0, 0xc421ada5a0, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-context.go:38 +0x302 fp=0xc42343f3c8 sp=0xc42343f240 pc=0x9620b2
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObject(0xc42177cf80, 0xc421d64a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc4220235c0, 0xc421ada5a0, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:130 +0x114 fp=0xc42343f550 sp=0xc42343f3c8 pc=0x96a944
mattermost | github.com/mattermost/mattermost-server/utils.(S3FileBackend).WriteFile(0xc421d64a20, 0xc423f18000, 0x150d65, 0x1ffe00, 0xc421f6f800, 0x7b, 0x81)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/file_backend_s3.go:153 +0x2a4 fp=0xc42343f910 sp=0xc42343f550 pc=0x9bce34
mattermost | github.com/mattermost/mattermost-server/app.(App).WriteFile(0xc420087d40, 0xc423f18000, 0x150d65, 0x1ffe00, 0xc421f6f800, 0x7b, 0xa)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:86 +0x8e fp=0xc42343f958 sp=0xc42343f910 pc=0xcbeb4e
mattermost | github.com/mattermost/mattermost-server/app.(App).DoUploadFile(0xc420087d40, 0xbeb409e419efc400, 0xc1d767c9c, 0x249a060, 0x13ebcc3, 0x6, 0xc421884f00, 0x1a, 0xc42291edc0, 0x1a, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:382 +0x690 fp=0xc42343fb10 sp=0xc42343f958 pc=0xcc3a00
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc421884f00, 0x1a, 0xc42291edc0, 0x1a, 0xc42174f850, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:329 +0x505 fp=0xc423440840 sp=0xc42343fb10 pc=0xcc29d5
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadMultipartFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc421884f00, 0x1a, 0xc42291edc0, 0x1a, 0xc422324060, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:300 +0x41b fp=0xc423440940 sp=0xc423440840 pc=0xcc247b
mattermost | github.com/mattermost/mattermost-server/api4.uploadFile(0xc422b57b30, 0x1d6e340, 0xc422b416c0, 0xc423402f00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/file.go:120 +0x562 fp=0xc423442d58 sp=0xc423440940 pc=0xd8fde2
mattermost | github.com/mattermost/mattermost-server/api4.handler.ServeHTTP(0xc420087d40, 0x148a640, 0x10001, 0x1d6e340, 0xc422b416c0, 0xc423402f00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/context.go:157 +0xb65 fp=0xc423443a60 sp=0xc423442d58 pc=0xd87ec5
mattermost | github.com/mattermost/mattermost-server/api4.(handler).ServeHTTP(0xc4227a4600, 0x1d6e340, 0xc422b416c0, 0xc423402f00)
mattermost | :1 +0x63 fp=0xc423443aa0 sp=0xc423443a60 pc=0xdcc8c3
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux.(Router).ServeHTTP(0xc4206800e0, 0x1d6e340, 0xc422b416c0, 0xc423402f00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux/mux.go:159 +0xed fp=0xc423443ba0 sp=0xc423443aa0 pc=0xa39cfd
mattermost | github.com/mattermost/mattermost-server/app.(CorsWrapper).ServeHTTP(0xc4217206f0, 0x1d6e340, 0xc422b416c0, 0xc423402d00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:83 +0xa2 fp=0xc423443bf0 sp=0xc423443ba0 pc=0xd0e6b2
mattermost | github.com/mattermost/mattermost-server/app.(RateLimiter).RateLimitHandler.func1(0x1d6e340, 0xc422b416c0, 0xc423402d00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/ratelimit.go:106 +0xca fp=0xc423443c48 sp=0xc423443bf0 pc=0xd54cba
mattermost | net/http.HandlerFunc.ServeHTTP(0xc421f82220, 0x1d6e340, 0xc422b416c0, 0xc423402d00)
mattermost | /usr/local/go/src/net/http/server.go:1918 +0x44 fp=0xc423443c70 sp=0xc423443c48 pc=0x6f71f4
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP(0x1d67540, 0xc421f82220, 0x1d631c0, 0x24ba100, 0x1, 0x1d6e340, 0xc422b416c0, 0xc423402d00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers/recovery.go:78 +0xb9 fp=0xc423443cc8 sp=0xc423443c70 pc=0xc20379
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.(recoveryHandler).ServeHTTP(0xc421ff97a0, 0x1d6e340, 0xc422b416c0, 0xc423402d00)
mattermost | :1 +0x76 fp=0xc423443d18 sp=0xc423443cc8 pc=0xc20b26
mattermost | net/http.serverHandler.ServeHTTP(0xc4212ba4e0, 0x1d6e340, 0xc422b416c0, 0xc423402d00)
mattermost | /usr/local/go/src/net/http/server.go:2619 +0xb4 fp=0xc423443d48 sp=0xc423443d18 pc=0x6f9fe4
mattermost | net/http.(conn).serve(0xc4228d0320, 0x1d6f0c0, 0xc4228259c0)
mattermost | /usr/local/go/src/net/http/server.go:1801 +0x71d fp=0xc423443fc8 sp=0xc423443d48 pc=0x6f60cd
mattermost | runtime.goexit()
mattermost | /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc423443fd0 sp=0xc423443fc8 pc=0x45ea61
mattermost | created by net/http.(Server).Serve
mattermost | /usr/local/go/src/net/http/server.go:2720 +0x288
mattermost |
mattermost | goroutine 1 [chan receive]:
mattermost | github.com/mattermost/mattermost-server/cmd/commands.runServer(0x13f6bb4, 0xb, 0xc420056e00, 0xc420056e40, 0x0, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/cmd/commands/server.go:191 +0xe68
mattermost | github.com/mattermost/mattermost-server/cmd/commands.serverCmdF(0x24800c0, 0x24ba100, 0x0, 0x0, 0x0, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/cmd/commands/server.go:53 +0x10d
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).execute(0x24800c0, 0xc420010180, 0x0, 0x0, 0x24800c0, 0xc420010180)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:753 +0x475
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).ExecuteC(0x24800c0, 0x24ba100, 0xc420109f40, 0x10f84cc)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:843 +0x334
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).Execute(0x24800c0, 0x0, 0xc420109f40)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:791 +0x2b
mattermost | github.com/mattermost/mattermost-server/cmd.Run(0xc420010180, 0x0, 0x0, 0xc420109f70, 0x11095a9)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/cmd/cmd.go:14 +0x6d
mattermost | main.main()
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/main.go:33 +0x63
mattermost |
mattermost | goroutine 2260 [chan receive]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.(ConsoleLogWriter).run(0xc422365500, 0x1d653c0, 0xc42000e018)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/termlog.go:33 +0x116
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.NewConsoleLogWriter
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/termlog.go:26 +0xcf
mattermost |
mattermost | goroutine 8 [syscall]:
mattermost | os/signal.signal_recv(0x0)
mattermost | /usr/local/go/src/runtime/sigqueue.go:131 +0xa6
mattermost | os/signal.loop()
mattermost | /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
mattermost | created by os/signal.init.0
mattermost | /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
mattermost |
mattermost | goroutine 2321 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc422355b00, 0xc422302540, 0xc4223f66c0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2466 [select]:
mattermost | github.com/mattermost/mattermost-server/jobs.(Watcher).Start(0xc421acc6c0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/jobs_watcher.go:51 +0x294
mattermost | created by github.com/mattermost/mattermost-server/jobs.(Workers).Start.func1
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/workers.go:80 +0x88
mattermost |
mattermost | goroutine 2468 [select, locked to thread]:
mattermost | runtime.gopark(0x148cdb0, 0x0, 0x13ec209, 0x6, 0x18, 0x1)
mattermost | /usr/local/go/src/runtime/proc.go:287 +0x12c
mattermost | runtime.selectgo(0xc4201d0750, 0xc42174c840)
mattermost | /usr/local/go/src/runtime/select.go:395 +0x1149
mattermost | runtime.ensureSigM.func1()
mattermost | /usr/local/go/src/runtime/signal_unix.go:511 +0x220
mattermost | runtime.goexit()
mattermost | /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1
mattermost |
mattermost | goroutine 2470 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go.(Client).loop(0xc4222fda00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go/analytics.go:338 +0x1b5
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go.(Client).startLoop
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/segmentio/analytics-go/analytics.go:232 +0x3f
mattermost |
mattermost | goroutine 2504 [select]:
mattermost | github.com/mattermost/mattermost-server/model.createTask.func1(0xc421ecda80, 0x4e94914f0000, 0xc4213c99e0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:49 +0x15a
mattermost | created by github.com/mattermost/mattermost-server/model.createTask
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:40 +0x10c
mattermost |
mattermost | goroutine 2849 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac8755970, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc422006f98, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc422006f98, 0xc422bb1d00, 0x1, 0x1)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc422006f80, 0xc422bb1d51, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc422006f80, 0xc422bb1d51, 0x1, 0x1, 0xc4200572c0, 0x0, 0x1d64601)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422b6cc30, 0xc422bb1d51, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | net/http.(connReader).backgroundRead(0xc422bb1d40)
mattermost | /usr/local/go/src/net/http/server.go:660 +0x62
mattermost | created by net/http.(connReader).startBackgroundRead
mattermost | /usr/local/go/src/net/http/server.go:656 +0xd8
mattermost |
mattermost | goroutine 2523 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac8755730, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc4217b1d98, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc4217b1d98, 0xc421166000, 0x1000, 0x1000)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc4217b1d80, 0xc421166000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc4217b1d80, 0xc421166000, 0x1000, 0x1000, 0xc4222afc50, 0x6f063a, 0xc4213ca1e8)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422325e50, 0xc421166000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | net/http.(connReader).Read(0xc4213ca1e0, 0xc421166000, 0x1000, 0x1000, 0xc4222afd38, 0x52410c, 0xc4217b1d80)
mattermost | /usr/local/go/src/net/http/server.go:753 +0x105
mattermost | bufio.(Reader).fill(0xc421217440)
mattermost | /usr/local/go/src/bufio/bufio.go:97 +0x11a
mattermost | bufio.(Reader).Peek(0xc421217440, 0x4, 0x4e16e1a235, 0x249a060, 0x0, 0x0, 0x249a060)
mattermost | /usr/local/go/src/bufio/bufio.go:129 +0x3a
mattermost | net/http.(conn).serve(0xc4228015e0, 0x1d6f0c0, 0xc421852bc0)
mattermost | /usr/local/go/src/net/http/server.go:1826 +0x88f
mattermost | created by net/http.(Server).Serve
mattermost | /usr/local/go/src/net/http/server.go:2720 +0x288
mattermost |
mattermost | goroutine 2853 [sleep]:
mattermost | time.Sleep(0x2a780209)
mattermost | /usr/local/go/src/runtime/time.go:65 +0x130
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc420050a80, 0xa, 0xc420050a20, 0xc4220224b0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x168
mattermost |
mattermost | goroutine 2322 [select]:
mattermost | database/sql.(DB).connectionCleaner(0xc420083360, 0x34630b8a000)
mattermost | /usr/local/go/src/database/sql/sql.go:793 +0x446
mattermost | created by database/sql.(DB).startCleanerLocked
mattermost | /usr/local/go/src/database/sql/sql.go:780 +0xba
mattermost |
mattermost | goroutine 2464 [sleep]:
mattermost | time.Sleep(0x34630b8a000)
mattermost | /usr/local/go/src/runtime/time.go:65 +0x130
mattermost | github.com/mattermost/mattermost-server/enterprise/compliance.(ComplianceInterfaceImpl).StartComplianceDailyJob.func1(0xc422324900)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/enterprise/compliance/compliance.go:276 +0x97
mattermost | created by github.com/mattermost/mattermost-server/enterprise/compliance.(ComplianceInterfaceImpl).StartComplianceDailyJob
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/enterprise/compliance/compliance.go:221 +0x3f
mattermost |
mattermost | goroutine 2480 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc4212ecc60, 0xc422302480, 0xc42174d620)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2486 [select]:
mattermost | github.com/mattermost/mattermost-server/model.createTask.func1(0xc421edfa80, 0x34630b8a000, 0xc4213c8d50)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:49 +0x15a
mattermost | created by github.com/mattermost/mattermost-server/model.createTask
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:40 +0x10c
mattermost |
mattermost | goroutine 2485 [select]:
mattermost | github.com/mattermost/mattermost-server/model.createTask.func1(0xc421edf480, 0x34630b8a000, 0xc4213c8d30)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:49 +0x15a
mattermost | created by github.com/mattermost/mattermost-server/model.createTask
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:40 +0x10c
mattermost |
mattermost | goroutine 2843 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac8755670, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc422007b98, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc422007b98, 0xc4220ac800, 0x1, 0x1)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc422007b80, 0xc4220ac8b1, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc422007b80, 0xc4220ac8b1, 0x1, 0x1, 0x2, 0x2, 0x20002)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422b6cc98, 0xc4220ac8b1, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | net/http.(connReader).backgroundRead(0xc4220ac8a0)
mattermost | /usr/local/go/src/net/http/server.go:660 +0x62
mattermost | created by net/http.(connReader).startBackgroundRead
mattermost | /usr/local/go/src/net/http/server.go:656 +0xd8
mattermost |
mattermost | goroutine 2505 [select]:
mattermost | github.com/mattermost/mattermost-server/model.createTask.func1(0xc421d225c0, 0x4e94914f0000, 0xc4213c9a10)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:49 +0x15a
mattermost | created by github.com/mattermost/mattermost-server/model.createTask
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:40 +0x10c
mattermost |
mattermost | goroutine 2867 [chan receive]:
mattermost | net/http.(Transport).dialConn(0x247ed20, 0x1d6f100, 0xc42001c060, 0x0, 0xc421d1fba0, 0x5, 0xc422b66960, 0x45, 0x6, 0x0, ...)
mattermost | /usr/local/go/src/net/http/transport.go:1157 +0xdf4
mattermost | net/http.(Transport).getConn.func4(0x247ed20, 0x1d6f100, 0xc42001c060, 0xc4211cea20, 0xc4212f0ba0)
mattermost | /usr/local/go/src/net/http/transport.go:943 +0x78
mattermost | created by net/http.(Transport).getConn
mattermost | /usr/local/go/src/net/http/transport.go:942 +0x393
mattermost |
mattermost | goroutine 2262 [syscall]:
mattermost | syscall.Syscall6(0xe8, 0x6, 0xc422459c24, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
mattermost | github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix.EpollWait(0x6, 0xc422459c24, 0x7, 0x7, 0xffffffffffffffff, 0xc42155b080, 0xc4223fb400, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1544 +0x77
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(fdPoller).wait(0xc422362c60, 0x148ac00, 0xc422362c60, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x91
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(Watcher).readEvents(0xc421931e50)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x194
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.NewWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x23a
mattermost |
mattermost | goroutine 2577 [select]:
mattermost | github.com/mattermost/mattermost-server/app.(WebConn).writePump(0xc42120fef0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:164 +0x243
mattermost | github.com/mattermost/mattermost-server/app.(WebConn).Pump.func1(0xc42120fef0, 0xc421e16660)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:110 +0x2b
mattermost | created by github.com/mattermost/mattermost-server/app.(WebConn).Pump
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:109 +0x6b
mattermost |
mattermost | goroutine 2551 [select]:
mattermost | net/http.(persistConn).roundTrip(0xc4234077a0, 0xc422022660, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/transport.go:1970 +0x608
mattermost | net/http.(Transport).RoundTrip(0x247ed20, 0xc4222fc000, 0x247ed20, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/transport.go:413 +0x992
mattermost | net/http.send(0xc4222fc000, 0x1d64cc0, 0x247ed20, 0x0, 0x0, 0x0, 0xc422324080, 0x1d6b400, 0xc4233cbea8, 0x1)
mattermost | /usr/local/go/src/net/http/client.go:249 +0x1a9
mattermost | net/http.(Client).send(0xc422031ef0, 0xc4222fc000, 0x0, 0x0, 0x0, 0xc422324080, 0x0, 0x1, 0x0)
mattermost | /usr/local/go/src/net/http/client.go:173 +0xfd
mattermost | net/http.(Client).Do(0xc422031ef0, 0xc4222fc000, 0x20, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/client.go:602 +0x28d
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.do(0xc42177c180, 0xc421d64240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422031ef0, 0xc421ada260, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:496 +0x50
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.executeMethod(0xc42177c180, 0xc421d64240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422031ef0, 0xc421ada260, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:601 +0x382
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.uploadPart(0xc42177c180, 0xc421d64240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422031ef0, 0xc421ada260, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-multipart.go:283 +0x494
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectMultipartStreamNoLength(0xc42177c180, 0xc421d64240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422031ef0, 0xc421ada260, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:219 +0x71e
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectCommon(0xc42177c180, 0xc421d64240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422031ef0, 0xc421ada260, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:152 +0x642
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObjectWithContext(0xc42177c180, 0xc421d64240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422031ef0, 0xc421ada260, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-context.go:38 +0x302
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObject(0xc42177c180, 0xc421d64240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422031ef0, 0xc421ada260, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:130 +0x114
mattermost | github.com/mattermost/mattermost-server/utils.(S3FileBackend).WriteFile(0xc421d641e0, 0xc422d10000, 0x1436e2, 0x1ffe00, 0xc421f6e100, 0x7b, 0x81)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/file_backend_s3.go:153 +0x2a4
mattermost | github.com/mattermost/mattermost-server/app.(App).WriteFile(0xc420087d40, 0xc422d10000, 0x1436e2, 0x1ffe00, 0xc421f6e100, 0x7b, 0xa)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:86 +0x8e
mattermost | github.com/mattermost/mattermost-server/app.(App).DoUploadFile(0xc420087d40, 0xbeb409e3cf4a26c9, 0xbd7361577, 0x249a060, 0x13ebcc3, 0x6, 0xc421884460, 0x1a, 0xc42291edc0, 0x1a, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:382 +0x690
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc421884460, 0x1a, 0xc42291edc0, 0x1a, 0xc42174e580, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:329 +0x505
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadMultipartFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc421884460, 0x1a, 0xc42291edc0, 0x1a, 0xc421d1aed0, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:300 +0x41b
mattermost | github.com/mattermost/mattermost-server/api4.uploadFile(0xc422b57950, 0x1d6e340, 0xc422b41500, 0xc423402900)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/file.go:120 +0x562
mattermost | github.com/mattermost/mattermost-server/api4.handler.ServeHTTP(0xc420087d40, 0x148a640, 0x10001, 0x1d6e340, 0xc422b41500, 0xc423402900)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/context.go:157 +0xb65
mattermost | github.com/mattermost/mattermost-server/api4.(handler).ServeHTTP(0xc4227a4600, 0x1d6e340, 0xc422b41500, 0xc423402900)
mattermost | :1 +0x63
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux.(Router).ServeHTTP(0xc4206800e0, 0x1d6e340, 0xc422b41500, 0xc423402900)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux/mux.go:159 +0xed
mattermost | github.com/mattermost/mattermost-server/app.(CorsWrapper).ServeHTTP(0xc4217206f0, 0x1d6e340, 0xc422b41500, 0xc423402700)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:83 +0xa2
mattermost | github.com/mattermost/mattermost-server/app.(RateLimiter).RateLimitHandler.func1(0x1d6e340, 0xc422b41500, 0xc423402700)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/ratelimit.go:106 +0xca
mattermost | net/http.HandlerFunc.ServeHTTP(0xc421f82220, 0x1d6e340, 0xc422b41500, 0xc423402700)
mattermost | /usr/local/go/src/net/http/server.go:1918 +0x44
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP(0x1d67540, 0xc421f82220, 0x1d631c0, 0x24ba100, 0x1, 0x1d6e340, 0xc422b41500, 0xc423402700)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers/recovery.go:78 +0xb9
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.(recoveryHandler).ServeHTTP(0xc421ff97a0, 0x1d6e340, 0xc422b41500, 0xc423402700)
mattermost | :1 +0x76
mattermost | net/http.serverHandler.ServeHTTP(0xc4212ba4e0, 0x1d6e340, 0xc422b41500, 0xc423402700)
mattermost | /usr/local/go/src/net/http/server.go:2619 +0xb4
mattermost | net/http.(conn).serve(0xc4228d14a0, 0x1d6f0c0, 0xc4225b2200)
mattermost | /usr/local/go/src/net/http/server.go:1801 +0x71d
mattermost | created by net/http.(Server).Serve
mattermost | /usr/local/go/src/net/http/server.go:2720 +0x288
mattermost |
mattermost | goroutine 2776 [select]:
mattermost | net/http.(persistConn).roundTrip(0xc420075680, 0xc4211ce5a0, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/transport.go:1970 +0x608
mattermost | net/http.(Transport).RoundTrip(0x247ed20, 0xc4222fc600, 0x247ed20, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/transport.go:413 +0x992
mattermost | net/http.send(0xc4222fc600, 0x1d64cc0, 0x247ed20, 0x0, 0x0, 0x0, 0xc4223240f8, 0xc42342de68, 0x41360d, 0xc4223240f8)
mattermost | /usr/local/go/src/net/http/client.go:249 +0x1a9
mattermost | net/http.(Client).send(0xc421fe9200, 0xc4222fc600, 0x0, 0x0, 0x0, 0xc4223240f8, 0x0, 0x1, 0x0)
mattermost | /usr/local/go/src/net/http/client.go:173 +0xfd
mattermost | net/http.(Client).Do(0xc421fe9200, 0xc4222fc600, 0x40, 0xc4220d6160, 0xc4220d6160)
mattermost | /usr/local/go/src/net/http/client.go:602 +0x28d
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.do(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:496 +0x50
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.executeMethod(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:601 +0x382
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.initiateMultipartUpload(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-multipart.go:210 +0x2c7
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newUploadID(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-common.go:106 +0x16a
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectMultipartStreamNoLength(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:184 +0x20c
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectCommon(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:152 +0x642
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObjectWithContext(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-context.go:38 +0x302
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObject(0xc42177dd00, 0xc421d65380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc421fe9200, 0xc421ab9380, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:130 +0x114
mattermost | github.com/mattermost/mattermost-server/utils.(S3FileBackend).WriteFile(0xc421d65320, 0xc42356c000, 0x15616f, 0x1ffe00, 0xc4201e2500, 0x7b, 0x81)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/file_backend_s3.go:153 +0x2a4
mattermost | github.com/mattermost/mattermost-server/app.(App).WriteFile(0xc420087d40, 0xc42356c000, 0x15616f, 0x1ffe00, 0xc4201e2500, 0x7b, 0xa)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:86 +0x8e
mattermost | github.com/mattermost/mattermost-server/app.(App).DoUploadFile(0xc420087d40, 0xbeb409e4331aafba, 0xc36a1684d, 0x249a060, 0x13ebcc3, 0x6, 0xc422ac21e0, 0x1a, 0xc42291edc0, 0x1a, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:382 +0x690
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc422ac21e0, 0x1a, 0xc42291edc0, 0x1a, 0xc42179f1c0, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:329 +0x505
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadMultipartFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc422ac21e0, 0x1a, 0xc42291edc0, 0x1a, 0xc4223240c8, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:300 +0x41b
mattermost | github.com/mattermost/mattermost-server/api4.uploadFile(0xc422b57a40, 0x1d6e340, 0xc422b415e0, 0xc423402c00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/file.go:120 +0x562
mattermost | github.com/mattermost/mattermost-server/api4.handler.ServeHTTP(0xc420087d40, 0x148a640, 0x10001, 0x1d6e340, 0xc422b415e0, 0xc423402c00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/context.go:157 +0xb65
mattermost | github.com/mattermost/mattermost-server/api4.(handler).ServeHTTP(0xc4227a4600, 0x1d6e340, 0xc422b415e0, 0xc423402c00)
mattermost | :1 +0x63
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux.(Router).ServeHTTP(0xc4206800e0, 0x1d6e340, 0xc422b415e0, 0xc423402c00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux/mux.go:159 +0xed
mattermost | github.com/mattermost/mattermost-server/app.(CorsWrapper).ServeHTTP(0xc4217206f0, 0x1d6e340, 0xc422b415e0, 0xc423402a00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:83 +0xa2
mattermost | github.com/mattermost/mattermost-server/app.(RateLimiter).RateLimitHandler.func1(0x1d6e340, 0xc422b415e0, 0xc423402a00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/ratelimit.go:106 +0xca
mattermost | net/http.HandlerFunc.ServeHTTP(0xc421f82220, 0x1d6e340, 0xc422b415e0, 0xc423402a00)
mattermost | /usr/local/go/src/net/http/server.go:1918 +0x44
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP(0x1d67540, 0xc421f82220, 0x1d631c0, 0x24ba100, 0x1, 0x1d6e340, 0xc422b415e0, 0xc423402a00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers/recovery.go:78 +0xb9
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.(recoveryHandler).ServeHTTP(0xc421ff97a0, 0x1d6e340, 0xc422b415e0, 0xc423402a00)
mattermost | :1 +0x76
mattermost | net/http.serverHandler.ServeHTTP(0xc4212ba4e0, 0x1d6e340, 0xc422b415e0, 0xc423402a00)
mattermost | /usr/local/go/src/net/http/server.go:2619 +0xb4
mattermost | net/http.(conn).serve(0xc42218ae60, 0x1d6f0c0, 0xc422a20800)
mattermost | /usr/local/go/src/net/http/server.go:1801 +0x71d
mattermost | created by net/http.(Server).Serve
mattermost | /usr/local/go/src/net/http/server.go:2720 +0x288
mattermost |
mattermost | goroutine 2847 [select]:
mattermost | net/http.(persistConn).roundTrip(0xc4223be900, 0xc4212826c0, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/transport.go:1970 +0x608
mattermost | net/http.(Transport).RoundTrip(0x247ed20, 0xc4222fc900, 0x247ed20, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/transport.go:413 +0x992
mattermost | net/http.send(0xc4222fc900, 0x1d64cc0, 0x247ed20, 0x0, 0x0, 0x0, 0xc422324138, 0xc4232ba3a8, 0x41360d, 0xc422324138)
mattermost | /usr/local/go/src/net/http/client.go:249 +0x1a9
mattermost | net/http.(Client).send(0xc422030300, 0xc4222fc900, 0x0, 0x0, 0x0, 0xc422324138, 0x0, 0x1, 0x0)
mattermost | /usr/local/go/src/net/http/client.go:173 +0xfd
mattermost | net/http.(Client).Do(0xc422030300, 0xc4222fc900, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/client.go:602 +0x28d
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.do(0xc42177c000, 0xc421d640c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422030300, 0xc421ada000, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:496 +0x50
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.executeMethod(0xc42177c000, 0xc421d640c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422030300, 0xc421ada000, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api.go:601 +0x382
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.uploadPart(0xc42177c000, 0xc421d640c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422030300, 0xc421ada000, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-multipart.go:283 +0x494
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectMultipartStreamNoLength(0xc42177c000, 0xc421d640c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422030300, 0xc421ada000, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:219 +0x71e
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.putObjectCommon(0xc42177c000, 0xc421d640c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422030300, 0xc421ada000, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:152 +0x642
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObjectWithContext(0xc42177c000, 0xc421d640c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422030300, 0xc421ada000, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object-context.go:38 +0x302
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.PutObject(0xc42177c000, 0xc421d640c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc422030300, 0xc421ada000, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/api-put-object.go:130 +0x114
mattermost | github.com/mattermost/mattermost-server/utils.(S3FileBackend).WriteFile(0xc421d64000, 0xc422aa4000, 0x10cb, 0x20cb, 0xc422593560, 0x81, 0xc422302000)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/file_backend_s3.go:153 +0x2a4
mattermost | github.com/mattermost/mattermost-server/app.(App).WriteFile(0xc420087d40, 0xc422aa4000, 0x10cb, 0x20cb, 0xc422593560, 0x81, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:86 +0x8e
mattermost | github.com/mattermost/mattermost-server/app.(App).generateThumbnailImage(0xc420087d40, 0x1d6df00, 0xc4223fa080, 0xc422593560, 0x81, 0xfc0, 0xbd0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:499 +0x163
mattermost | github.com/mattermost/mattermost-server/app.(App).HandleImages.func1(0xc4229ca270, 0xc420087d40, 0xc42174e720, 0xc422593560, 0x81, 0xfc0, 0xbd0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:402 +0x93
mattermost | created by github.com/mattermost/mattermost-server/app.(App).HandleImages
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:400 +0x154
mattermost |
mattermost | goroutine 2574 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac87555b0, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc4212b9318, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc4212b9318, 0xc422102000, 0x2000, 0x2000)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc4212b9300, 0xc422102000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc4212b9300, 0xc422102000, 0x2000, 0x2000, 0x4cff9c, 0xc4223194ad, 0x13e6e52)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422b6cfa0, 0xc422102000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | bufio.(Reader).fill(0xc42294fda0)
mattermost | /usr/local/go/src/bufio/bufio.go:97 +0x11a
mattermost | bufio.(Reader).Peek(0xc42294fda0, 0x2, 0x76228fc601, 0x1af8bec981f910, 0x22, 0x148c468, 0xc4228bfe00)
mattermost | /usr/local/go/src/bufio/bufio.go:129 +0x3a
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(Conn).read(0xc422a13e00, 0x2, 0x44ded8, 0x810b1f7d, 0x7fcac8755640, 0xc4220f06c0, 0x1)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/conn_read.go:12 +0x40
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(Conn).advanceFrame(0xc422a13e00, 0x132e1e0, 0x0, 0x1c0000000000)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/conn.go:780 +0x5c
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(Conn).NextReader(0xc422a13e00, 0x44ded8, 0x1af8a7810b048c, 0xc4232bea58, 0x81a911, 0xc4232bea58)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/conn.go:940 +0xa3
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket.(Conn).ReadJSON(0xc422a13e00, 0x1241b20, 0xc4212a9e10, 0x249a060, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/websocket/json.go:50 +0x2f
mattermost | github.com/mattermost/mattermost-server/app.(WebConn).readPump(0xc42120fef0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:138 +0x17e
mattermost | github.com/mattermost/mattermost-server/app.(WebConn).Pump(0xc42120fef0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_conn.go:113 +0x79
mattermost | github.com/mattermost/mattermost-server/api4.connectWebSocket(0xc422392f00, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1d00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/websocket.go:39 +0x320
mattermost | github.com/mattermost/mattermost-server/api4.handler.ServeHTTP(0xc420087d40, 0x1489fe8, 0x100, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1d00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/context.go:157 +0xb65
mattermost | github.com/mattermost/mattermost-server/api4.(handler).ServeHTTP(0xc4227ef720, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1d00)
mattermost | :1 +0x63
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux.(Router).ServeHTTP(0xc4206800e0, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1d00)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux/mux.go:159 +0xed
mattermost | github.com/mattermost/mattermost-server/app.(CorsWrapper).ServeHTTP(0xc4217206f0, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1700)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:83 +0xa2
mattermost | github.com/mattermost/mattermost-server/app.(RateLimiter).RateLimitHandler.func1(0x1d6e340, 0xc4228f8fc0, 0xc4228e1700)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/ratelimit.go:106 +0xca
mattermost | net/http.HandlerFunc.ServeHTTP(0xc421f82220, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1700)
mattermost | /usr/local/go/src/net/http/server.go:1918 +0x44
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP(0x1d67540, 0xc421f82220, 0x1d631c0, 0x24ba100, 0x1, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1700)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers/recovery.go:78 +0xb9
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.(recoveryHandler).ServeHTTP(0xc421ff97a0, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1700)
mattermost | :1 +0x76
mattermost | net/http.serverHandler.ServeHTTP(0xc4212ba4e0, 0x1d6e340, 0xc4228f8fc0, 0xc4228e1700)
mattermost | /usr/local/go/src/net/http/server.go:2619 +0xb4
mattermost | net/http.(conn).serve(0xc42298bf40, 0x1d6f0c0, 0xc42257c980)
mattermost | /usr/local/go/src/net/http/server.go:1801 +0x71d
mattermost | created by net/http.(Server).Serve
mattermost | /usr/local/go/src/net/http/server.go:2720 +0x288
mattermost |
mattermost | goroutine 2885 [runnable]:
mattermost | net/http.(persistConn).writeLoop(0xc4223be900)
mattermost | /usr/local/go/src/net/http/transport.go:1756
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1187 +0xa53
mattermost |
mattermost | goroutine 2667 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc42186dd40, 0xc422303ec0, 0xc421d591a0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2668 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc42186dda0, 0xc422303d40, 0xc421d59200)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2848 [runnable]:
mattermost | image/jpeg.(encoder).emitHuff(0xc422302240, 0x1, 0x100000000)
mattermost | /usr/local/go/src/image/jpeg/writer.go:271 +0x81
mattermost | image/jpeg.(encoder).writeBlock(0xc422302240, 0xc4215114b0, 0x0, 0xffffff6c, 0xc4215114b0)
mattermost | /usr/local/go/src/image/jpeg/writer.go:383 +0x253
mattermost | image/jpeg.(encoder).writeSOS(0xc422302240, 0x1d6df00, 0xc4223fa080)
mattermost | /usr/local/go/src/image/jpeg/writer.go:551 +0x2c8
mattermost | image/jpeg.Encode(0x1d629c0, 0xc4218de690, 0x1d6df00, 0xc4223fa080, 0xc4232feea0, 0x148ac38, 0xc4223fa080)
mattermost | /usr/local/go/src/image/jpeg/writer.go:634 +0x2c6
mattermost | github.com/mattermost/mattermost-server/app.(App).generatePreviewImage(0xc420087d40, 0x1d6df00, 0xc4223fa080, 0xc4225934d0, 0x83, 0xfc0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:516 +0xb3
mattermost | github.com/mattermost/mattermost-server/app.(App).HandleImages.func2(0xc4229ca270, 0xc420087d40, 0xc42174e720, 0xc4225934d0, 0x83, 0xfc0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:407 +0x89
mattermost | created by github.com/mattermost/mattermost-server/app.(App).HandleImages
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:405 +0x1c9
mattermost |
mattermost | goroutine 2876 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac86b6738, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc42177d998, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc42177d998, 0xc422c76400, 0x400, 0x400)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc42177d980, 0xc422c76400, 0x400, 0x400, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc42177d980, 0xc422c76400, 0x400, 0x400, 0xc42257e000, 0x7fcac87b1000, 0x0)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc4223240c0, 0xc422c76400, 0x400, 0x400, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | crypto/tls.(block).readFromUntil(0xc421fffbf0, 0x7fcac87115c0, 0xc4223240c0, 0x5, 0xc4223240c0, 0xc4220223c0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:488 +0x95
mattermost | crypto/tls.(Conn).readRecord(0xc421faa000, 0x148ce17, 0xc421faa120, 0x42f57b)
mattermost | /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
mattermost | crypto/tls.(Conn).Read(0xc421faa000, 0xc422b5a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
mattermost | net/http.(persistConn).Read(0xc4200757a0, 0xc422b5a000, 0x1000, 0x1000, 0xc422b66550, 0xc42202e358, 0x458ad0)
mattermost | /usr/local/go/src/net/http/transport.go:1391 +0x140
mattermost | bufio.(Reader).fill(0xc4212e52c0)
mattermost | /usr/local/go/src/bufio/bufio.go:97 +0x11a
mattermost | bufio.(Reader).Peek(0xc4212e52c0, 0x1, 0x0, 0x0, 0x0, 0xc42202e120, 0x0)
mattermost | /usr/local/go/src/bufio/bufio.go:129 +0x3a
mattermost | net/http.(persistConn).readLoop(0xc4200757a0)
mattermost | /usr/local/go/src/net/http/transport.go:1539 +0x185
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1186 +0xa2e
mattermost |
mattermost | goroutine 2669 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc42186de00, 0xc422303bc0, 0xc421d59260)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2590 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac87558b0, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc422006698, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc422006698, 0xc42009d400, 0x400, 0x400)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc422006680, 0xc42009d400, 0x400, 0x400, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc422006680, 0xc42009d400, 0x400, 0x400, 0xc421b385ee, 0x8, 0xc422b038a0)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422b6ce08, 0xc42009d400, 0x400, 0x400, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | crypto/tls.(block).readFromUntil(0xc421b08600, 0x7fcac87115c0, 0xc422b6ce08, 0x5, 0xc422b6ce08, 0x0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:488 +0x95
mattermost | crypto/tls.(Conn).readRecord(0xc421fab500, 0x148ce17, 0xc421fab620, 0x0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
mattermost | crypto/tls.(Conn).Read(0xc421fab500, 0xc4229ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
mattermost | bufio.(Reader).Read(0xc421b5b200, 0xc4228f9618, 0x9, 0x9, 0xc422b03c70, 0x42caf6, 0x148cee0)
mattermost | /usr/local/go/src/bufio/bufio.go:213 +0x30b
mattermost | io.ReadAtLeast(0x1d62900, 0xc421b5b200, 0xc4228f9618, 0x9, 0x9, 0x9, 0xc422b03cd0, 0xc422b03cd0, 0x406be2)
mattermost | /usr/local/go/src/io/io.go:309 +0x86
mattermost | io.ReadFull(0x1d62900, 0xc421b5b200, 0xc4228f9618, 0x9, 0x9, 0xc421b5b260, 0xc422b03d08, 0xc400000001)
mattermost | /usr/local/go/src/io/io.go:327 +0x58
mattermost | net/http.http2readFrameHeader(0xc4228f9618, 0x9, 0x9, 0x1d62900, 0xc421b5b200, 0x0, 0xc400000000, 0xc422b03df0, 0x6df199)
mattermost | /usr/local/go/src/net/http/h2_bundle.go:1516 +0x7b
mattermost | net/http.(http2Framer).ReadFrame(0xc4228f95e0, 0xc422119aa0, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/h2_bundle.go:1774 +0xa4
mattermost | net/http.(http2clientConnReadLoop).run(0xc422b03fb0, 0x148c6f8, 0xc4212adfb0)
mattermost | /usr/local/go/src/net/http/h2_bundle.go:7862 +0x92
mattermost | net/http.(http2ClientConn).readLoop(0xc422949ba0)
mattermost | /usr/local/go/src/net/http/h2_bundle.go:7788 +0x9d
mattermost | created by net/http.(http2Transport).newClientConn
mattermost | /usr/local/go/src/net/http/h2_bundle.go:7053 +0x6b9
mattermost |
mattermost | goroutine 2880 [select]:
mattermost | net/http.(persistConn).writeLoop(0xc420075680)
mattermost | /usr/local/go/src/net/http/transport.go:1759 +0x165
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1187 +0xa53
mattermost |
mattermost | goroutine 2836 [runnable]:
mattermost | internal/poll.runtime_pollWait(0x7fcac8755070, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc421bdc198, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc421bdc198, 0xc424130000, 0x1000, 0x1000)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc421bdc180, 0xc424130000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc421bdc180, 0xc424130000, 0x1000, 0x1000, 0xc42257e000, 0x7fcac87b1000, 0x0)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422b6dd98, 0xc424130000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | crypto/tls.(block).readFromUntil(0xc42412c6c0, 0x7fcac87115c0, 0xc422b6dd98, 0x5, 0xc422b6dd98, 0x7fcac87b1000)
mattermost | /usr/local/go/src/crypto/tls/conn.go:488 +0x95
mattermost | crypto/tls.(Conn).readRecord(0xc422a19c00, 0x148ce17, 0xc422a19d20, 0x42f57b)
mattermost | /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
mattermost | crypto/tls.(Conn).Read(0xc422a19c00, 0xc424131000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
mattermost | net/http.(persistConn).Read(0xc4234077a0, 0xc424131000, 0x1000, 0x1000, 0xc422b673b0, 0xc42233e538, 0x458ad0)
mattermost | /usr/local/go/src/net/http/transport.go:1391 +0x140
mattermost | bufio.(Reader).fill(0xc42340ccc0)
mattermost | /usr/local/go/src/bufio/bufio.go:97 +0x11a
mattermost | bufio.(Reader).Peek(0xc42340ccc0, 0x1, 0x0, 0x0, 0x0, 0xc4200508a0, 0x0)
mattermost | /usr/local/go/src/bufio/bufio.go:129 +0x3a
mattermost | net/http.(persistConn).readLoop(0xc4234077a0)
mattermost | /usr/local/go/src/net/http/transport.go:1539 +0x185
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1186 +0xa2e
mattermost |
mattermost | goroutine 2670 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc42186de60, 0xc421d74240, 0xc421d592c0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2826 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac8755a30, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc422007518, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc422007518, 0xc4220ac000, 0x1, 0x1)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc422007500, 0xc4220ac0d1, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc422007500, 0xc4220ac0d1, 0x1, 0x1, 0xc421e9e000, 0x0, 0x1d64601)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422b6cc58, 0xc4220ac0d1, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | net/http.(connReader).backgroundRead(0xc4220ac0c0)
mattermost | /usr/local/go/src/net/http/server.go:660 +0x62
mattermost | created by net/http.(connReader).startBackgroundRead
mattermost | /usr/local/go/src/net/http/server.go:656 +0xd8
mattermost |
mattermost | goroutine 2858 [sleep]:
mattermost | time.Sleep(0x2f7ae743)
mattermost | /usr/local/go/src/runtime/time.go:65 +0x130
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc4200519e0, 0xa, 0xc420051920, 0xc4220164e0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x168
mattermost |
mattermost | goroutine 2883 [runnable]:
mattermost | internal/poll.runtime_pollWait(0x7fcac86b6678, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc4217b1318, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc4217b1318, 0xc424118000, 0x1000, 0x1000)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc4217b1300, 0xc424118000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc4217b1300, 0xc424118000, 0x1000, 0x1000, 0x10, 0x10, 0x3cb)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422324018, 0xc424118000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | crypto/tls.(block).readFromUntil(0xc421e18f00, 0x7fcac87115c0, 0xc422324018, 0x5, 0xc422324018, 0x14)
mattermost | /usr/local/go/src/crypto/tls/conn.go:488 +0x95
mattermost | crypto/tls.(Conn).readRecord(0xc420111880, 0x14, 0x303, 0xc4217b1318)
mattermost | /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
mattermost | crypto/tls.(clientHandshakeState).readFinished(0xc421569e68, 0xc420111975, 0xc, 0xc, 0x0, 0x0)
mattermost | /usr/local/go/src/crypto/tls/handshake_client.go:584 +0x53
mattermost | crypto/tls.(Conn).clientHandshake(0xc420111880, 0x148cee0, 0xc4201119a0)
mattermost | /usr/local/go/src/crypto/tls/handshake_client.go:244 +0x10aa
mattermost | crypto/tls.(Conn).Handshake(0xc420111880, 0x0, 0x0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:1307 +0x18f
mattermost | net/http.(Transport).dialConn.func3(0x0, 0xc420111880, 0xc42257cbc0, 0xc4212e5920)
mattermost | /usr/local/go/src/net/http/transport.go:1151 +0x42
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1147 +0xdbe
mattermost |
mattermost | goroutine 2866 [sleep]:
mattermost | time.Sleep(0x30eb693c)
mattermost | /usr/local/go/src/runtime/time.go:65 +0x130
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc4212f0a80, 0xa, 0xc4212f0660, 0xc4211ce210)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x168
mattermost |
mattermost | goroutine 2884 [runnable]:
mattermost | net/http.(persistConn).readLoop(0xc4223be900)
mattermost | /usr/local/go/src/net/http/transport.go:1504
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1186 +0xa2e
mattermost |
mattermost | goroutine 2546 [semacquire]:
mattermost | sync.runtime_Semacquire(0xc4229ca27c)
mattermost | /usr/local/go/src/runtime/sema.go:56 +0x39
mattermost | sync.(WaitGroup).Wait(0xc4229ca270)
mattermost | /usr/local/go/src/sync/waitgroup.go:131 +0x72
mattermost | github.com/mattermost/mattermost-server/app.(App).HandleImages(0xc420087d40, 0xc42174e630, 0x1, 0x1, 0xc42174e680, 0x1, 0x1, 0xc421ada160, 0x1, 0x1)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:411 +0x1d7
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc421884be0, 0x1a, 0xc42291edc0, 0x1a, 0xc4229c0c30, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:347 +0xd45
mattermost | github.com/mattermost/mattermost-server/app.(App).UploadMultipartFiles(0xc420087d40, 0x13ebcc3, 0x6, 0xc421884be0, 0x1a, 0xc42291edc0, 0x1a, 0xc422b6dd50, 0x1, 0x1, ...)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/file.go:300 +0x41b
mattermost | github.com/mattermost/mattermost-server/api4.uploadFile(0xc422b57860, 0x1d6e340, 0xc422b41420, 0xc423402600)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/file.go:120 +0x562
mattermost | github.com/mattermost/mattermost-server/api4.handler.ServeHTTP(0xc420087d40, 0x148a640, 0x10001, 0x1d6e340, 0xc422b41420, 0xc423402600)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/api4/context.go:157 +0xb65
mattermost | github.com/mattermost/mattermost-server/api4.(handler).ServeHTTP(0xc4227a4600, 0x1d6e340, 0xc422b41420, 0xc423402600)
mattermost | :1 +0x63
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux.(Router).ServeHTTP(0xc4206800e0, 0x1d6e340, 0xc422b41420, 0xc423402600)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/mux/mux.go:159 +0xed
mattermost | github.com/mattermost/mattermost-server/app.(CorsWrapper).ServeHTTP(0xc4217206f0, 0x1d6e340, 0xc422b41420, 0xc423402400)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:83 +0xa2
mattermost | github.com/mattermost/mattermost-server/app.(RateLimiter).RateLimitHandler.func1(0x1d6e340, 0xc422b41420, 0xc423402400)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/ratelimit.go:106 +0xca
mattermost | net/http.HandlerFunc.ServeHTTP(0xc421f82220, 0x1d6e340, 0xc422b41420, 0xc423402400)
mattermost | /usr/local/go/src/net/http/server.go:1918 +0x44
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP(0x1d67540, 0xc421f82220, 0x1d631c0, 0x24ba100, 0x1, 0x1d6e340, 0xc422b41420, 0xc423402400)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers/recovery.go:78 +0xb9
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/gorilla/handlers.(recoveryHandler).ServeHTTP(0xc421ff97a0, 0x1d6e340, 0xc422b41420, 0xc423402400)
mattermost | :1 +0x76
mattermost | net/http.serverHandler.ServeHTTP(0xc4212ba4e0, 0x1d6e340, 0xc422b41420, 0xc423402400)
mattermost | /usr/local/go/src/net/http/server.go:2619 +0xb4
mattermost | net/http.(conn).serve(0xc4228d0460, 0x1d6f0c0, 0xc422825b00)
mattermost | /usr/local/go/src/net/http/server.go:1801 +0x71d
mattermost | created by net/http.(Server).Serve
mattermost | /usr/local/go/src/net/http/server.go:2720 +0x288
mattermost |
mattermost | goroutine 2513 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac87557f0, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc42177df18, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc42177df18, 0xc4227f2000, 0x1000, 0x1000)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc42177df00, 0xc4227f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc42177df00, 0xc4227f2000, 0x1000, 0x1000, 0xc421f57c50, 0x6f063a, 0xc421e33928)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422325d00, 0xc4227f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | net/http.(connReader).Read(0xc421e33920, 0xc4227f2000, 0x1000, 0x1000, 0xc421f57d38, 0x52410c, 0xc42177df00)
mattermost | /usr/local/go/src/net/http/server.go:753 +0x105
mattermost | bufio.(Reader).fill(0xc4212168a0)
mattermost | /usr/local/go/src/bufio/bufio.go:97 +0x11a
mattermost | bufio.(Reader).Peek(0xc4212168a0, 0x4, 0x50bc84d095, 0x249a060, 0x0, 0x0, 0x249a060)
mattermost | /usr/local/go/src/bufio/bufio.go:129 +0x3a
mattermost | net/http.(conn).serve(0xc420177360, 0x1d6f0c0, 0xc421ab5580)
mattermost | /usr/local/go/src/net/http/server.go:1826 +0x88f
mattermost | created by net/http.(Server).Serve
mattermost | /usr/local/go/src/net/http/server.go:2720 +0x288
mattermost |
mattermost | goroutine 2467 [select]:
mattermost | github.com/mattermost/mattermost-server/jobs.(Schedulers).Start.func1.1()
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/schedulers.go:79 +0x4ad
mattermost | sync.(Once).Do(0xc4201be798, 0xc4217757b8)
mattermost | /usr/local/go/src/sync/once.go:44 +0xbe
mattermost | github.com/mattermost/mattermost-server/jobs.(Schedulers).Start.func1(0xc4201be770)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/schedulers.go:61 +0x4c
mattermost | created by github.com/mattermost/mattermost-server/jobs.(Schedulers).Start
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/jobs/schedulers.go:60 +0xb9
mattermost |
mattermost | goroutine 2292 [syscall]:
mattermost | syscall.Syscall6(0xe8, 0xa, 0xc422479c24, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
mattermost | github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix.EpollWait(0xa, 0xc422479c24, 0x7, 0x7, 0xffffffffffffffff, 0xc42176af00, 0xc4223fb540, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/golang.org/x/sys/unix/zsyscall_linux_amd64.go:1544 +0x77
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(fdPoller).wait(0xc42239a6e0, 0x148ac00, 0xc42239a6e0, 0x0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify_poller.go:86 +0x91
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.(Watcher).readEvents(0xc4223b6870)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:192 +0x194
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify.NewWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x23a
mattermost |
mattermost | goroutine 2263 [select]:
mattermost | github.com/mattermost/mattermost-server/utils.NewConfigWatcher.func1(0xc4223655a0, 0xc421931e50, 0xc421d98800, 0x1e, 0xc421d98800, 0x1e, 0xc422374740)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/config.go:184 +0x1e5
mattermost | created by github.com/mattermost/mattermost-server/utils.NewConfigWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/config.go:179 +0x1bb
mattermost |
mattermost | goroutine 2881 [chan receive]:
mattermost | net/http.(Transport).getConn.func2.1(0xc4212f0ba0, 0x247ed20, 0x148c8b0)
mattermost | /usr/local/go/src/net/http/transport.go:932 +0x50
mattermost | created by net/http.(Transport).getConn.func2
mattermost | /usr/local/go/src/net/http/transport.go:931 +0x7a
mattermost |
mattermost | goroutine 2873 [sleep]:
mattermost | time.Sleep(0xe74bf53)
mattermost | /usr/local/go/src/runtime/time.go:65 +0x130
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer.func2(0xc4212f1200, 0xa, 0xc4212f11a0, 0xc421243aa0)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:82 +0x84
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go.Client.newRetryTimer
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/minio/minio-go/retry.go:72 +0x168
mattermost |
mattermost | goroutine 2477 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc4212ecb40, 0xc422302a80, 0xc42174d440)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2478 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc4212ecba0, 0xc422302900, 0xc42174d500)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2479 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc4212ecc00, 0xc422302780, 0xc42174d560)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2261 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.NewFileLogWriter.func1(0xc420082e60)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/filelog.go:87 +0x140
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go.NewFileLogWriter
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/alecthomas/log4go/filelog.go:77 +0x26a
mattermost |
mattermost | goroutine 2837 [select]:
mattermost | net/http.(persistConn).writeLoop(0xc4234077a0)
mattermost | /usr/local/go/src/net/http/transport.go:1759 +0x165
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1187 +0xa53
mattermost |
mattermost | goroutine 2666 [select]:
mattermost | github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher.func1(0xc42186dce0, 0xc421d740c0, 0xc421d59140)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:182 +0xf8
mattermost | created by github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql.(mysqlConn).startWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/vendor/github.com/go-sql-driver/mysql/connection_go18.go:179 +0xce
mattermost |
mattermost | goroutine 2476 [select]:
mattermost | github.com/mattermost/mattermost-server/model.createTask.func1(0xc421a51c80, 0xd18c2e28000, 0xc4213c8930)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:49 +0x15a
mattermost | created by github.com/mattermost/mattermost-server/model.createTask
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/model/scheduled_task.go:40 +0x10c
mattermost |
mattermost | goroutine 2450 [select]:
mattermost | github.com/mattermost/mattermost-server/app.(App).HubStart.func1(0xc420087d40)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:88 +0x5d6
mattermost | created by github.com/mattermost/mattermost-server/app.(App).HubStart
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:80 +0x516
mattermost |
mattermost | goroutine 2447 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac8755eb0, 0x72, 0xffffffffffffffff)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc42177ce98, 0x72, 0xc421892e00, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc42177ce98, 0xffffffffffffff00, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Accept(0xc42177ce80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:335 +0x1e2
mattermost | net.(netFD).accept(0xc42177ce80, 0xc421892fd8, 0x401c27, 0xc42218aee0)
mattermost | /usr/local/go/src/net/fd_unix.go:238 +0x42
mattermost | net.(TCPListener).accept(0xc422324390, 0x6fa448, 0x45b1a0, 0xc421893020)
mattermost | /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
mattermost | net.(TCPListener).Accept(0xc422324390, 0x148c6c0, 0xc42218ae60, 0x1d6f180, 0xc42234a5d0)
mattermost | /usr/local/go/src/net/tcpsock.go:247 +0x49
mattermost | net/http.(Server).Serve(0xc4212ba4e0, 0x1d6e0c0, 0xc422324390, 0x0, 0x0)
mattermost | /usr/local/go/src/net/http/server.go:2695 +0x1b2
mattermost | github.com/mattermost/mattermost-server/app.(App).StartServer.func2(0xc420087d40, 0xc4229d66e0, 0x1d6e0c0, 0xc422324390)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:197 +0x774
mattermost | created by github.com/mattermost/mattermost-server/app.(App).StartServer
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/server.go:180 +0xcb7
mattermost |
mattermost | goroutine 2448 [select]:
mattermost | github.com/mattermost/mattermost-server/app.(Hub).Start.func1()
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:375 +0x42a
mattermost | github.com/mattermost/mattermost-server/app.(Hub).Start.func2()
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:434 +0x4c
mattermost | created by github.com/mattermost/mattermost-server/app.(Hub).Start
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:451 +0x128
mattermost |
mattermost | goroutine 2449 [select]:
mattermost | github.com/mattermost/mattermost-server/app.(Hub).Start.func1()
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:375 +0x42a
mattermost | github.com/mattermost/mattermost-server/app.(Hub).Start.func2()
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:434 +0x4c
mattermost | created by github.com/mattermost/mattermost-server/app.(Hub).Start
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/app/web_hub.go:451 +0x128
mattermost |
mattermost | goroutine 2863 [IO wait]:
mattermost | internal/poll.runtime_pollWait(0x7fcac87551f0, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc421522918, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc421522918, 0xc42240a000, 0x1, 0x1)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc421522900, 0xc42240a0d1, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc421522900, 0xc42240a0d1, 0x1, 0x1, 0xc421faa000, 0x148cee0, 0xc421faa120)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422b6db18, 0xc42240a0d1, 0x1, 0x1, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | net/http.(connReader).backgroundRead(0xc42240a0c0)
mattermost | /usr/local/go/src/net/http/server.go:660 +0x62
mattermost | created by net/http.(connReader).startBackgroundRead
mattermost | /usr/local/go/src/net/http/server.go:656 +0xd8
mattermost |
mattermost | goroutine 2879 [runnable]:
mattermost | internal/poll.runtime_pollWait(0x7fcac8755130, 0x72, 0x0)
mattermost | /usr/local/go/src/runtime/netpoll.go:173 +0x57
mattermost | internal/poll.(pollDesc).wait(0xc42177db18, 0x72, 0xffffffffffffff00, 0x1d68080, 0x1d60bf0)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0xae
mattermost | internal/poll.(pollDesc).waitRead(0xc42177db18, 0xc422c76c00, 0x400, 0x400)
mattermost | /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
mattermost | internal/poll.(FD).Read(0xc42177db00, 0xc422c76c00, 0x400, 0x400, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/internal/poll/fd_unix.go:126 +0x18a
mattermost | net.(netFD).Read(0xc42177db00, 0xc422c76c00, 0x400, 0x400, 0xc4200220f8, 0xc423300778, 0x41360d)
mattermost | /usr/local/go/src/net/fd_unix.go:202 +0x52
mattermost | net.(conn).Read(0xc422324118, 0xc422c76c00, 0x400, 0x400, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/net/net.go:176 +0x6d
mattermost | crypto/tls.(block).readFromUntil(0xc42122bb90, 0x7fcac87115c0, 0xc422324118, 0x5, 0xc422324118, 0xc421faa000)
mattermost | /usr/local/go/src/crypto/tls/conn.go:488 +0x95
mattermost | crypto/tls.(Conn).readRecord(0xc421faa380, 0x148ce17, 0xc421faa4a0, 0xae1d29622c4afa1f)
mattermost | /usr/local/go/src/crypto/tls/conn.go:590 +0xe0
mattermost | crypto/tls.(Conn).Read(0xc421faa380, 0xc422445000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
mattermost | /usr/local/go/src/crypto/tls/conn.go:1134 +0x110
mattermost | net/http.(persistConn).Read(0xc420075680, 0xc422445000, 0x1000, 0x1000, 0x0, 0x0, 0xc423300b68)
mattermost | /usr/local/go/src/net/http/transport.go:1391 +0x140
mattermost | bufio.(Reader).fill(0xc4212e53e0)
mattermost | /usr/local/go/src/bufio/bufio.go:97 +0x11a
mattermost | bufio.(Reader).Peek(0xc4212e53e0, 0x1, 0xc42202f080, 0xc421578480, 0xc42257e000, 0x7fcac87b1000, 0x0)
mattermost | /usr/local/go/src/bufio/bufio.go:129 +0x3a
mattermost | net/http.(persistConn).readLoop(0xc420075680)
mattermost | /usr/local/go/src/net/http/transport.go:1539 +0x185
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1186 +0xa2e
mattermost |
mattermost | goroutine 2316 [select]:
mattermost | github.com/mattermost/mattermost-server/utils.NewHTMLTemplateWatcher.func1(0xc4223b59c0, 0xc4223b6870, 0xc42239a6a0, 0x15)
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/html.go:55 +0x1ef
mattermost | created by github.com/mattermost/mattermost-server/utils.NewHTMLTemplateWatcher
mattermost | /var/lib/jenkins/jobs/msr/jobs/d/jobs/enterprise-release/workspace/src/github.com/mattermost/mattermost-server/utils/html.go:50 +0x2d4
mattermost |
mattermost | goroutine 2317 [chan receive]:
mattermost | database/sql.(DB).connectionOpener(0xc420083360)
mattermost | /usr/local/go/src/database/sql/sql.go:871 +0x53
mattermost | created by database/sql.Open
mattermost | /usr/local/go/src/database/sql/sql.go:609 +0x1ee
mattermost |
mattermost | goroutine 2877 [select]:
mattermost | net/http.(persistConn).writeLoop(0xc4200757a0)
mattermost | /usr/local/go/src/net/http/transport.go:1759 +0x165
mattermost | created by net/http.(Transport).dialConn
mattermost | /usr/local/go/src/net/http/transport.go:1187 +0xa53
mattermost exited with code 2
`

[lindy65]

Thanks @alecisec,

Our developers are working with a community member on a ticket that might improve this issue...