Mattermost-server: Got SIGSEGV: segmentation violation when I tried to add the user

[swood]

Summary

I got SIGSEGV when I tried to add the new user via command line.

Steps to reproduce

I have this problem with mattermost-preview version:
Mattermost Version: 5.3.0-rc2
Database Schema Version: 5.3.0
Generate jsonl by the documentation and execute command /mm/mattermost/bin# ./mattermost import bulk test.json --apply

Expected behavior

I have a simple jsonl file with users data:
{ "type": "version", "version": 1 }
{"type": "user","user": {"username": "bulk_imported_user2","email": "[email protected]","password": "some_password","nickname": "bulk_imported","first_name": "Bulk","last_name": "Impoeted","position": "Developer","roles": "system_user","locale": "en","teams": [{ "name": "TeamName", "roles": "team_user", "channels": []}]}}

Observed behavior (that appears unintentional)

That jsonl worked before. But now I received the next output:

{"level":"info","msg":"Stopping Server..."}
{"level":"info","msg":"stopping websocket hub connections"}
{"level":"warn","msg":"We appear to have already sent the stop checking for deadlocks command"}
{"level":"info","msg":"Server stopped"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xee6d4a]

goroutine 1 [running]:
github.com/mattermost/mattermost-server/app.(App).Config(...)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/app/config.go:27
github.com/mattermost/mattermost-server/cmd/mattermost/commands.InitDBCommandContextCobra(0x1c40c40, 0x6, 0x6, 0xc4202a6300)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/init.go:20 +0xea
github.com/mattermost/mattermost-server/cmd/mattermost/commands.bulkImportCmdF(0x1c40c40, 0xc42012c800, 0x1, 0x2, 0x0, 0x0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/import.go:94 +0x4e
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).execute(0x1c40c40, 0xc42012c7c0, 0x2, 0x2, 0x1c40c40, 0xc42012c7c0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).ExecuteC(0x1c3a3c0, 0x1133980, 0x200, 0x17)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).Execute(0x1c3a3c0, 0xc42009a058, 0x0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/mattermost/mattermost-server/cmd/mattermost/commands.Run(0xc4200b6100, 0x4, 0x4, 0xc4200ddf78, 0xc42009a058)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/root.go:14 +0x6d
main.main()
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/main.go:30 +0x63

Possible fixes

No, I can't fix the problem.

[amyblais]

Hi @swood, thank you for reaching out.

mattermost-preview version

Are you referring to the Docker Preview (https://github.com/mattermost/mattermost-docker-preview)?

[swood]

Yes, exactly.

[swood]

I have some investigation about this problem. It reproduces only if there are a lot of users in current installation. If I try it with an empty container all works fine.
I checked it with latest container version with binary from 19.09.2018 and md5sum e035c6b203951b3fd2923bb83480281c.

Maybe someone could help with it?

Also, I can't view a version of mattermost via cli:

./mattermost version

{"level":"info","msg":"Stopping Server..."}
{"level":"info","msg":"stopping websocket hub connections"}
{"level":"warn","msg":"We appear to have already sent the stop checking for deadlocks command"}
{"level":"info","msg":"Server stopped"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xee6fea]

goroutine 1 [running]:
github.com/mattermost/mattermost-server/app.(App).Config(...)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/app/config.go:27
github.com/mattermost/mattermost-server/cmd/mattermost/commands.InitDBCommandContextCobra(0x1c3df20, 0xc42020bc70, 0x0, 0x0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/init.go:20 +0xea
github.com/mattermost/mattermost-server/cmd/mattermost/commands.versionCmdF(0x1c3df20, 0x1c89488, 0x0, 0x0, 0x0, 0x0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/version.go:25 +0x2b
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).execute(0x1c3df20, 0x1c89488, 0x0, 0x0, 0x1c3df20, 0x1c89488)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).ExecuteC(0x1c3a3c0, 0x1133980, 0x200, 0x65)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).Execute(0x1c3a3c0, 0xc420098058, 0x0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/mattermost/mattermost-server/cmd/mattermost/commands.Run(0xc4200b8010, 0x1, 0x1, 0xc42020bf78, 0xc420098058)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/root.go:14 +0x6d
main.main()
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/main.go:30 +0x63

[amyblais]

I apologize for the delay - to confirm, you're using this command to create a new user: https://docs.mattermost.com/administration/command-line-tools.html#mattermost-user-create?

I'll ask our team to help test this out.

[swood]

Hello.

Yes, that's right:

./mattermost user create --firstname Joe --system_admin --email [email protected] --username joe --password Password1
{"level":"info","msg":"Stopping Server..."}
{"level":"info","msg":"stopping websocket hub connections"}
{"level":"warn","msg":"We appear to have already sent the stop checking for deadlocks command"}
{"level":"info","msg":"Server stopped"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xee6fea]

goroutine 1 [running]:
github.com/mattermost/mattermost-server/app.(App).Config(...)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/app/config.go:27
github.com/mattermost/mattermost-server/cmd/mattermost/commands.InitDBCommandContextCobra(0x1c3c760, 0x32, 0x1c8a880, 0x145a000)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/init.go:20 +0xea
github.com/mattermost/mattermost-server/cmd/mattermost/commands.userCreateCmdF(0x1c3c760, 0xc42056e120, 0x0, 0x9, 0x0, 0x0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/user.go:308 +0x4e
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).execute(0x1c3c760, 0xc42056e090, 0x9, 0x9, 0x1c3c760, 0xc42056e090)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).ExecuteC(0x1c3a3c0, 0x49, 0x200, 0x65)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra.(Command).Execute(0x1c3a3c0, 0xc42009a058, 0x0)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/mattermost/mattermost-server/cmd/mattermost/commands.Run(0xc4200c8010, 0xb, 0xb, 0xc42021ff78, 0xc42009a058)
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/commands/root.go:14 +0x6d
main.main()
/home/ubuntu/jenkins/workspace/msr/mattermost-platform-release/src/github.com/mattermost/mattermost-server/cmd/mattermost/main.go:30 +0x63

In another words: I can't use mattermost-cli at this moment.

[amyblais]

Hi @swood I checked with our team on this and a community member noticed that you're on RC-2 version - would you be able to update to the latest version and see if that helps?

[fbartels]

Hi @swood,

this reminds me a bit of the issue I reported two weeks ago (https://github.com/mattermost/mattermost-server/issues/9421). Does the issue also occur for your if you disable the plugin framework in Mattermost?

[swood]

Hello, @amyblais. As I said before I have the same problem with the latest version.
And also my installation haven't any plugins except standard like Jira.

[fbartels]

And also my installation haven't any plugins except standard like Jira.

I would still recommend to retest this when setting "Enable": false, within the PluginSettings block in your config.json. Maybe you stumbled upon the same issue I have.

[DSchalla]

https://github.com/mattermost/mattermost-server/blob/v5.3.0-rc2/cmd/mattermost/commands/init.go#L19 If Line 19 returns an error and a is nil for whatever reason, a.Config() would trigger the runtime panic, no?

Line 20 should be moved to line 26 I guess so that if there is an issue with the DB connection e.g. the actual error is printed, some of the error cases:

• translation Files cannot be loaded
• config file cannot be loaded
• SetupInviteEmailRateLimiting fails (for whatever reason that should happen)
• Wrong subpath settings ...

This makes it a bit hard to debug this right now. I guess one could check what exactly goes wrong with strace to check which file open fails (my current guess), but this should be fixed upstream to avoid it at all? For now I'd check if the translation files are in place, the config is placed as intended, and your current working directory is the mattermost directory itself. CC @jwilander @crspeller

[jwilander]

Yes we shouldn't definitely fix that. @amyblais can you file a bug ticket with @DSchalla's comment and linking to this thread?

[DSchalla]

@amyblais I will send a PR later today to fix that, feel free to file the bug ticket never the less of course.

[swood]

@fbartels yes, I tried, but it didn't help. Unfortunately.
@jwilander @amyblais when I could try a build with fix from @DSchalla?

[jwilander]

When a PR gets submitted a build is automatically created based off master, so you could test with that

[DSchalla]

PR got submitted, please be aware that this fix will not solve the problem itself, but instead will point out the actual root-cause why the App Runtime could not be started @swood

Jenkins Build: https://build.mattermost.com/job/mp/job/mattermost-platform-pr-split/4129/

[swood]

I can see that the build was successful. Could anyone review and merge it?

[amyblais]

Link to associated Jira ticket: https://mattermost.atlassian.net/browse/MM-12468.