Mattermost-server: Mattermost reports read only filesystem when in fact the file system is writable

[bennodepenno]

Per Mattermost guidelines, GitHub issues are for bug reports: http://www.mattermost.org/filing-issues/.

For troubleshooting see: http://forum.mattermost.org/.
For feature proposals see: http://www.mattermost.org/feature-requests/

If you've found a bug--something appears unintentional--please follow these steps:

1. Confirm you’re filing a new issue. Search existing tickets in Jira to ensure that the ticket does not already exist.
2. Confirm your issue does not involve security. Otherwise, please see our Responsible Disclosure Policy.
3. File a new issue using the format below. Mattermost will confirm steps to reproduce and file in Jira, or ask for more details if there is trouble reproducing it. If there's already an existing bug in Jira, it will be linked back to the GitHub issue so you can track when it gets fixed.

Summary

Pasting a image file in a channel results in message: Encountered an error creating the directory for the new file. The mattermost log file shows:

{"level":"error","ts":1596529841.3149927,"caller":"mlog/log.go:175","msg":"Encountered an error creating the directory for the new file.","path":"/api/v4/files","request_id":"p5jcmadhs7bfdyut35udqggqiw","ip_addr":"192.168.7.28","user_id":"5sdzax8pfty78bie37taum31ac","method":"POST","err_where":"WriteFile","http_code":500,"err_details":"directory=/srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary, err=mkdir /srv/data01/mattermost/20200804/teams: read-only file system"}
{"level":"error","ts":1596529841.3214974,"caller":"app/file.go:816","msg":"Unable to upload","path":"20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary/Image Pasted at 2020-8-4 10-30_preview.jpg","error":"WriteFile: Encountered an error creating the directory for the new file., directory=/srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary, err=mkdir /srv/data01/mattermost/20200804/teams: read-only file system"}
{"level":"error","ts":1596529841.3312554,"caller":"app/file.go:816","msg":"Unable to upload","path":"20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary/Image Pasted at 2020-8-4 10-30_thumb.jpg","error":"WriteFile: Encountered an error creating the directory for the new file., directory=/srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary, err=mkdir /srv/data01/mattermost/20200804/teams: read-only file system"}

The file system in question is not read only and can be written to by the mattermost user.

Steps to reproduce

Upgrade mattermost to 5.25.2-1 and copy/paste a screenshot image into any channel.

Expected behavior

Copy/paste of binaries should work as before

Observed behavior (that appears unintentional)

I've tested on two separate mattermost installations (version 5.25.1-1 and 5.25.2-1), in both cases pasting small pictures result in the error message. Using a browser in stead of an dedicated client results in the same behaviour. Both mattermost installations are based on Arch Linux OS. As user mattermost, I'm able to write to the file system reported read only by mattermost. In both instances, I've checked the free space available as well as the availability of free inodes. Both are plenty available. Please let me know if you need more information or how I can contribute solving this issue.

Possible fixes

I have not found any fix or workaround for this unfortunately.

[amyblais]

The error indicates that the user running Mattermost does not have the right permission to the directory. Can you please check the permission and ownership of the directory?

In the config.json you can check where the file settings directory is set to (e.g. /mattermost/data).

[bennodepenno]

Hi,

The permissions are correct:

$ ls -al /srv/data01/
total 32
drwxr-xr-x   5 root       root        4096 Aug  4 10:03 .
drwxr-xr-x   5 root       root        4096 Oct  5  2017 ..
drwxr-xr-x   2 root       root        4096 Aug  3 21:00 backup
drwx------   2 root       root       16384 Oct  5  2017 lost+found
drwxr-xr-x 181 mattermost mattermost  4096 Aug  4 10:54 mattermost

In our case /srv/data01/mattermost is configured in the config.json:

...
"FileSettings": {
        "EnableFileAttachments": true,
        "EnableMobileUpload": true,
        "EnableMobileDownload": true,
        "MaxFileSize": 104857600,
        "DriverName": "local",
        "Directory": "/srv/data01/mattermost/",
        "EnablePublicLink": false,
        "PublicLinkSalt": "hmn8djxr7iq87b8wij3pd3nfmqitxqsy",
        "InitialFont": "nunito-bold.ttf",
        "AmazonS3AccessKeyId": "",
        "AmazonS3SecretAccessKey": "",
        "AmazonS3Bucket": "",
        "AmazonS3Region": "",
        "AmazonS3Endpoint": "s3.amazonaws.com",
        "AmazonS3SSL": false,
        "AmazonS3SignV2": false,
        "AmazonS3SSE": false,
        "AmazonS3Trace": false
    },

The mattermost service runs under the mattermost user. As an experiment, I ran:
sudo -u mattermost touch /srv/data01/mattermost/myfile which succeeds.

[streamer45]

@bennodepenno

Can you try running:

sudo -u mattermost mkdir -p /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary

[bennodepenno]

Hi,

The sudo -u mattermost mkdir -p /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary works fine. The file-system is writable.

[root@mattermost ~]# ll /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary
total 8.0K
drwxr-xr-x 2 mattermost mattermost 4.0K Aug  5 15:42 .
drwxr-xr-x 3 mattermost mattermost 4.0K Aug  5 15:42 ..

[streamer45]

@bennodepenno

A couple of questions:

• Is this something that started happening after a while or it never worked?
• Which filesystem is your OS running on?

[bennodepenno]

It has worked before, I don't know exactly which version it stopped working.
The file system used is ext4: /dev/mapper/system-data01 on /srv/data01 type ext4 (rw,relatime,data=ordered)

[streamer45]

• Does this happen consistently now?
• How do you run the mattermost executable? (Through a service or manually?)

Also, can you try running (delete it first if exists):

sudo -u mattermost mkdir -m 0750 -p /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary

The last related change we had was more than 8 months ago and it just affected the permissions on creation.

[bennodepenno]

It happens consistently in any channel. We run mattermost as a systemd service.

[root@mattermost ~]# rmdir /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary
[root@mattermost ~]# ll /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary
ls: cannot access '/srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary': No such file or directory
[root@mattermost ~]#  sudo -u mattermost mkdir -m 0750 -p /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary
[root@mattermost ~]# ll /srv/data01/mattermost/20200804/teams/noteam/channels/y1gjjwapetfyfbiafnh4jhbrhw/users/5sdzax8pfty78bie37taum31ac/ayyg5epyfprb9p7ynpapw3sary
total 8.0K
drwxr-x--- 2 mattermost mattermost 4.0K Aug  5 16:33 .
drwxr-xr-x 3 mattermost mattermost 4.0K Aug  5 16:33 ..

This is the service file used to start mattermost:

[root@mattermost ~]# cat /usr/lib/systemd/system/mattermost.service
[Unit]
Description=Mattermost Chat Server
After=mysqld.service
After=postgresql.service

[Service]
Type=simple
User=mattermost
Group=mattermost
PermissionsStartOnly=true
WorkingDirectory=/usr/share/webapps/mattermost/
StateDirectory=mattermost
LogsDirectory=mattermost
ExecStart=/usr/bin/mattermost
Restart=on-failure
CapabilityBoundingSet=
NoNewPrivileges=True
#SecureBits=noroot-locked
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/etc/webapps/mattermost/config.json
PrivateTmp=true
PrivateDevices=true
PrivateUsers=true
ProtectHostname=true
ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
LockPersonality=true
#Not compatible with NodeJS
#MemoryDenyWriteExecute=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
SystemCallFilter=@system-service

[Install]
WantedBy=multi-user.target

[streamer45]

I'd probably try to add another entry of ReadWritePaths with your data directory (/srv/data01/mattermost/).

[bennodepenno]

Like so:

    "FileSettings": {
        "EnableFileAttachments": true,
        "EnableMobileUpload": true,
        "EnableMobileDownload": true,
        "MaxFileSize": 104857600,
        "DriverName": "local",
        "Directory": "/srv/data01/mattermost/",
        "ReadWritePaths": "/srv/data01/mattermost/",
        "EnablePublicLink": false,
        "PublicLinkSalt": "hmn8djxr7iq87b8wij3pd3nfmqitxqsy",
        "InitialFont": "nunito-bold.ttf",
        "AmazonS3AccessKeyId": "",
        "AmazonS3SecretAccessKey": "",
        "AmazonS3Bucket": "",
        "AmazonS3Region": "",
        "AmazonS3Endpoint": "s3.amazonaws.com",
        "AmazonS3SSL": false,
        "AmazonS3SignV2": false,
        "AmazonS3SSE": false,
        "AmazonS3Trace": false
    },

I've modified the config as shown above, restarted the mattermost service and tried pasting a screenshot in a channel. Unfortunately, the results are the same, read-only file system.

[streamer45]

Oh, I meant in the systemd service file, right below ReadWritePaths=/etc/webapps/mattermost/config.json.
This seems unrelated to mattermost.

[bennodepenno]

After the restart of the service, the ReadWritePaths parameter is removed from the config.json file. Cannot make the config persistent.

[bennodepenno]

Ah, sorry. I misunderstood. Will try it now.

[bennodepenno]

That did the trick!. I've copied the /usr/lib/systemd/system/mattermost.service file to /etc/systemd/system, added the line:
ReadWritePaths=/srv/data01/mattermost/ after ReadWritePaths=/etc/webapps/mattermost/config.json and reloaded systemd. After restarting the mattermost service, pasting binaries works! Thank you for your help.