mattermost-server 🚀 - Email notification still sent while disabled in UI

@mthld Do you see any Mattermost log errors when the issue occurs?

amyblais on 16 Oct 2019

Hi @amyblais,

Thank you for reaching.
Here they are:

2019-10-17T12:30:09.484+0200    debug   mlog/sugar.go:15    Fetching all standup channels   {"plugin_id": "standup-raven", "Fetching all standup channels": null}
2019-10-17T12:30:09.485+0200    debug   mlog/sugar.go:15    Found 2 standup channels    {"plugin_id": "standup-raven", "Found 2 standup channels": null}
2019-10-17T12:30:09.485+0200    debug   mlog/sugar.go:15    Filtering channels for sending notifications    {"plugin_id": "standup-raven", "Filtering channels for sending notifications": null}
2019-10-17T12:30:09.485+0200    debug   mlog/sugar.go:15    Processing channel: jgrdic1ex7rnxrihwdi78hisjw  {"plugin_id": "standup-raven", "Processing channel: jgrdic1ex7rnxrihwdi78hisjw": null}
2019-10-17T12:30:09.485+0200    debug   mlog/sugar.go:15    Fetching notification status for channel: jgrdic1ex7rnxrihwdi78hisjw    {"plugin_id": "standup-raven", "Fetching notification status for channel: jgrdic1ex7rnxrihwdi78hisjw": null}
2019-10-17T12:30:09.486+0200    debug   mlog/sugar.go:15    notification status for channel: jgrdic1ex7rnxrihwdi78hisjw, &{true true false} {"plugin_id": "standup-raven", "notification status for channel: jgrdic1ex7rnxrihwdi78hisjw, &{true true false}": null}
2019-10-17T12:30:09.486+0200    debug   mlog/sugar.go:15    Fetching standup config for channel: jgrdic1ex7rnxrihwdi78hisjw {"plugin_id": "standup-raven", "Fetching standup config for channel: jgrdic1ex7rnxrihwdi78hisjw": null}
2019-10-17T12:30:09.487+0200    debug   mlog/sugar.go:15    Standup config for channel: jgrdic1ex7rnxrihwdi78hisjw, &{jgrdic1ex7rnxrihwdi78hisjw 2019-10-17 10:00:00 +0200 CEST 2019-10-17 12:30:00 +0200 CEST user_aggregated [rjq8m341y3btie17ju4ha3w3rw 48j3p6mr87gajpru64mscd49hh] [yesterday today tomorrow] true} {"plugin_id": "standup-raven", "Standup config for channel: jgrdic1ex7rnxrihwdi78hisjw, &{jgrdic1ex7rnxrihwdi78hisjw 2019-10-17 10:00:00 +0200 CEST 2019-10-17 12:30:00 +0200 CEST user_aggregated [rjq8m341y3btie17ju4ha3w3rw 48j3p6mr87gajpru64mscd49hh] [yesterday today tomorrow] true}": null}
2019-10-17T12:30:09.487+0200    debug   mlog/sugar.go:15    Channel [jgrdic1ex7rnxrihwdi78hisjw] needs standup report   {"plugin_id": "standup-raven", "Channel [jgrdic1ex7rnxrihwdi78hisjw] needs standup report": null}
2019-10-17T12:30:09.488+0200    debug   mlog/sugar.go:15    Processing channel: tayh4xcehjrhjg5pnbewj1sa9w  {"plugin_id": "standup-raven", "Processing channel: tayh4xcehjrhjg5pnbewj1sa9w": null}
2019-10-17T12:30:09.488+0200    debug   mlog/sugar.go:15    Fetching notification status for channel: tayh4xcehjrhjg5pnbewj1sa9w    {"plugin_id": "standup-raven", "Fetching notification status for channel: tayh4xcehjrhjg5pnbewj1sa9w": null}
2019-10-17T12:30:09.489+0200    debug   mlog/sugar.go:15    Fetching standup config for channel: tayh4xcehjrhjg5pnbewj1sa9w {"plugin_id": "standup-raven", "Fetching standup config for channel: tayh4xcehjrhjg5pnbewj1sa9w": null}
2019-10-17T12:30:09.490+0200    debug   mlog/sugar.go:15    Standup config for channel: tayh4xcehjrhjg5pnbewj1sa9w, &{tayh4xcehjrhjg5pnbewj1sa9w 2019-10-17 08:00:00 +0200 CEST 2019-10-17 13:00:00 +0200 CEST user_aggregated [] [yesterday today tomorrow] false} {"plugin_id": "standup-raven", "Standup config for channel: tayh4xcehjrhjg5pnbewj1sa9w, &{tayh4xcehjrhjg5pnbewj1sa9w 2019-10-17 08:00:00 +0200 CEST 2019-10-17 13:00:00 +0200 CEST user_aggregated [] [yesterday today tomorrow] false}": null}
2019-10-17T12:30:09.490+0200    debug   mlog/sugar.go:15    Notifications filtered: open: 0, close: 0, reports: 1   {"plugin_id": "standup-raven", "Notifications filtered: open: 0, close: 0, reports: 1": null}
2019-10-17T12:30:09.491+0200    info    mlog/sugar.go:19    Sending standup report for channel: jgrdic1ex7rnxrihwdi78hisjw time: 20191017   {"plugin_id": "standup-raven", "Sending standup report for channel: jgrdic1ex7rnxrihwdi78hisjw time: 20191017": null}
2019-10-17T12:30:09.491+0200    debug   mlog/sugar.go:15    Fetching standup config for channel: jgrdic1ex7rnxrihwdi78hisjw {"plugin_id": "standup-raven", "Fetching standup config for channel: jgrdic1ex7rnxrihwdi78hisjw": null}
2019-10-17T12:30:09.492+0200    debug   mlog/sugar.go:15    Standup config for channel: jgrdic1ex7rnxrihwdi78hisjw, &{jgrdic1ex7rnxrihwdi78hisjw 2019-10-17 10:00:00 +0200 CEST 2019-10-17 12:30:00 +0200 CEST user_aggregated [rjq8m341y3btie17ju4ha3w3rw 48j3p6mr87gajpru64mscd49hh] [yesterday today tomorrow] true} {"plugin_id": "standup-raven", "Standup config for channel: jgrdic1ex7rnxrihwdi78hisjw, &{jgrdic1ex7rnxrihwdi78hisjw 2019-10-17 10:00:00 +0200 CEST 2019-10-17 12:30:00 +0200 CEST user_aggregated [rjq8m341y3btie17ju4ha3w3rw 48j3p6mr87gajpru64mscd49hh] [yesterday today tomorrow] true}": null}
2019-10-17T12:30:09.493+0200    info    mlog/sugar.go:19    Could not fetch standup for user: 48j3p6mr87gajpru64mscd49hh    {"plugin_id": "standup-raven", "Could not fetch standup for user: 48j3p6mr87gajpru64mscd49hh": null}
2019-10-17T12:30:09.496+0200    debug   mlog/sugar.go:15    Generating user aggregated standup report for channel: jgrdic1ex7rnxrihwdi78hisjw   {"plugin_id": "standup-raven", "Generating user aggregated standup report for channel: jgrdic1ex7rnxrihwdi78hisjw": null}
2019-10-17T12:30:09.569+0200    debug   app/post_metadata.go:219    Failed to get dimensions of an image in a post  {"post_id": "r1fo1p5futypfxdxeb5uwyenko", "image_url": "/api/v4/users/rjq8m341y3btie17ju4ha3w3rw/image", "error": "Get https://chat.clic2000.fr/api/v4/users/rjq8m341y3btie17ju4ha3w3rw/image: address forbidden, you may need to set AllowedUntrustedInternalConnections to allow an integration access to your internal network"}
2019-10-17T12:30:09.623+0200    debug   mlog/sugar.go:15    Fetching notification status for channel: jgrdic1ex7rnxrihwdi78hisjw    {"plugin_id": "standup-raven", "Fetching notification status for channel: jgrdic1ex7rnxrihwdi78hisjw": null}
2019-10-17T12:30:09.627+0200    debug   mlog/sugar.go:15    notification status for channel: jgrdic1ex7rnxrihwdi78hisjw, &{true true false} {"plugin_id": "standup-raven", "notification status for channel: jgrdic1ex7rnxrihwdi78hisjw, &{true true false}": null}
2019-10-17T12:30:10.793+0200    debug   web/handlers.go:58  POST - /api/v4/notifications/ack
2019-10-17T12:30:10.931+0200    debug   web/handlers.go:58  GET - /
2019-10-17T12:30:15.639+0200    debug   app/email_batching.go:110   Email batching job ran. 1 user(s) still have notifications pending.
2019-10-17T12:30:15.919+0200    debug   mailservice/mail.go:243 sending mail to [email protected] with subject of '[Mattermost par Clic2000] New Notification for October 17, 2019'
2019-10-17T12:30:25.910+0200    debug   web/handlers.go:58  GET - /

mthld on 17 Oct 2019

Hello,

It still happens on a daily basis, can someone reproduce this? Apart being annoying to users it seems to be a major problem regarding GDPR compliance...

mthld on 26 Oct 2019

@mthld Our QAs have been trying to test this but haven't been able to so far. Following your report, they have embedded a mention in an incoming webhook, and triggered it under different notifications settings (including Never, with email batched, as appears to be the scenario in the report), but they receive or don't receive emails as expected.

We are wondering about a couple other aspects of the report:

The notification screenshot is from a webhook but you also mention a plugin. Are there unexpected notifications coming from multiple different source triggers or just one webhook?
And/or is there a sense that the existence of the plugin is somehow affecting the behavior of notifications triggered by the webhook?
Are there any configurations in the plugin/webhooks that relate to sending email notifications?
Report describes multiple people receiving unexpected notifications; the example seems to show a username that matches the reporter. Is each user receiving a notification only when it contains their own username, or also when the bot post is triggered by other users? (In other words, is the notification that bypasses the Never setting behaving as some sort of channel-wide notification, or is it specific to when it includes the user's username?)

amyblais on 29 Oct 2019

Hi @amyblais,

Sorry for the delay, here are my answers:

We receive the emails from both the webhook and the plugin
I can deactivate the plugin and I'll still get the emails from the webhook
There is nothing related to this
I personally do receive the emails when my username and/or first name mentioned in the message, see the following screenshot (Your non-case sensitive username "mathilde")

Screenshot 2019-11-14 at 11 51 55

mthld on 14 Nov 2019

FYI we even receive email notification when we are marked as « away » and someone is writing to us in private chat.

Something is definitely broken on Mattermost end, there is no plugin or webhook involved in this.

mthld on 14 Nov 2019

Could you help with more information about your environment from here and share your config settings? Do you have log errors from the time this issue occurred https://github.com/mattermost/mattermost-server/issues/12704#issuecomment-554077595?

amyblais on 15 Nov 2019

Hi Amy,

There is 0 log errors at the time this issue occurred (approximatively 2019-11-14T21:35:00.000+0100).

Here are the informations about our environment:

OS: Debian GNU/Linux 10 "Buster"
Kernel: 4.19.0-6-amd64
Mattermost Version: 5.16.2
Database Schema Version: 5.16.0
DB server: Postgresql 11.5-1+deb10u1

Here is our config file (minus some redacted private infos):

$ cat config/config.json 
{
    "ServiceSettings": {
        "SiteURL": "https://chat.clic2000.fr",
        "WebsocketURL": "",
        "LicenseFileLocation": "",
        "ListenAddress": ":8065",
        "ConnectionSecurity": "",
        "TLSCertFile": "",
        "TLSKeyFile": "",
        "TLSMinVer": "1.2",
        "TLSStrictTransport": false,
        "TLSStrictTransportMaxAge": 63072000,
        "TLSOverwriteCiphers": [],
        "UseLetsEncrypt": false,
        "LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
        "Forward80To443": false,
        "TrustedProxyIPHeader": [
            "X-Forwarded-For",
            "X-Real-IP"
        ],
        "ReadTimeout": 300,
        "WriteTimeout": 300,
        "MaximumLoginAttempts": 10,
        "GoroutineHealthThreshold": -1,
        "GoogleDeveloperKey": "",
        "EnableOAuthServiceProvider": false,
        "EnableIncomingWebhooks": true,
        "EnableOutgoingWebhooks": true,
        "EnableCommands": true,
        "EnableOnlyAdminIntegrations": true,
        "EnablePostUsernameOverride": true,
        "EnablePostIconOverride": true,
        "EnableLinkPreviews": false,
        "EnableTesting": false,
        "EnableDeveloper": false,
        "EnableSecurityFixAlert": true,
        "EnableInsecureOutgoingConnections": false,
        "AllowedUntrustedInternalConnections": "",
        "EnableMultifactorAuthentication": false,
        "EnforceMultifactorAuthentication": false,
        "EnableUserAccessTokens": false,
        "AllowCorsFrom": "",
        "CorsExposedHeaders": "",
        "CorsAllowCredentials": false,
        "CorsDebug": false,
        "AllowCookiesForSubdomains": false,
        "SessionLengthWebInDays": 30,
        "SessionLengthMobileInDays": 30,
        "SessionLengthSSOInDays": 30,
        "SessionCacheInMinutes": 10,
        "SessionIdleTimeoutInMinutes": 0,
        "WebsocketSecurePort": 443,
        "WebsocketPort": 80,
        "WebserverMode": "gzip",
        "EnableCustomEmoji": true,
        "EnableEmojiPicker": true,
        "EnableGifPicker": false,
        "GfycatApiKey": "REDACTED",
        "GfycatApiSecret": "REDACTED",
        "RestrictCustomEmojiCreation": "all",
        "RestrictPostDelete": "all",
        "AllowEditPost": "always",
        "PostEditTimeLimit": -1,
        "TimeBetweenUserTypingUpdatesMilliseconds": 5000,
        "EnablePostSearch": true,
        "MinimumHashtagLength": 3,
        "EnableUserTypingMessages": true,
        "EnableChannelViewedMessages": true,
        "EnableUserStatuses": true,
        "ExperimentalEnableAuthenticationTransfer": true,
        "ClusterLogTimeoutMilliseconds": 2000,
        "CloseUnusedDirectMessages": false,
        "EnablePreviewFeatures": true,
        "EnableTutorial": true,
        "ExperimentalEnableDefaultChannelLeaveJoinMessages": true,
        "ExperimentalGroupUnreadChannels": "disabled",
        "ExperimentalChannelOrganization": true,
        "ImageProxyType": "",
        "ImageProxyURL": "",
        "ImageProxyOptions": "",
        "EnableAPITeamDeletion": false,
        "ExperimentalEnableHardenedMode": false,
        "DisableLegacyMFA": false,
        "ExperimentalStrictCSRFEnforcement": false,
        "EnableEmailInvitations": true,
        "ExperimentalLdapGroupSync": false,
        "DisableBotsWhenOwnerIsDeactivated": true,
        "EnableBotAccountCreation": true,
        "EnableSVGs": true
    },
    "TeamSettings": {
        "SiteName": "Mattermost par Clic2000",
        "MaxUsersPerTeam": 50,
        "EnableTeamCreation": true,
        "EnableUserCreation": true,
        "EnableOpenServer": false,
        "EnableUserDeactivation": true,
        "RestrictCreationToDomains": "",
        "EnableCustomBrand": true,
        "CustomBrandText": "",
        "CustomDescriptionText": "",
        "RestrictDirectMessage": "team",
        "RestrictTeamInvite": "all",
        "RestrictPublicChannelManagement": "all",
        "RestrictPrivateChannelManagement": "all",
        "RestrictPublicChannelCreation": "all",
        "RestrictPrivateChannelCreation": "all",
        "RestrictPublicChannelDeletion": "all",
        "RestrictPrivateChannelDeletion": "all",
        "RestrictPrivateChannelManageMembers": "all",
        "EnableXToLeaveChannelsFromLHS": false,
        "UserStatusAwayTimeout": 300,
        "MaxChannelsPerTeam": 2000,
        "MaxNotificationsPerChannel": 1000000,
        "EnableConfirmNotificationsToChannel": true,
        "TeammateNameDisplay": "username",
        "ExperimentalViewArchivedChannels": false,
        "ExperimentalEnableAutomaticReplies": false,
        "ExperimentalHideTownSquareinLHS": false,
        "ExperimentalTownSquareIsReadOnly": false,
        "ExperimentalPrimaryTeam": "",
        "ExperimentalDefaultChannels": []
    },
    "ClientRequirements": {
        "AndroidLatestVersion": "",
        "AndroidMinVersion": "",
        "DesktopLatestVersion": "",
        "DesktopMinVersion": "",
        "IosLatestVersion": "",
        "IosMinVersion": ""
    },
    "SqlSettings": {
        "DriverName": "postgres",
        "DataSource": "postgres://mattermost:REDACTED@localhost:5432/mattermost?sslmode=disable\u0026connect_timeout=10",
        "DataSourceReplicas": [],
        "DataSourceSearchReplicas": [],
        "MaxIdleConns": 20,
        "ConnMaxLifetimeMilliseconds": 3600000,
        "MaxOpenConns": 300,
        "Trace": false,
        "AtRestEncryptKey": "REDACTED",
        "QueryTimeout": 30
    },
    "LogSettings": {
        "EnableConsole": false,
        "ConsoleLevel": "INFO",
        "ConsoleJson": false,
        "EnableFile": true,
        "FileLevel": "ERROR",
        "FileJson": false,
        "FileLocation": "",
        "EnableWebhookDebugging": true,
        "EnableDiagnostics": false
    },
    "NotificationLogSettings": {
        "EnableConsole": true,
        "ConsoleLevel": "DEBUG",
        "ConsoleJson": true,
        "EnableFile": true,
        "FileLevel": "INFO",
        "FileJson": true,
        "FileLocation": ""
    },
    "PasswordSettings": {
        "MinimumLength": 16,
        "Lowercase": true,
        "Number": true,
        "Uppercase": true,
        "Symbol": true
    },
    "FileSettings": {
        "EnableFileAttachments": true,
        "EnableMobileUpload": true,
        "EnableMobileDownload": true,
        "MaxFileSize": 52428800,
        "DriverName": "local",
        "Directory": "./data/",
        "EnablePublicLink": false,
        "PublicLinkSalt": "REDACTED",
        "InitialFont": "nunito-bold.ttf",
        "AmazonS3AccessKeyId": "",
        "AmazonS3SecretAccessKey": "",
        "AmazonS3Bucket": "",
        "AmazonS3Region": "",
        "AmazonS3Endpoint": "s3.amazonaws.com",
        "AmazonS3SSL": true,
        "AmazonS3SignV2": false,
        "AmazonS3SSE": false,
        "AmazonS3Trace": false
    },
    "EmailSettings": {
        "EnableSignUpWithEmail": true,
        "EnableSignInWithEmail": true,
        "EnableSignInWithUsername": false,
        "SendEmailNotifications": true,
        "UseChannelInEmailNotifications": false,
        "RequireEmailVerification": true,
        "FeedbackName": "REDACTED",
        "FeedbackEmail": "REDACTED",
        "ReplyToAddress": "REDACTED",
        "FeedbackOrganization": "",
        "EnableSMTPAuth": true,
        "SMTPUsername": "REDACTED",
        "SMTPPassword": "REDACTED",
        "SMTPServer": "REDACTED",
        "SMTPPort": "465",
        "ConnectionSecurity": "TLS",
        "SendPushNotifications": true,
        "PushNotificationServer": "https://push-test.mattermost.com",
        "PushNotificationContents": "generic",
        "EnableEmailBatching": true,
        "EmailBatchingBufferSize": 256,
        "EmailBatchingInterval": 30,
        "EnablePreviewModeBanner": true,
        "SkipServerCertificateVerification": false,
        "EmailNotificationContentsType": "full",
        "LoginButtonColor": "",
        "LoginButtonBorderColor": "",
        "LoginButtonTextColor": ""
    },
    "RateLimitSettings": {
        "Enable": false,
        "PerSec": 10,
        "MaxBurst": 100,
        "MemoryStoreSize": 10000,
        "VaryByRemoteAddr": true,
        "VaryByUser": false,
        "VaryByHeader": ""
    },
    "PrivacySettings": {
        "ShowEmailAddress": true,
        "ShowFullName": true
    },
    "SupportSettings": {
        "TermsOfServiceLink": "https://about.mattermost.com/default-terms/",
        "PrivacyPolicyLink": "https://about.mattermost.com/default-privacy-policy/",
        "AboutLink": "https://about.mattermost.com/default-about/",
        "HelpLink": "https://about.mattermost.com/default-help/",
        "ReportAProblemLink": "https://about.mattermost.com/default-report-a-problem/",
        "SupportEmail": "[email protected]",
        "CustomTermsOfServiceEnabled": false,
        "CustomTermsOfServiceReAcceptancePeriod": 365
    },
    "AnnouncementSettings": {
        "EnableBanner": false,
        "BannerText": "",
        "BannerColor": "#f2a93b",
        "BannerTextColor": "#333333",
        "AllowBannerDismissal": true
    },
    "ThemeSettings": {
        "EnableThemeSelection": true,
        "DefaultTheme": "default",
        "AllowCustomThemes": true,
        "AllowedThemes": []
    },
    "GitLabSettings": {
        "Enable": false,
        "Secret": "",
        "Id": "",
        "Scope": "",
        "AuthEndpoint": "",
        "TokenEndpoint": "",
        "UserApiEndpoint": ""
    },
    "GoogleSettings": {
        "Enable": false,
        "Secret": "",
        "Id": "",
        "Scope": "profile email",
        "AuthEndpoint": "https://accounts.google.com/o/oauth2/v2/auth",
        "TokenEndpoint": "https://www.googleapis.com/oauth2/v4/token",
        "UserApiEndpoint": "https://www.googleapis.com/plus/v1/people/me"
    },
    "Office365Settings": {
        "Enable": false,
        "Secret": "",
        "Id": "",
        "Scope": "User.Read",
        "AuthEndpoint": "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
        "TokenEndpoint": "https://login.microsoftonline.com/common/oauth2/v2.0/token",
        "UserApiEndpoint": "https://graph.microsoft.com/v1.0/me"
    },
    "LdapSettings": {
        "Enable": false,
        "EnableSync": false,
        "LdapServer": "",
        "LdapPort": 389,
        "ConnectionSecurity": "",
        "BaseDN": "",
        "BindUsername": "",
        "BindPassword": "",
        "UserFilter": "",
        "GroupFilter": "",
        "GroupDisplayNameAttribute": "",
        "GroupIdAttribute": "",
        "FirstNameAttribute": "",
        "LastNameAttribute": "",
        "EmailAttribute": "",
        "UsernameAttribute": "",
        "NicknameAttribute": "",
        "IdAttribute": "",
        "PositionAttribute": "",
        "LoginIdAttribute": "",
        "SyncIntervalMinutes": 60,
        "SkipCertificateVerification": false,
        "QueryTimeout": 60,
        "MaxPageSize": 0,
        "LoginFieldName": "",
        "LoginButtonColor": "",
        "LoginButtonBorderColor": "",
        "LoginButtonTextColor": "",
        "Trace": false
    },
    "ComplianceSettings": {
        "Enable": false,
        "Directory": "./data/",
        "EnableDaily": false
    },
    "LocalizationSettings": {
        "DefaultServerLocale": "en",
        "DefaultClientLocale": "en",
        "AvailableLocales": ""
    },
    "SamlSettings": {
        "Enable": false,
        "EnableSyncWithLdap": false,
        "EnableSyncWithLdapIncludeAuth": false,
        "Verify": true,
        "Encrypt": true,
        "SignRequest": false,
        "IdpUrl": "",
        "IdpDescriptorUrl": "",
        "AssertionConsumerServiceURL": "",
        "ScopingIDPProviderId": "",
        "ScopingIDPName": "",
        "IdpCertificateFile": "",
        "PublicCertificateFile": "",
        "PrivateKeyFile": "",
        "IdAttribute": "",
        "FirstNameAttribute": "",
        "LastNameAttribute": "",
        "EmailAttribute": "",
        "UsernameAttribute": "",
        "NicknameAttribute": "",
        "LocaleAttribute": "",
        "PositionAttribute": "",
        "LoginButtonText": "With SAML",
        "LoginButtonColor": "",
        "LoginButtonBorderColor": "",
        "LoginButtonTextColor": ""
    },
    "NativeAppSettings": {
        "AppDownloadLink": "https://about.mattermost.com/downloads/",
        "AndroidAppDownloadLink": "https://about.mattermost.com/mattermost-android-app/",
        "IosAppDownloadLink": "https://about.mattermost.com/mattermost-ios-app/"
    },
    "ClusterSettings": {
        "Enable": false,
        "ClusterName": "",
        "OverrideHostname": "",
        "NetworkInterface": "",
        "BindAddress": "",
        "AdvertiseAddress": "",
        "UseIpAddress": true,
        "UseExperimentalGossip": false,
        "ReadOnlyConfig": true,
        "GossipPort": 8074,
        "StreamingPort": 8075,
        "MaxIdleConns": 100,
        "MaxIdleConnsPerHost": 128,
        "IdleConnTimeoutMilliseconds": 90000
    },
    "MetricsSettings": {
        "Enable": false,
        "BlockProfileRate": 0,
        "ListenAddress": ":8067"
    },
    "ExperimentalSettings": {
        "ClientSideCertEnable": false,
        "ClientSideCertCheck": "secondary",
        "EnableClickToReply": false,
        "LinkMetadataTimeoutMilliseconds": 5000,
        "RestrictSystemAdmin": false
    },
    "AnalyticsSettings": {
        "MaxUsersForStatistics": 2500
    },
    "ElasticsearchSettings": {
        "ConnectionUrl": "http://dockerhost:9200",
        "Username": "elastic",
        "Password": "changeme",
        "EnableIndexing": false,
        "EnableSearching": false,
        "EnableAutocomplete": false,
        "Sniff": true,
        "PostIndexReplicas": 1,
        "PostIndexShards": 1,
        "ChannelIndexReplicas": 1,
        "ChannelIndexShards": 1,
        "UserIndexReplicas": 1,
        "UserIndexShards": 1,
        "AggregatePostsAfterDays": 365,
        "PostsAggregatorJobStartTime": "03:00",
        "IndexPrefix": "",
        "LiveIndexingBatchSize": 1,
        "BulkIndexingTimeWindowSeconds": 3600,
        "RequestTimeoutSeconds": 30,
        "SkipTLSVerification": false,
        "Trace": ""
    },
    "DataRetentionSettings": {
        "EnableMessageDeletion": false,
        "EnableFileDeletion": false,
        "MessageRetentionDays": 365,
        "FileRetentionDays": 365,
        "DeletionJobStartTime": "02:00"
    },
    "MessageExportSettings": {
        "EnableExport": false,
        "ExportFormat": "actiance",
        "DailyRunTime": "01:00",
        "ExportFromTimestamp": 0,
        "BatchSize": 10000,
        "GlobalRelaySettings": {
            "CustomerType": "A9",
            "SmtpUsername": "",
            "SmtpPassword": "",
            "EmailAddress": ""
        }
    },
    "JobSettings": {
        "RunJobs": true,
        "RunScheduler": true
    },
    "PluginSettings": {
        "Enable": true,
        "EnableUploads": true,
        "AllowInsecureDownloadUrl": false,
        "EnableHealthCheck": true,
        "Directory": "./plugins",
        "ClientDirectory": "./client/plugins",
        "Plugins": {
            "com.mattermost.nps": {
                "enablesurvey": false
            },
            "github": {
                "encryptionkey": "REDACTED",
                "githuboauthclientid": "REDACTED",
                "githuboauthclientsecret": "REDACTED",
                "githuborg": "clic2000",
                "webhooksecret": "REDACTED"
            },
            "rssfeed": {
                "heartbeat": "10",
                "showdescription": true,
                "username": "admin"
            },
            "standup-raven": {
                "botusername": "standup-raven",
                "timezone": "Europe/Paris",
                "workweekend": "5",
                "workweekstart": "1"
            }
        },
        "PluginStates": {
            "com.mattermost.custom-attributes": {
                "Enable": true
            },
            "com.mattermost.nps": {
                "Enable": false
            },
            "github": {
                "Enable": false
            },
            "rssfeed": {
                "Enable": false
            },
            "standup-raven": {
                "Enable": true
            }
        },
        "EnableMarketplace": true,
        "MarketplaceUrl": "https://api.integrations.mattermost.com"
    },
    "DisplaySettings": {
        "CustomUrlSchemes": [],
        "ExperimentalTimezone": false
    },
    "GuestAccountsSettings": {
        "Enable": false,
        "AllowEmailAccounts": true,
        "EnforceMultifactorAuthentication": false,
        "RestrictCreationToDomains": ""
    },
    "ImageProxySettings": {
        "Enable": false,
        "ImageProxyType": "local",
        "RemoteImageProxyURL": "",
        "RemoteImageProxyOptions": ""
    }
}

Thank you for helping us with this.
Have a good day,

mthld on 15 Nov 2019

Hi there, this seems similar to #11434 (Point 1 in OP, and my point in my followup comment), where email notifications are still sent even if user has them set to never. @amyblais Indicated this was resolved in v5.14, however we are still seeing this issue while on v5.14.2.

S1M0N on 18 Nov 2019

👍1

Ticket was opened for this here: https://mattermost.atlassian.net/browse/MM-20260. Devs said the issue might have something to do with email batching.

amyblais on 19 Nov 2019

🎉1

Thanks for the update @amyblais. We did have batching enabled on our server, so I've just disabled it and will test on our end and update with results.

S1M0N on 19 Nov 2019

Following up on the above regarding email batching, we disabled batching and issue still presents.

S1M0N on 19 Nov 2019

👍1

@S1M0N, @mthld, I'm the dev currently working on this issue and I think I have may found the root cause. Looks like it is related to a user-channel notification preference set on the server, which is not exposed through the UI. This setting if corrupted will result in the behaviour you guys are seeing.

Can you both confirm whether this issue is happening for _everyone_ or just a few individuals?

Do you mind running this query and sharing the result for user/channel combination that's seeing the issue?

SELECT ChannelId, UserId, NotifyProps
FROM ChannelMembers
WHERE UserId IN (SELECT Id FROM Users WHERE Username='<USERNAME>')
AND ChannelId IN (SELECT Id FROM Channels WHERE NAME='<CHANNEL_NAME>')

If the email field in the returned JSON has any value other than default or false that user in that channel will get email notifications regardless of what their account email notification setting is.

{
    "desktop": "mention",
    "email": "default", // expected value for this field.
    "ignore_channel_mentions": "on",
    "mark_unread": "all",
    "push": "all"
}

To fix the issue set the email field to default through SQL and update any setting in the Notification Preference for user in that channel for the server to update its internal cache.

Once that is done, run the query again to make sure all is well. Try triggering the email notification, the user should _not_ get any email notification for that channel if the account is set to never receive email notifications.

I've confirmed that this is an issue on 5.14 as well as on our latest version. Will be putting in a fix that honours account email notification setting first and then looks at channel email notification setting.

ali-farooq0 on 21 Nov 2019

Hi @ali-farooq0

Here are my answers:

Can you both confirm whether this issue is happening for _everyone_ or just a few individuals?

It happens to all team members.

Do you mind running this query and sharing the result for user/channel combination that's seeing the issue?
SELECT ChannelId, UserId, NotifyProps
FROM ChannelMembers
WHERE UserId IN (SELECT Id FROM Users WHERE Username='<USERNAME>')
AND ChannelId IN (SELECT Id FROM Channels WHERE NAME='<CHANNEL_NAME>')
If the email field in the returned JSON has any value other than default or false that user in that channel will get email notifications regardless of what their account email notification setting is.
{
    "desktop": "mention",
    "email": "default", // expected value for this field.
    "ignore_channel_mentions": "on",
    "mark_unread": "all",
    "push": "all"
}

It doesn't seems to be the case:

mattermost=# SELECT ChannelId, UserId, NotifyProps 
FROM ChannelMembers
WHERE UserId IN (SELECT Id FROM Users WHERE Username='myrdin')
AND ChannelId IN (SELECT Id FROM Channels WHERE NAME='dev');
         channelid          |           userid           |                                 notifyprops                                  
----------------------------+----------------------------+------------------------------------------------------------------------------
 wswdwqj3n7gpzfrdr79dc18k6w | 48j3p6mr87gajpru64mscd49hh | {"desktop":"default","email":"default","mark_unread":"all","push":"default"}
(1 row)

Let's try another combination:

mattermost=# SELECT ChannelId, UserId, NotifyProps
FROM ChannelMembers
WHERE UserId IN (SELECT Id FROM Users WHERE Username='mathilde')
AND ChannelId IN (SELECT Id FROM Channels WHERE NAME='standup');
         channelid          |           userid           |                                                   notifyprops                                                    
----------------------------+----------------------------+------------------------------------------------------------------------------------------------------------------
 jgrdic1ex7rnxrihwdi78hisjw | rjq8m341y3btie17ju4ha3w3rw | {"desktop":"default","email":"default","ignore_channel_mentions":"default","mark_unread":"all","push":"default"}
(1 row)

Thanks for your help.

mthld on 21 Nov 2019

Appreciate the quick response @mthld. Well that's unfortunate ;p Do you mind running the following to confirm their email settings as well, thanks!

(SELECT Id, NotifyProps, json_extract(NotifyProps, '$.email') FROM Users WHERE Username='myrdin')
(SELECT Id, NotifyProps, json_extract(NotifyProps, '$.email') FROM Users WHERE Username='mathilde')

ali-farooq0 on 21 Nov 2019

Dear @ali-farooq0 ,

Appreciate the quick response @mthld. Well that's unfortunate ;p Do you mind running the following to confirm their email settings as well, thanks!
(SELECT Id, NotifyProps, json_extract(NotifyProps, '$.email') FROM Users WHERE Username='myrdin')
(SELECT Id, NotifyProps, json_extract(NotifyProps, '$.email') FROM Users WHERE Username='mathilde')

It seems like json_extract is a MySQL function, can you tell me how to run this for Postgresql?
Thanks,

mthld on 21 Nov 2019

@mthld
Whoops, this should do it:

(SELECT Id, NotifyProps FROM Users WHERE Username='myrdin')
(SELECT Id, NotifyProps FROM Users WHERE Username='mathilde')

ali-farooq0 on 21 Nov 2019

@ali-farooq0 thanks!

@mthld
Whoops, this should do it:

(SELECT Id, NotifyProps FROM Users WHERE Username='myrdin')
(SELECT Id, NotifyProps FROM Users WHERE Username='mathilde')

Here we are:

{
  "auto_responder_active":"false",
  "auto_responder_message":"Hello, I am out of office and unable to respond to messages.",
  "channel":"true",
  "comments":"never",
  "desktop":"all",
  "desktop_sound":"true",
  "email":"true",
  "first_name":"false",
  "mention_keys":"myrdin",
  "push":"all",
  "push_status":"away"
}

and...

{
  "channel":"true",
  "comments":"never",
  "desktop":"mention",
  "email":"true",
  "first_name":"false",
  "mention_keys":"mathilde",
  "push":"all",
  "push_status":"away"
}

🙏🏻

mthld on 21 Nov 2019

Thanks @mthld! looks like email is set to true which means that the account email notification is not set to never so it _is_ expected for these users to get email notifications. Can you confirm what's shown in the notification section in the UI below for them?

ali-farooq0 on 21 Nov 2019

Hi @ali-farooq0,

Sorry for the delay, so far this is all users, but we have a low usage on this server at then moment and I haven't been able to test as much as I'd like. Also, some of our users do have email notifications turned on so they are expect them and have no complaints.

I attempted the first query you specified above but received no data back from the DB, however on the second query I received the below.

(SELECT Id, NotifyProps FROM Users WHERE Username='simon');

{
  "auto_responder_active":"false",
  "auto_responder_message":"Hello, I am out of office and unable to respond to messages.",
  "channel":"true",
  "comments":"never",
  "desktop":"mention",
  "desktop_sound":"true",
  "email":"true",
  "first_name":"true",
  "mention_keys":"simon",
  "push":"mention",
  "push_status":"online"
}

I checked my settings and see the following:

S1M0N on 21 Nov 2019

Hey @ali-farooq0

Thanks @mthld! looks like email is set to true which means that the account email notification is not set to never so it _is_ expected for these users to get email notifications. Can you confirm what's shown in the notification section in the UI below for them?

Here it is:
Screenshot 2019-11-21 at 19 03 13

@myrdinm one's will follow soon.

mthld on 21 Nov 2019

Here is mine :

Screenshot 2019-11-21 at 19 09 08

myrdn on 21 Nov 2019

❤1

Thanks guys! Looks like @myrdinm has it enabled so that is expected. However for @mthld and @S1M0N the UI is showing Never which is not in sync with the DB.

@mthld and @S1M0N, do you guys mind just retriggering the email setting by

Set it to Immediately, click Save.
Set it to Never, click Save
Query the DB again using (SELECT Id, NotifyProps FROM Users WHERE Username='myrdin') and paste the result?

ali-farooq0 on 21 Nov 2019

@ali-farooq0 you're welcome,

Thanks guys! Looks like @myrdinm has it enabled so that is expected. However for @mthld and @S1M0N the UI is showing Never which is not in sync with the DB.

@mthld and @S1M0N, do you guys mind just retriggering the email setting by

Set it to Immediately, click Save.

Set it to Never, click Save

Query the DB again using (SELECT Id, NotifyProps FROM Users WHERE Username='myrdin') and paste the result?

Here we are, it looks like that have done it:

{
  "auto_responder_active":"false",
  "auto_responder_message":"Hello, I am out of office and unable to respond to messages.",
  "channel":"true",
  "comments":"never",
  "desktop":"mention",
  "desktop_sound":"true",
  "email":"false",
  "first_name":"false",
  "mention_keys":"mathilde",
  "push":"all",
  push_status":"away"
}

mthld on 21 Nov 2019

👍1

Yeaaa... that fixed it. Do you mind testing out the notification now? Looks like there is an issue with UI being out of sync with the DB.

ali-farooq0 on 21 Nov 2019

@ali-farooq0, that seems to have done it for me as well here:

{
  "auto_responder_active":"false",
  "auto_responder_message":"Hello, I am out of office and unable to respond to messages.",
  "channel":"true",
  "comments":"never",
  "desktop":"mention",
  "desktop_sound":"true",
  "email":"false",
  "first_name":"true",
  "mention_keys":"simon",
  "push":"mention",
  "push_status":"online"
}

S1M0N on 21 Nov 2019

👍1

Yeaaa... that fixed it. Do you mind testing out the notification now? Looks like there is an issue with UI being out of sync with the DB.

LGTM I triggered a webhook that post something in a room and didn't get an email. 👍🏻

mthld on 21 Nov 2019

Excellent! Phew, glad we figured out what caused it. Now I have to figure how it got into that state 🤔

ali-farooq0 on 21 Nov 2019

👍2

Fix is scheduled for v5.19,

amyblais on 7 Jan 2020

Mattermost-server: Email notification still sent while disabled in UI

Summary

Steps to reproduce

Expected behavior

Observed behavior

Possible fixes

Most helpful comment

All 29 comments

Related issues