Sentry: Sentry 8.5.0 doesn't send emails

[cocojas]

A clean install of the 8.5.0 version of Sentry won't send emails. Everything is configured correctly, and sending the test email always works, but no other emails are ever sent. This includes user invite emails, forgot password emails, and notification emails. There are no errors in any of the logs anywhere, and the mail exchanger never sees any attempted connection from the sentry server.

I've seen a lot of closed issues from earlier versions of 8.x regarding emails, is this still an ongoing, known issue? I didn't see any open issues, however.

Happy to provide additional information if necessary.

[dcramer]

Can you confirm that the beat process is running and Redis is configured correctly? Alert emails go through the digest background, which needs both of those.

[cocojas]

Yes, redis is configured correctly, and the celery beat and workers are running and connecting to redis. I did change them to start with sentry run cron and sentry run worker since the other ones gave me deprecated warnings. There is nothing in the logs for the beat process or the worker processes. Redis is version 3.2, could that be part of the problem?

[mattrobenolt]

I'm not entirely sure if there are issues with redis 3.2 or not. We recommend 3.0 and that's what we run ourselves, and haven't tested against 3.2.

If you run sentry run worker --log-level=INFO or sentry run worker --log-level=DEBUG, it should ideally shed some more light on the issue.

[cocojas]

This is a snippet of logs at the debug level, everything seems to be running, but it still definitely isn't trying to connect to the mail host or route the mail, even tho the logs say it's succeeding:

[2016-06-08 03:01:22,999: DEBUG/MainProcess] Task accepted: sentry.tasks.options.sync_options[8a87208c-20bb-4d29-8943-c54bd47e4a14] pid:18
[2016-06-08 03:01:23,005: INFO/MainProcess] Task sentry.tasks.options.sync_options[8a87208c-20bb-4d29-8943-c54bd47e4a14] succeeded in 0.00888878293335s: None
[2016-06-08 03:01:27,197: INFO/MainProcess] Received task: sentry.tasks.email.send_email[bf381f93-5245-4191-8873-131c8f09d4b9]
[2016-06-08 03:01:27,198: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fdecb714758> (args:('sentry.tasks.email.send_email', 'bf381f93-5245-4191-8873-131c8f09d4b9', [], {'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fdeca54b850>}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': [], 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': 'email', u'exchange': 'default'}, 'expires': None, u'hostname': 'celery@8833ee7ac0f4', 'task': 'sentry.tasks.email.send_email', 'callbacks': None, u'correlation_id': 'bf381f93-5245-4191-8873-131c8f09d4b9', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fdeca54b850>}, 'eta': None, u'reply_to': '2842c30b-a410-3a24-b004-5b0adef9a84f', 'id': 'bf381f93-5245-4191-8873-131c8f09d4b9', u'headers': {}}) kwargs:{})
[2016-06-08 03:01:27,203: DEBUG/MainProcess] Task accepted: sentry.tasks.email.send_email[bf381f93-5245-4191-8873-131c8f09d4b9] pid:20
[2016-06-08 03:01:27,216: INFO/MainProcess] Task sentry.tasks.email.send_email[bf381f93-5245-4191-8873-131c8f09d4b9] succeeded in 0.0172282271087s: None

Incidentally, I had to run sentry run worker -l DEBUG for this to work. It doesn't seem to like the --log-level option anymore.

[mattrobenolt]

Oh, sorry, it's loglevel, not log-level. :)

The logs look ok, so the only thing I can suggest is that you are running with a different configuration or have email disabled on your config.yml.

You can confirm these settings as well by doing:

$ sentry config list mail.*

to get all of the mail options, then checking them with:

$ sentry config get mail.backend

and making sure everything is what you expect it to be.

Other than that, email definitely works, so it's just a misconfiguration somewhere.

[cocojas]

That was it! I was playing with the setup using docker, and it didn't even occur to me that the workers would need the same configuration as the main sentry process. They're sharing a config now and everything's working. Thanks for the help @mattrobenolt!

[mattrobenolt]

:(

This has come up a number of times with our docker users. Is there something that we could be doing differently to assert that everything is the same across all containers? I haven't seen anything that does anything different than what we do. Is this just documentation problem on our side? Since we're lacking in that department for sure at the moment.

Just looking to solicit feedback here.

[cocojas]

No, I think you guys did a pretty good job. In re-reading the documentation at docker hub, you clearly call out the mail variables. It might be useful to just add a little bit to that documentation that the variables are needed across all the workers and the website to work correctly. That's the part that wasn't immediately clear to me.

I was going to suggest doing a little docker compose file, where you could more clearly call out the environment variables, but since you need to run upgrade interactively, it probably would just confuse people more.

One thing that might be useful to add to the examples documentation might be something in-between the on-build and the regular container, where you just want to provide your own configs. What I ended up doing, was something like:

• create a directory somewhere on the host for the config files, like /mnt/sentry/cfg
• create/edit the config.yml and sentry.conf.py files
• start all workers and the web process with the same config:

$ docker run -d --name my-sentry -v /mnt/sentry/cfg:/etc/sentry:ro -p 9000:9000 -e SENTRY_SECRET_KEY='<secret-key>' --link sentry-redis:redis --link sentry-postgres:postgres sentry
$ docker run -d --name sentry-cron -v /mnt/sentry/cfg:/etc/sentry:ro -e SENTRY_SECRET_KEY='<secret-key>' --link sentry-postgres:postgres --link sentry-redis:redis sentry run cron
$ docker run -d --name sentry-worker1 -v /mnt/sentry/cfg:/etc/sentry:ro -e SENTRY_SECRET_KEY='<secret-key>' --link sentry-postgres:postgres --link sentry-redis:redis sentry run worker

This way too, the operator can edit the config files and just issue a docker restart sentry sentry-cron sentry-workerX, or even do a docker kill -s SIGHUP sentry if you support reloading configs via signals.

[mattrobenolt]

Thanks for the feedback. :) Pretty much all of what you suggested is happening very soon.

The Docker Hub docs are just a quickstart before we had official docs.

[lippoliv]

Hey guys.
Thanks for this Thread. I had no idea why my Test-Mail got sent but no other mail (Invitation / Exception) got sent. I also played around with dockerfiles, wich is actually really nice for me as of I am no pro server admin (just software developer).

Now it makes sense that some mails "aren't send". As of the workers hadn't hat the right configuration.
It was really confusing that the "Test Mailsettings" worked, while the other mails didnt.

I added the needed ENV to the worker it worked directly :)

[angelicaortega-natgeo]

I have a quick question regarding this, I have seen a lot of information about the configuration for the mail, what is not clear enough to me is as far as I config the sentry.conf.py or the config.yml and regenerating the containers it should be taken by it self Right? So I'm not sure if the best way is configure it as cocojas mentioned or even generating an .env file in the Sentry folder, or starting the containers with the proper config.