Plots2: When mails fail to send, queue them for later.

[icarito]

Mail systems can sometimes have interruptions (e.g. we have a throttle from google for example).

Could we queue up failed outgoing emails?

There are subtle bugs like https://github.com/publiclab/plots2/issues/4774#issuecomment-461646291 that are confusing and if we had a proper mail queue cache we would avoid them.

One non-coding way to do this is to implement an MTA container and configure it for such retry queue.
Or it can be done at the application level. MTAs are designed and for this task, but on the other hand this feature would only be pre-configured when running with Docker if we went the container path.

[dewanhimanshu]

We can use redis and resque for this . Resque also provide a web interface for admin.

[grvsachdeva]

Actually, we already have redis and sidekiq setup so how about using them? Thanks!

[dewanhimanshu]

Thats great . yes we can use sidekiq also for this . Is there a web ui for it also?

[grvsachdeva]

Yes - https://publiclab.org/sidekiq

On Wed, Feb 13, 2019 at 1:32 PM Himanshu Dewan [e-mail]
wrote:

Thats great . yes we can use sidekiq also for this . Is there a web ui for
it also?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/publiclab/plots2/issues/4782#issuecomment-463097800,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AT6S9iDpbYNwYfZFNZCCzgxQaeWQpYJoks5vM8Z5gaJpZM4ayXiO
.

[dewanhimanshu]

Thats great . Then its better to use sidekiq and redis . We can create a background jobs/thread for the mails .

[grvsachdeva]

Yup. you want to work on this @dewanhimanshu? Also, what do you think about this @icarito @jywarren? By queuing email for later, the user will not experience any delay or conflict in events like Signup, liking a comment, commenting, etc wherever email is sent in the background. Thanks!

[dewanhimanshu]

I will be difficult for me to work as my mid sem exams are comming . But thanks @gauravano for giving me this opportunity. User experience will remain smooth . Actually i have used background jobs in my projects and it improves the user experience . Mail will be send on seperate thread instead on main server thread.

[grvsachdeva]

Yeah, I have also built a project in past using Redis and Resque, the user experience really improves with using queue. We can start this project after a week or two(I also have mid sem from next weekend :see_no_evil:). Let's work on this one in March or if someone wants to implement this, please go ahead.
And, all the best @dewanhimanshu for your exams!
Thanks!

[dewanhimanshu]

Thank you @gauravano . All the best for your exams also.

[grvsachdeva]

Thank you!

[jywarren]

Ah, it looks like we can use deliver_later in Rails now: https://guides.rubyonrails.org/action_mailer_basics.html#calling-the-mailer

Active Job's default behavior is to execute jobs via the :async adapter. So, you can use deliver_later now to send emails asynchronously. Active Job's default adapter runs jobs with an in-process thread pool. It's well-suited for the development/test environments, since it doesn't require any external infrastructure, but it's a poor fit for production since it drops pending jobs on restart. If you need a persistent backend, you will need to use an Active Job adapter that has a persistent backend (Sidekiq, Resque, etc).

I believe we are already set up with all this!

It's worth trying -- we can try to change any instance of deliver_now to deliver_later -- try one of these! https://github.com/publiclab/plots2/search?q=deliver_now

We'd love help with this if anyone's interested!

[jywarren]

Because the default runner in ActiveJob is inline, i wonder if perhaps it won't even break the tests! https://weblog.rubyonrails.org/2014/8/20/Rails-4-2-beta1/

[jywarren]

Well, here's an initial test for all admin mailings: https://github.com/publiclab/plots2/pull/4967

[jywarren]

This worked! https://github.com/publiclab/plots2/pull/4967 It required wrapping tests in a block starting with perform_enqueued_jobs do and end.

We could merge this now, then repeat for remaining instances of deliver_now:

https://github.com/publiclab/plots2/search?q=deliver_now

[jywarren]

I think we should be good to merge this, since it only affects admin emails. We can try it out before reapplying this to other email systems. I've opened a new issue to repeat this process in other parts of the codebase:

https://github.com/publiclab/plots2/issues/5311