Postal: Send limit improvements

[RLejolivet]

Hello,

Having just now tried the "Send limit" system, I found it pretty strict. Here are a few suggestions of options that could be given to administrators:

• Allow to completely disable the limit. I tried putting 0 or -1 as the limit but that just blocked all messages instantly.
• Allow automatic releasing of held messages once the number of messages went back below the maximum. This would make for a simple rate limiting feature
• Add a complete rate limiting system, allowing custom rules (like IP pool rules but this time for rate limits): limit mails towards X domain but not Y domain etc...

[adamcooke]

I like these suggestions 😄 Certainly the first two. I'll get enhancement issues added for them shortly.

[ro78]

+1 for automatic release of held messages once the quota is back to normal.

[navinp]

Hello,

For example:
If my hourly send limit is set to 600 (so 10 emails per minute) and I send 1800 emails.

It will reach the maximum per hour, so will it resume the following hours?
First hour 1 to 600 emails
Second hour 601 to 1200 emails
Third hour 1201 to 1800 emails and so completing the batch.

[willpower232]

At the minute, if the limit is reached, other messages submitted during that hour will be put on hold and then retried within Postal at a later stage. It will not be as precise as your example so if you are expecting to send that many messages, increase the limit above 1800 per hour.

If you want to schedule your messages, you would have to do this in your application that sends messages to Postal or use something else.

[navinp]

Hello,

So If working with my example:

If I submit 6.000 emails, 600 emails will be processed and 5.400 emails will be Held to be processed later. I have seen a week later.

Is this correct?

Why not in the queue to be processed the hour after?

And yes, it's better to have a software that handles this hourly, but most of my customers does not have this, I thought this was handled by Postal.

[navinp]

For example this is what I am seeing for a message that was sent just now:

This message has been held. By releasing the message, we will allow it to continue on its way to its destination. It will be held until July 26, 2018 15:51.

So if I understand correctly:
Messages sent today, and that didn't make it into the 600 emails hourly threshold, will be hold until July 26 before being released.

Or I need to manually release all messages one by one.

Am I right?

[navinp]

If my customer needs to send 20.000 emails, I would in this case need to change the hour limit to 20.000 which is not so good, from a "not being blocked by major ESP's" perspective.

If I set the hour rate 20.000/24 = 833 emails per hour (thinking that in 24 hours all emails will be sent)

The customer proceeds and sends 20.000 emails, this means, that in reality only 833 emails were sent, and the other 19.000+ emails will be sent next week, 7 days later?

[willpower232]

Actually I might be mistaken, a hold might keep them from being sent altogether until the time mentioned, after that point it would have to be resubmitted to Postal.

If you have held messages, you will need to release them manually if you want them sent.

If you are sending very large volumes of email you could just remove the limit altogether.

[navinp]

I have a customer on a dedicated server, who has now over 1 million emails Held :/

Where I thought that the hourly limit meant, that every hour x amount of emails were going to be processed ;/

Removing the hourly limit => I was doing the IP warm up, by increasing the daily limit.

Removing the hourly limit would just blast everything getting the IP blacklisted asap.

But now I know that by increasing the daily limit didn't help :/

[navinp]

This is my suggestion:

The emails must remain in the queue and not go to held, maybe in certain situations only.

If the hourly limit is 50, and a customer sends 5.000 emails, it means it will take 100 hours.

After 100 hours (or 4+ days) all 5.000 emails must have been processed.

==
At the moment:
50 emails per hour
5.000 sent
50 emails processed
4.950 held to be processed 7 days later

1 week later another 50 emails processed, the remaining 4.900 to be processed 7 days later (so 2 weeks later only 2% processed)

[navinp]

I was thinking today, a simple solution for this would be instead of pushing held emails 7 days later, just push it to the next hour. This way it will continu sending and process the emails in the same day.

What say you? @willpower232

[willpower232]

If you want to accomplish this, you could do something similar to the expire_held_messages_job.rb but if the reason for holding was that the receiving email server was very sure about your messages being spam, potentially more messages would hard fail and your reputation could be impacted further.

[navinp]

This would apply to messages that are being held, because a threshold was met, at the moment all messages are pushed 7 days later... but yes, I don't know what the consequences are for this change, would have to test to find out

[navinp]

How are other big companies like Sendgrid doing this?

If someone sends out 40.000 emails, they won't send all 40.000 at once.

I know MailChimp does not send all at once, because my friends and I are on a mailing list, and we never receive the same email at the same time, from a company we are subscribed at.

So it's obvious they schedule the emails to send out x amount per hour, next hour following same x amount again etc etc.

So in Postal if x amount is 500 p/h and someone sends 4.000, it should take max 8 hours to send the 4.000 emails.

So the suggestion I had regarding changing the 7 days period, applies to a situation like this.

This would not apply for emails that were bounced/failed, because indeed as you were saying, it could hurt my IP extra.

[navinp]

So this is what I meant, below you see is a message that is Held.

The only reason for this is, because the hourly limit was met.

The message will be released 7 days later.

The customer send 10.000 messages, now 7.500 of those messages will be released 7 days later.

Instead of releasing this the next hour.

This message has been held. By releasing the message, we will allow it to continue on its way to its destination. It will be held until August 18, 2018 13:12.

Release message Cancel hold

August 11, 2018 13:12
HELD
Message held because send limit (2500) has been reached.

[CHTJonas]

I wrote a dumb script which seems to solves the problem of automatically requeuing messages, all be it by taking a sledgehammer to a nail. First open a Rails console by typing bin/postal console and then copy in the following:

server_id = 1
retry_limit = 140
db = Server.find(server_id).message_db
query = { where: { held: 1 }, limit: retry_limit }
begin
  sleep 3600
  messages = db.messages(query)
  messages.each { |m| m.add_to_message_queue(manual: true) }
end until messages.length == 0

Obviously replace server_id and retry_limit with the ID of the server instance in question and the number of emails that you want to retry per hour (note you should set this to be less that your server's send limit).

I have no idea whether anything in the above is bad practice or not, but it seems to work!

[willpower232]

Certainly easier than clicking through the UI, nice one!

[navossoc]

Everything is great, but I know, it is a lot of work to do...

So, a a simple "dumb" rate limit of x/messages per second would be nice and will help a lot.

[losingle]

I wrote a dumb script which seems to solves the problem of automatically requeuing messages, all be it by taking a sledgehammer to a nail. First open a Rails console by typing bin/postal console and then copy in the following:

server_id = 1
retry_limit = 140
db = Server.find(server_id).message_db
query = { where: { held: 1 }, limit: retry_limit }
begin
  sleep 3600
  messages = db.messages(query)
  messages.each { |m| m.add_to_message_queue(manual: true) }
end until messages.length == 0

Obviously replace server_id and retry_limit with the ID of the server instance in question and the number of emails that you want to retry per hour (note you should set this to be less that your server's send limit).

I have no idea whether anything in the above is bad practice or not, but it seems to work!

Perhaps you could create a task to schedule automatically

class RequeueHeldMessagesJob < Postal::Job
  def perform
    Server.all.each do |server|
      if !server.queued_messages.first
        retry_limit = server.send_limit.nil? ? 200 : server.send_limit - server.throughput_stats[:outgoing]
        db = server.message_db
        query = { where: { held: 1 }, limit: retry_limit }
        messages = db.messages(query)
        messages.each { |m| m.add_to_message_queue(manual: true) }
      end
    end
  end
end

[dragoangel]

Also to add additional point: can you add rate limit per credentials?