Postal: How Postal works?

[afanjul]

Hi people, I didn't find anything about it, so I would like to ask you how postal works (regarding processes, cron, queue, smtp, etc). I mean, what is the flow chart throw all of this "boxes"? I can try to figure out this:

• Every time we send an email using postal, our smtp server get the email and save it in qeued_messages table (in database)
• There is a 1-minute (configurable?) cron job (the worker?) that get rows from the queued_messages.
• Then they are grouped by batch_key (domain) to only ask for mx records 1 time per domain
• ... here is maybe something regarding sending status, locked messages, etc.
• Then this worker send a unique (or multiple?) request to mx servers using smtp protocol
• The server could send a request to a webhook (based on message status) if configured

Just an figured out approach that surely is wrong... would be fine if somebody could write the right flow diagram (even in text) and I promise to turn it into a beautiful graphical flow chart...

[adamcooke]

It would be nice to have some documentation that outlines exactly how all this stuff works. We'll look at doing that but in the meantime:

1. A message is received that needs to be processed. This might be received from the API or from SMTP server.

2. This message's metadata is added straight away into your mail server's messages table. The raw headers & body are stripped out and stored in their own tables. They are pieced together again when needed.

3. A link to the message is also added to the queued_messages. At this point, the message will also be allocated to an IP pool based on the configuration.

4. A job will be dispatched to RabbitMQ to "unqueue" the message. The actual backend queue that is published to depends on the IP pool configuration.

5. The unqueue job will process the message in a worker. Depending on the type of message a number of different things happen. For this example, we'll assume it's an outgoing message that is destined for another SMTP server.

6. The first thing the unqueueing job will do is look at the queued_messages table to see if there are any other messages that can be processed at the same time. This uses the batch_key (which for outgoing messages is the domain of the recipient) to find messages that will need to be sent to the same ultimate SMTP server (to avoid multiple connections to the same SMTP server in a short period of time). Any messages that are processed by the worker are locked to ensure they aren't processed simultaneously by multiple workers.

7. Each message that is being unqueued will run through a routine that does things like spam checking and other processing.

8. Once this is complete and the message is ready to be sent to the external SMTP server, the worker will lookup & connect to the appropriate SMTP server, send the message and log the output as a delivery attempt. If there are mulitple MX records for the domain, each record will be tried in priority order. If the message is accepted, the message will be removed from the queued_messages table and no further attempts will be made to send it. If the message isn't accepted, the number of attempts & the retry_after timestamp will be updated on the queued message and it will left in the table for requeueing again later.

9. Every minute, the requeuer process will look for any suitable messages that are sitting in the queued_messages table and will publish a message to the appropriate queue to have the message dequeued again by a worker (same as step 4). This handles retrying soft fails.

[longhorninmd]

How are the queues organized? One master one, or multiple ones by domain or IP?