Ghost: Feature: Subscribers V2/V2.1

[kirrg001]

Subscribers V2

After Subscriber V1 #6764, we will work on Subscriber V2.
We already started #6942.

V2 will send emails to all your subscribers. We will only support mailgun for the first version.
Later we can support also sendgrid for example.

Basic requirements

• we want to offer a way you can schedule your newsletter as best as possible ("schedule newsletter on every monday of the month")
• we want to provide statistics for each newsletter
• we want to support sending batch emails to your subscribers
• the blog should be able to configure from which email address the newsletter should be sent
  

  Admin UI

Ghost Admin needs a new section in the subscribers area:

• a field to enable/disable newsletter
• supported UI options to schedule a newsletter: daily + time, weekly + day of week + time, monthly + day of month + time
• ember will send each update via PUT /settings
• ember needs to read the server config to decide wether you can enable newsletter or not
• ember needs a new section for override the from mail address (account section)
• ember needs to implement a UI to RRULE specification
• ember needs to show the statistics for each newsletter

Using RRULE to schedule a newsletter

• defined in http://www.ietf.org/rfc/rfc2445.txt
• this is how a RRULE looks like RRULE:FREQ=MONTHLY;BYDAY=MO,TU,WE,TH,FR;BYSETPOS=-1
• ember will send this string over when enabling the newsletter
• the RRULE gets parsed when the newsletter event is triggered (created/updated/deleted)
• based on the event, we parse the RRULE and get the next execution date
• with the date we ask the scheduler to schedule the newsletter (time: next monday 10Am, url: /schedules/newsletter/)
• we will use https://github.com/jkbrzt/rrule - this is not anymore supported, but i will look for a fork
  

  Store newsletter information

We keep it simple for now and create a new settings entry.

key: newsletter
value: { RRULE: String, lastExecutionAt: datetime, status: String [enabled, disabled]}

Store newsletter from email address

A new settings key should be created for from email.

key: newsletter_from_address
value: String (email)

newsletter scheduling component

• newsletter functionality will use either the SchedulingDefault adapter or the SchedulingPro adapter on Ghost(Pro)
• the newsletter adapter will listen on updates of the settings newsletter key
• it will receive the RRULE expression and based on this it will calculate the next newsletter date
• if status is disabled and has changed, we will unschedule otherwise we will reschedule to always delete the old newsletter job
• we ensure the periodical newsletter feature by updating the lastExecutionAt property when the current newsletter was sent, then newsletter event get's triggered
• lastExecutionAt should be updated even if the newsletter couldn't be send out

• if the blog restarts we need to ensure scheduling the newsletter (see post scheduling)

  mailgun adapter

• write a maligun API adapter

• for Ghost(Pro) we have to setup a new domain for mailgun - this domain will be used for all pro blogs
• should offer the following functions: send
• use https://documentation.mailgun.com/user_manual.html#batch-sending (max allowed is 1000 per call)
• if errors like "monthly limit reached", we stderr the error for now, thats it
• offer a way to use a different mail adapter for newsletter (not high pro)
  

  create endpoints

PUT /schedules/newsletter?client_credentials

• schedules a newsletter
• we have to build the template
• we have to get all published posts based on the from.....to
• to is now, from is lastExecution in settings newsletter
• it will choose the mail provider and executes it (like data.mailgun.send())
• when the mail was send, we update the newsletter settings lastExecution to the new one and so we can ensure reoccurrence
• we can ensure a newsletter does not get send twice by checking the lastExecutedAt

POST /newsletter/test

• send a test newsletter

POST /newsletter/unsubscribe/:hash

• public endpoint to unsubscribe
• look what is best practise for unsubscribe urls

write a piece of code to generate emails

• a list of objects should create a mail template
• offer unsubscribe link
  

  Subscribers V2.1

V2.1 will provide stats for your newsletter.

challenges

• how to offer a quick response of stats
• should we cache stats in our database for already existing queries?
  

  create endpoints

GET /newsletter/statistics

• based on the provider (for now only mailgun)
• returns statistics for each newsletter
• { newsletter_tag: { delivered: 100, opened: 10 }}

mailgun adapter

• should offer the following functions: getStats, getEvents
• stats API stores data minimum 6 month (so each blog can get stats for the last 6 month)
• events API stores data only 30 days (this means you can only get information about to whom you send your newsletter to for 30days)
• we are using tags to identify each blog and each newsletter
• maligun tags have a maximum length of 128 chars (which is unpossible to break)
• mailgun tags for self hoster: blogUrl-YYYY-MM-DD HH:mm:ss (if > 128 chars, we shorten)
• mailgun tags for Ghost(Pro): userId-YYYY-MM-DD HH:mm:ss
  

  Store newsletter information

We add newsletter_tags to the newsletter settings key.

key: newsletter
value: {...newsletter_tags: Array...}

• newsletter_tags: it's an array of strings. we will keep track of all newsletter a blog send out, so we can always get statistics for each newsletter (how many recipients opened the newsletter). Each newsletter tag needs to be unique. We can ensure this by creating the tag like this: blogDomain + YYYY-MM-DD HH:mm:ss.

[nuclearpengy]

wow - this looks like it's going to be epic. :)

[rawsh]

Woohoo, I can't wait to test this!
Are you going to use the mailgun lists and add subs to mailgun lists or just keep the local copy? I would personally prefer the mailgun lists because you can use your own settings and send custom emails to the subs easily (e.g. announcements). Perhaps the mailgun email list could be [email protected]

[jloh]

@rawsh I guess that depends whether we're able to give our own mailgun keys for GhostPro or whether that wont be exposed by the interface and just allow you to use custom from addresses. Remember this feature isn't even realised yet!

Ideally I'd prefer to be able to setup and manage my own mailgun account, but that mightn't be possible with this release.

[rawsh]

@jloh Will the subs newsletter be accessible from hosted ghost? I would also like to use my own api key.
One big feature I think is missing from Ghost Pro is ssh/sftp access. Then we could set that stuff up ourselves. The main reason I am not using Ghost Pro is that I don't have access to my files, and I don't want to open up support every time I need to change something.

[kirrg001]

RE: custom mailgun account:
It is not planned for V2 that you can use your own mailgun account with Ghost(Pro). For self hosted blogs it is planned that you can add your mailgun credentials into config.js.

RE: mailing lists vs. batches mailgun:
Mailing lists are a good alternative to batches.

I can see two disadvantages right now:

• for Ghost(Pro) we would have to transmit all the subscribers for each blog to mailgun (imagine if we would have 50k blogs and each blog has 1000 subscribers) - every Ghost(Pro) user is using the same mailgun Ghost account
• brings complexity - for each new subscriber we need to sync with mailgun

[kevinansfield]

@rawsh, @jloh: the key point here is that this is the first iteration, we're intentionally keeping it limited to contain the scope and ship something in order to learn where the pain points are and which additional features and configuration is needed to satisfy the many different use cases. Your comments are definitely being taken on board and will be considered in future iterations as we want to ensure that the feature is viable for as many users as possible whether on Ghost(Pro) or self-hosted.

[nuclearpengy]

Simplicity is part of what makes Ghost so awesome. The difficult trick is finding the balance between very customisable(yet complicated) and simple(yet effective).

[sakulstra]

Will we be able to customize the newsletters look/create a custom theme?
Will it be possible to send a custom(not autogenerated) newsletter?

[ErisDS]

@sakulstra Not for v1 - if I can draw your attention to this comment 😊

[kirrg001]

I just discovered that mailgun only lets you create 4000 unique tags. There will be an update for V2.1 in the next days.

[kirrg001]

We will not finish Subscribers V2 within the next three month.
But you can still ask questions here 👍

[dkbast]

I think the it would be nice to have a event which is called each time a new subscriber is added. There could be a "code injection" field to handle this event and kick off a custom script. This might prove as a good intermediary solution until the mailgun feature is implemented (and for anybody who would like to run his own script)

[ErisDS]

Closing this with the later tag for when we start working on subscriber type features again after the launch of 1.0

[EAT-CODE-KITE-REPEAT]

Please open this again. This is the most useful feature ever :)