Nextflow: Built in e-mail function

[ewels]

Hi @pditommaso,

As mentioned on gitter, I put together a minimal example of a pure-groovy e-mail feature that renders a HTML template and sends a properly formatted MIME multipart HTML e-mail (with embedded images, eg. for a NextFlow logo). You can see this code here:

https://gist.github.com/ewels/2da5023f0b045f92434ad0cb5ad7e8aa

Note that it also has a plain-text template too, it's possible to send a multipart text / HTML e-mail using this. But I couldn't figure out how to send multipart mail with embedded images _and_ the plain text. So I commented that bit out.

Anyway - my feature request is to make something like this into a core function or option. Users could then customise to varying degrees if they wanted to - logos / headers / extra fields etc, or writing their own HTML templates.

The messy bit is the part of actually sending the e-mail. The reason that I've used pure groovy code here is to provide reliable cross-platform functionality. This is fine, however SMTP server details may need to be specified if smtp.localhost:25 doesn't work (params.email_host etc).

Your previous examples (and others' suggestions on gitter) mention using mail or mailx. I looked into these (and sendmail), but couldn't find a reliable way to use them to send HTML (multipart) e-mails on them on all systems. Most commands seem to work on linux _or_ OSX, but not both. These tools have the advantage of (usually) already knowing details for smtp addresses on the system.

Phil

[ewels]

ps. A bit more context - I'm already using mail in our NGI-RNAseq pipeline (see code here) and it works beautifully when I run locally on OSX:

---

---

However, when I run remotely on linux I get this instead:

---

---

See StackOverflow threads such as this one for discussion about differences in how linux / OSX handle multipart e-mail.

[pditommaso]

Thank Phil. Are you aware of any standard way to fetch the current SMPT configuration from the underlying system. I would like to maintain as much as possible the zero-config philosophy of NF, hence ideally the built-in mail function should work out of the box w/o extra config.

[pditommaso]

I see, thus the smtp.localhost:25 could be used as the default smtp server. I'm wondering if there's a way to extract a custom SMTP configuration?

[ewels]

Yes exactly - that's what I set in the gist above, but it doesn't work for me on OSX - I get this error:

javax.mail.MessagingException: Could not connect to SMTP host: localhost, port: 25;

Apparently mail and mailx use sendmail - see https://superuser.com/a/137522

But I totally agree that it would be much better to not have to set any smtp details etc. if we can avoid it.

[ewels]

Having played around with this a little more, I think that sending HTML e-mail with sendmail is the most robust method. In my testing, it's the only way I've found to reliably send HTML e-mail on both OSX and Linux systems. I've enclosed it in a _try/catch_ loop which falls back to a plain text e-mail if the command fails:

// Send the HTML e-mail
if (params.email) {
  def subject = "NGI-RNAseq Pipeline Complete: $workflow.runName"
  try {
    // Try to send HTML e-mail using sendmail
    def html_email = "To: $params.email\nSubject: $subject\nMime-Version: 1.0\nContent-Type: text/html\n\n$email_html";
    def smproc = [ 'sendmail', '-t' ].execute() << html_email
    log.debug "[NGI-RNAseq] Sent summary e-mail using sendmail"
  } catch (all) {
    // Catch failures and try with plaintext
    [ 'mail', '-s', subject, params.email ].execute() << email_txt
    log.debug "[NGI-RNAseq] Sendmail failed, failing back to sending summary e-mail using mail"
  }
  log.info "[NGI-RNAseq] Sent summary e-mail to $params.email"
}

Not as tidy as writing it all in Groovy, but it does work without any configuration. Haven't tried putting in the multipart (image attachment) stuff in yet though.

[ewels]

Ok, and finished product now done. Code generates a plaintext report, html report and wraps all of that together using a sendmail template. This is rendered and then sent using sendmail, falling back to plaintext with mail if sendmail fails.

Quite a lot of trial and error to get to this point, but seems to be working really nicely now! Good e-mail rendering in all clients I've tried so far, and working on all systems I've tested.

[pditommaso]

Excellent work. You could even create a small groovy helper class and include in the lib/ folder to automatically import it in your script. This would make much easier to test it w/o the need to launch the overall pipeline execution.

[pditommaso]

A first implementation could to add a built-in function that send an email by using the java api when the user provide the required configuration setting. It would be enough to add a static method in the Nextflow class. For example:

    static boolean sendEmail(Map param) {

        def cfg = Global.session.config

        if( cfg.email ) {
            // java send email
        }
        else if( macOs ) {
            sysexec sendmail
        }
        else {
            sysexec  mail
        }
    }

Having a a config similar to:

email {
  smtp = 'smtp.gmail.com'
  port = 4555
  sender = '[email protected]'
}

[ewels]

Yes, this could definitely work! Probably needs email.to as well, so we know who to send the e-mail to ;) (maybe with a command line option too?)

Note that this new e-mail functionality could tap in nicely to https://github.com/nextflow-io/hack17/issues/4: use a generalised system to render an e-mail template and then use the above function to e-mail it. The e-mail template could be some kind of multipart template for sendmail perhaps? Or maybe something more generic which can work with groovy html e-mails as well. Not sure.

Phil

[pditommaso]

OK, the above commit add the sendMail built-in function. It allows uses Java mail API when settings are provided or fallback on system sendmail. For example:

sendMail to: '[email protected]', from: '[email protected]', content: 'Hello there'

Or in a more idiomatic wait

sendMail {
   to '[email protected]', 
   from: '[email protected]',
   subject: 'Notification' 
   attach '/some/file'
   attach '/even/multiple/files' 
   content: '''
    Look ma! 
      multilines 
           email content :) 
    ''''
}

The mail settings can be provided in the nextflow.config file in the mail scope. For example:

mail {
  from = '[email protected]'

  smtp.host = 'email-smtp.us-east-1.amazonaws.com'
  smtp.port = 587
  smtp.user = 'xxx'
  smtp.password = 'yyy'
  smtp.auth = true
  smtp.starttls.enable = true
  smtp.starttls.required = true
}

The usual Java mail properties can be used.

What to do next:

• still not sure about the function name: sendMail, mail or sendmail (?)
• eventually rename the content attribute to body (?)
• implement the support for mail if sendmail is missing
• error handling, what should happen on sending failure? exception? just a warning?
• mail templating .. (pandora box!)

Comments are welcome (and thanks to @edgano for contributing).

[ewels]

Brilliant stuff! Confusing nomenclature though 😉 So we have a java function called sendmail which falls back to the system sendmail command which itself falls back to the system mail command...!

How does the Java functionality handle multipart MIME email content? I see that there are variables for attachments, but is it possible to have HTML emails with attached images embedded?

I think it would be brilliant to have a built-in template that can have content dropped in. There could be variables for body content, footer, header, title, logo file etc. Then people can always use their own template as well if they want to. I wrote a multipart HTML emails for our templates and I think they make a huge difference to how easy they are to skim-read (though they were a bit of a pain to write!).

Thanks for working on this both!

[pditommaso]

Neatly .. the NF built-in function is (currently) named sendMail and it falls back to sendmail, not yet to mail.

How does the Java functionality handle multipart MIME email content?

Forget to say that's a type attribute that allows you to set the mail content mime-type (basically this line).

I see that there are variables for attachments, but is it possible to have HTML emails with attached images embedded?

Yes, the attachment(s) mime-type should be automatically discovered.

I think it would be brilliant to have a built-in template that can have content dropped in.

It makes sense

I wrote a multipart HTML emails for our templates and I think they make a huge difference to how easy they are to skim-read

Do you an example? How do you handle HTML emails when using mail? You were mentioning it's not possible to handle HTML content.

[ewels]

As I understand it, I don't think you can send HTML emails using mail. We just fall back to sending the plain-text version in this case (here).

I think it's good to generate both HTML and txt versions of the email anyway - I find it useful to have a plaintext version to read on the command line (we save both to the results directory as they're essentially pipeline reports). It's also good practice for the 0.1% of people who use an email client that doesn't handle HTML.

Phil

[pditommaso]

Being when you use mail you cannot either send attachments. It's a pure text email right?

[edgano]

Yep.
With the sendmail you can send an attachment. But with mail I didn't found the way

[pditommaso]

Another round of improvements here:

1) content has been renamed to body and it's optional eg

sendMail {
   to '[email protected]', 
   from '[email protected]'

   '''
    Look ma! 
      multilines 
           email content :) 
    ''''
}

1. Mail alternative content (text and html) is now supported
2. Body mime type is automatically detected
3. Explicit plain text body can be specified with text attribute
4. Support of mail sys command.

@ewels you may consider this my christmas present for you :D

[ewels]

📧 📫 🎅 🎄 🎁 🎉

Not sure when, but will try to have a play asap. Is there a minimal example of this in action somewhere that I can copy from? eg. how I should actually call it..?

[pditommaso]

That's a built-in function that you can use in the pipeline script. You can specify the mail parameters both as named parameter or by using a closure eg

sendMail from: '[email protected]', to: '[email protected]', subject: 'Hi', body: 'mail content' 

or

sendMail {
   to '[email protected]', 
   from '[email protected]'

   '''
    Look ma! 
      multilines 
           email content :) 
    ''''
}

Have a look at the Mail class the the complete list of parameters.

[ewels]

ok nice - so I can just put that snippet directly into my pipeline script? Fab! Apologies, I know you put that further up but it looked too simple to be just that!