K-9: iCalendar support

[towo]

Having the ability to view and accept (including sending a reply (!)) meetings on mobile would be a pretty great option.

Steps required for this:

1. ical/vcalendar parsing (copy and paste logic from e.g. GNOME Evolution (https://git.gnome.org/browse/evolution/)
2. Displaying event information in mail.
3. Add parsed event to calendar.
4. Offer to send response to organizer/all attendees.
5. Create response (yet another vcalendar) and send it, if requested.

Step four should probably not forget the security signing hook (e.g. openkeychain).

[AHoros]

From the duplicated issue #1661 I would escalate one additional simple sub-feature:

• filter inbox by emails containing attachment with *.ics name (sometimes there is more than one attachments in one email, but if at least one of them is calendar invite that we want to highlight this email)

NOTE: development could be separated into two stages: basic & advanced
with basic we can get big improvement in usability with these three sub-features:

1. ical/vcalendar parsing
2. Displaying event information in mail.
3. Filter inbox by emails containing attachment with ical/vcalendar files.

And for advanced and more complex development:

1. Offer to send response to organizer/all attendees.
2. Create response (yet another vcalendar) and send it, if requested.
3. the security signing hook (e.g. openkeychain).

[philipwhiuk]

In terms of quick off-hand estimation

• Basic is likely to be 1 & 2. (without 2, 1 is pointless). We don't really do huge feature branches here (we did one for PGP-MIME but that's an exceptional case) - this will likely be a single PR.
• 3 is complex.
• 4 is probably fairly simple
• 5 is probably fairly simple
• 6 I have no idea on.

The ordering of 3, 4, 5 and 6 will depend on usage. Personally, sending a response is more useful to me than filtering email by attachment (which is a pre-requisite for 3 I don't think we currently support) so I'd be inclined to do 4 and 5 first. But I'm not picking this up as there's lots of higher priority things to fix for me.

[philipwhiuk]

Potential library for iCal parsing: https://github.com/mangstadt/biweekly

[philipwhiuk]

Seeing noname.ics in my email inbox keeps bugging me so I started looking into this.

I've done some bits of work to K-9 for a while now, but the handling of Part's actual data still confuses me. I'm not quite sure yet where the parsing of the ICS file ought to take place yet.

What I do know is that the ICS file has a lot of complexity to it! There's lots of properties that probably most people never use. And there's a fair bit people do use.

I suspect a first PR will add basic support for simple events with participants.

We certainly don't need to deal with every field. Firstly because it's an email app, so stuff like Alarms is obviously out of scope. Secondly because supporting endless complexity is burdensome. And finally because it's a mobile client, so trying to fit in lots of minor information is going to use up valuable real estate.

Given ICS supports attachments inline there will need to be tests done on iCalendar files with large attachments to make sure we don't crash K-9 trying to parse an ICS file.

[Valodim]

Parsing this is probably best implemented by creating a new type of Viewable, similar to how mail headers are displayed in a special manner

[philipwhiuk]

So I didn't go down the Viewable route. It is a new Part type but in several places we make the assumption that viewable actually means HTML / TEXT / WebView capable. So instead I added some extra arguments.

Someone familiar with the database/LocalPart code will probably tell me I'm doing stuff in the wrong place. I know the constructor of ICalPart is doing stuff it really shouldn't. What I don't know is quite where that logic does belong.

Anyway: spike development: https://github.com/philipwhiuk/k-9/commits/ICSsupport

And fancy image:

Note that the names are fine in reality - I just blanked them except for mine.
That's an actual Outlook iCalendar attachment sent to another of my accounts, forwarded from me to me, being parsed and displayed.

I still need to do lots of work (in addition to feedback on LocalParts etc):

• [x] I need to add a way to download the ICS file again. It no longer shows as an attachment.
• [ ] It would be nice to get the usernames displayed in a way that you can send emails to them.
• [x] I appear to have broken Textual parts (I noticed this as wrote this comment)...
• [ ] I need to make sure forwarding sends the ICS file as an attachment.
• [x] I should add the 'Organizer' field if it's not the sender.

[philipwhiuk]

Assigned to me as I'm actively developing this.

I fixed Textual parts mostly. However:

• [x] Device crashing on sending e-mail (might be unrelated)
• [x] NPE for HTML emails - definitely related - source below.

com.fsck.k9.mailstore.LockableDatabase$WrappedException: com.fsck.k9.mail.MessagingException: Error appending message: **subject**
    at com.fsck.k9.mailstore.LocalFolder$12.doDbWork(LocalFolder.java:1090)
    at com.fsck.k9.mailstore.LocalFolder$12.doDbWork(LocalFolder.java:1078)
    at com.fsck.k9.mailstore.LockableDatabase.execute(LockableDatabase.java:275)
    at com.fsck.k9.mailstore.LocalFolder.storeSmallMessage(LocalFolder.java:1078)
    at com.fsck.k9.controller.MessagingController$12.messageFinished(MessagingController.java:1406)
    at com.fsck.k9.mail.store.imap.ImapFolder.fetch(ImapFolder.java:770)
    at com.fsck.k9.controller.MessagingController.downloadSmallMessages(MessagingController.java:1393)
    at com.fsck.k9.controller.MessagingController.downloadMessages(MessagingController.java:1196)
    at com.fsck.k9.controller.MessagingController.synchronizeMailboxSynchronous(MessagingController.java:953)
    at com.fsck.k9.controller.MessagingController$9.run(MessagingController.java:739)
    at com.fsck.k9.controller.MessagingController.runInBackground(MessagingController.java:204)
    at com.fsck.k9.controller.MessagingController.access$000(MessagingController.java:115)
    at com.fsck.k9.controller.MessagingController$1.run(MessagingController.java:173)
    at java.lang.Thread.run(Thread.java:761)
Caused by: com.fsck.k9.mail.MessagingException: Error appending message: **subject**
    at com.fsck.k9.mailstore.LocalFolder.saveMessage(LocalFolder.java:1326)
    at com.fsck.k9.mailstore.LocalFolder$14.doDbWork(LocalFolder.java:1187)
    at com.fsck.k9.mailstore.LocalFolder$14.doDbWork(LocalFolder.java:1182)
    at com.fsck.k9.mailstore.LockableDatabase.execute(LockableDatabase.java:275)
    at com.fsck.k9.mailstore.LocalFolder.appendMessages(LocalFolder.java:1182)
    at com.fsck.k9.mailstore.LocalFolder.appendMessages(LocalFolder.java:1109)
    at com.fsck.k9.mailstore.LocalFolder$12.doDbWork(LocalFolder.java:1082)
    ... 13 more
Caused by: java.lang.NullPointerException: Attempt to invoke interface method 'boolean java.util.List.add(java.lang.Object)' on a null object reference
    at com.fsck.k9.mail.internet.MessageExtractor.findHtmlPart(MessageExtractor.java:385)
    at com.fsck.k9.mail.internet.MessageExtractor.findViewablesAndAttachments(MessageExtractor.java:159)
    at com.fsck.k9.message.extractors.AttachmentCounter.getAttachmentCount(AttachmentCounter.java:33)
    at com.fsck.k9.mailstore.LocalFolder.saveMessage(LocalFolder.java:1259)
    ... 19 more

[xenithorb]

I assume none of this has shipped yet in 5.115 on F-Droid?

The behavior I'm noticing is that K9 recognizes that there's an embedded .ics file but is always zero-byte and crashes the calendar handler.

However, for actual ics attachments, as such from gmail and friends, the handler correctly passes off the attachment to the calendar app and the appointment is correctly added.

So I assume this is all about parsing the embedded "non-attached" ics?

[philipwhiuk]

Correct - it's not been merged. It won't be merged until after the next stable release (5.2x) because I really want us to do a release, we are way overdue, and I don't create new blockers.

I'm not exactly sure what you mean by embedded - Outlook for example still creates attachments, it's just the UI presents it nicely. K-9 certainly should be passing off the ICS files correctly to the handler in 5.115

[brad-x]

Another mechanism in use is to simply put the vcalendar contents into the body of an email message (this is not Outlook compatible but seems to be in adoption among other mail clients/systems).

It looks like K-9 represents both the inline BEGIN:VCALENDAR and the MIME attachment as two separate attachments. So I think that's in the realm of what he means.

[vasyugan]

Hi, a comment says this was planned for 5.2, but on 5.4 which my phone has, it is not there. Are there still plans to release it?

[codethief]

I would absolutely love to have .ics support as well but considering the fact that the previous comments are already two years old, this seems to be dead in the water.

[therges]

I would like to vote for this too!

[vasyugan]

why were the last four comments (including one by me) marked as "spam"?

[Valodim]

Because they add nothing to the technical discussion. Please use reactions for "voting".

To summarize the current status: There was some work on iCal at some point (see #2741), but it was never finished. We already have our hands full at the moment, and iCalendar support is not on our immediate roadmap. If you'd like to pick up the development for this feature, you are of course welcome to!

[vasyugan]

Because they add nothing to the technical discussion. Please use reactions for "voting".

That's a very loose definition of "spam". Usually that means unsolicited advertisement, not unwanted comments. E.g. my own comment:

Hi, a comment says this was planned for 5.2, but on 5.4 which my phone has, it is not there. Are there still plans to release it?

This is not a vote but a question. How is it "spam"? Rather than marking it spam you could e.g. say "We don't know" or "it's ready when it's ready" or even "we've given up on it". In any case there would be more clarity after.

[cketti]

@vasyugan: The issue tracker is not here to give you clarity. It's a tool we use for development.
K-9 Mail is an open source project run by volunteers. So the answer to all kinds of questions about future plans is either "maybe" or "it's done when it's done".
If you feel the need for further discussion please use the mailing list.

PS: The options for reasons why a comment should be hidden are limited. I hope you're happier with "off topic".

[d-tamm]

Dead for more than 2 years. Is there any ongoing work to get K9 handle invites nicely?
If implementing an Accept/Reject reply mechanism is not too complex, this should get priority (after showing the invitation's content).