Server: Expose subscribed webcal calendars via CalDAV

[grafenhofer]

The calendar app has recently gained support for subscribing to third party webcal calendars (see https://github.com/owncloud/calendar/issues/132). It would be great to expose these imported calendars via CalDAV (e.g. to mobile clients). Nextcloud would act as a kind of proxy service. One usecase is increased privacy for (mobile) users (only the Nextcloud server connection gets exposed to the original webcal provider).

Related owncloud issue: https://github.com/owncloud/core/issues/26193

[raimund-schluessler]

Implementing a server side handling of WebCal would also allow to query a subscribed calendar in certain intervals, e.g. monthly or once a year. This would reduce the server load on public WebCal servers as requested e.g. on http://www.ifeiertage.de/

[marmeladema]

I am also interrested in this feature.
My use case is that i have some friends sharing their google calendar with me and i don't want to rely on anotherr app on every devices and subscribe to all their ics calendars on all my devices.

Thank you !

[georgehrke]

Please refrain from commenting when you can't add anything new. Use Github Reactions instead. :)

[georgehrke]

According to @nickvergessen, Webcal was implemented natively in DavDroid. This massively decreases the priority of this issue. Postponing to 14.

[dsonck92]

I would still like to see this feature. It would be nice to have it like a cache and possibly alter certain aspects like editing reminder settings.

[ralin3]

Last I checked, which was yesterday, this requires installation of
another app.Davdroid just informs you of the calendars and then tells you to
install that app. (ICSDroid)

On Mon, Oct 30, 2017, at 17:57, Georg Ehrke wrote:

According to @nickvergessen[1], Webcal was implemented natively in
DavDroid. This massively decreases the priority of this issue.
Postponing to 14.> — You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub[2], or mute the
thread[3].>

Links:

1. https://github.com/nickvergessen
2. https://github.com/nextcloud/server/issues/1497#issuecomment-340530675
3. https://github.com/notifications/unsubscribe-auth/ABssJmuymkKIJXpjdWhL9MGZ3kNMkJRxks5sxg4OgaJpZM4KEr2g

[sbug-1bit]

In about a year I'm moving to a Purism - Librem 5 (GNU/Linux native mobile) which does not have DavDroid.

Also, no support for this means I have to manually add the calendars to all applications / devices I use it on.

I understand everything have different priorities, I just wanted to come with some input on the issue.

[dsonck92]

Also another motivation is: it would unify the way the web client can load the calendars. The whole iCal side of the client would not be necessary anymore.

[ralin3]

I'm on mobile so no edit on my post, but as a matter of fact, the only
thing that Davdroid does is push the url's of the subscripted calendars
to ICSDroid.Then ICSDroid polls the servers directly, and works regardless of/in no
sync with nextcloud.To say that Davdroid supports it then, is to my eyes, wrong.

This has already been postponed two or three major versions and is a
feature that should have been there from first implementation of webcal.
This works clientside if you use closed-source applications, and/or if
you ignore NC.

On Mon, Oct 30, 2017, at 18:06, spacebug0 wrote:

In about a year I'm moving to a Purism - Librem 5 (GNU/Linux native
mobile) which does not have DavDroid.> Also, no support for this means I have to manually add the calendars
to all applications / devices I use it on.> I understand everything have different priorities, I just wanted to
come with some input on the issue.> — You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub[1], or mute the
thread[2].>

Links:

1. https://github.com/nextcloud/server/issues/1497#issuecomment-340533618
2. https://github.com/notifications/unsubscribe-auth/ABssJs8U_lwq-V3rYQNZeX01mTYOhoPLks5sxhAigaJpZM4KEr2g

[Peque]

@ralin3 I have DAVdroid-1.9-ose installed (from F-Droid), but I cannot seem to find where it is informing about the subscribed calendars. Could you explain how that works? (or if you have the same version installed)

[Weishaupt]

@Peque you need to refresh your calendars, then the webcals will be displayed.
On the issue itself I'd also appreciate if this feature request would finally become reality.

[georgehrke]

Stay calm everyone ...
I only moved this issue to the next Milestone, i didn't close it ...

[ralin3]

It's already been postponed.....
The issue is over a year old, that was for NC10.

Now it's been postponed untill 14.

On Mon, Oct 30, 2017, at 19:54, Georg Ehrke wrote:

Stay calm everyone ...
I only moved this issue to the next Milestone, i didn't close it ...> — You are receiving this because you were mentioned. Reply to this
email directly, view it on GitHub[1], or mute the thread[2].>

Links:

1. https://github.com/nextcloud/server/issues/1497#issuecomment-340564455
2. https://github.com/notifications/unsubscribe-auth/ABssJnOK3BYNH9JvhnflJujXNswkR2nZks5sxilagaJpZM4KEr2g

[georgehrke]

Nextcloud is free software, so please actually help implementing stuff instead of telling how slow we are.

We only have limited time, so we can’t implement every user request in time. So implement it if u can’t live without it

And if u can’t program, find someone who can or add a bounty

[ralin3]

I'm not complaining about slowness, if this issue had been tagged NC14
when it was made last year I would be disappointed but that's that, I'd
keep my voice shut.
But continuously telling it will come in the next version for 4
consecutive versions is not a positive for the project - I'm already
waiting for this to be postponed to NC15.
Considering your partnership with puri.sm, as already pointed out -
Davdroid doesn't work for them, and the reasoning behind postponing to
14 is also flawed, and this is apparently a very requested feature - it
also leaves the calendar app looking halfway done.I can't imagine users thinking it's okay to not view some of the
calendars on a mobile device...Do you also tell enterprise users to implement the fixes for themselves?

On Mon, Oct 30, 2017, at 21:17, Georg Ehrke wrote:

Nextcloud is free software, so please actually help implementing stuff
instead of telling how slow we are.> We only have limited time, so we can’t implement every user request in
time. So implement it if I can’t live without it> — You are receiving this because you were mentioned. Reply to this
email directly, view it on GitHub[1], or mute the thread[2].>

Links:

1. https://github.com/nextcloud/server/issues/1497#issuecomment-340587269
2. https://github.com/notifications/unsubscribe-auth/ABssJiGfFSZMVQ5mlCOYauxam6xscOWaks5sxjzygaJpZM4KEr2g

[theraser]

I think I just used this (missing) feature successfully - was it silently added in the meantime or is this issue about something different? I just hit this issue while searching for something completely different, didn't even know that this issue exists.

After adding an external hosted .ics file as a calendar subscription I was able to sync it using my Android device using the app "CalDav-Sync". No special configuration required. My aCalendar app shows every event just as expected.

[rubdos]

@theraser, cannot reproduce on SailfishOS; I suppose that CalDav-Sync handles the special case just like DavDroid then?

[georgehrke]

I think I just used this (missing) feature successfully - was it silently added in the meantime or is this issue about something different?

No, it wasn't implemented yet. Technically, WebCal calendars are already exposed via CalDAV as a special collection type called subscription. But that's not part of the CalDAV standard. Therefore it's only supported by a limited variety of Calendar applications like iOS, macOS and apparently certain Android apps.

Once this feature is implemented, WebCal calendars will be cached on the Nextcloud server and exposed as ordinary calendar collections that any caldav client will understand.

[moritz31]

What would be the basic steps to implement this feature, is this something a guy with only java experience can handle? Really missing this feature, thought about a java linux client which sync the webcal links, but that would be the wrong way round i guess

[georgehrke]

What would be the basic steps to implement this feature

My approach would be the following:

• add a database table structured like oc_calendarobjects for subscriptions. Call it oc_calendarsubscrobjects or something like that.
• create a class OCA\DAV\CalDAV\CachedSubscription that inherits from \Sabre\CalDAV\Calendar. Don't inherit it from OCA\DAV\CalDAV\Calendar directly, because it's shareable and making the CachedSubscription shareable will require quite some amount of work, so let's do one step after the other and add shareability later on.
• extend OCA\DAV\CalDAV\CalDAVBackend with methods to read and write from and into the oc_calendarsubscrobjects table. Use these methods in the CachedSubscription class.
• Add a background job, that periodically pulls the webcal url stored in oc_calendarsubscriptions, parse it, split it up (using the Sabre VObject Splitter) and write the individual objects into the oc_calendarsubscrobjects table. It's ok if you just delete all other objects of that subscription in oc_calendarsubscrobjects in advance, you don't have to compare single objects for changes because the calendar is read-only anyway. (Use a TimedJob for that)

Nice to have:

• respect the refresh rate given in oc_calendarsubscriptions instead of pulling with some hard coded refresh rate
• if the refresh rate is null, check the ics data for properties like X- PUBLISHED-TTL or REFRESH-INTERVAL and update the oc_calendarsubscriptions table correspondingly.
• respect the strip todo / alarms / attachments properties from oc_calendarsubscriptions

One of the most important things is to persevere the full subscription functionality for clients that actually do support our current way of handling of subscriptions. So please don't delete any subscription code, but allow the code to run side by side. The best way to distinguish if a client supports it would be by checking with a user agent regex respectively checking for a certain header.

if you want to tackle this issue, I'd be happy to guide if you have any further questions.

[moritz31]

would be generally intrested in introducing this new feature, but have to look, quiete busy weeks at university right now

[leandro-lucarella-sociomantic]

I know it might be a stretch, but what about also adding an option to set reminders for external calendars like Google Calendar does? No need to go with a per-event reminder, just setting global reminders that apply to all events in the external calendar would do.

[leandro-lucarella-sociomantic]

(OK, maybe that's just a different feature request, not really dependent on having this feature)

[leandro-lucarella-sociomantic]

https://github.com/nextcloud/server/issues/9703

[georgehrke]

@moritz31 Did you make any progress on this?
Otherwise I would take over so we get this into Nextcloud 14

[moritz31]

@georgehrke would be cool if you take over, i get next free time at end of july, maybe i can then help a bit

[georgehrke]

You all can help resolve this issue by telling me what clients don't support subscriptions natively:
Clients that do natively support it: iOS, macOS, Fantastical, DavDroid

Clients that don't:
Thunderbird does not support it.

What about:

• [ ] KOrganizer
• [ ] Evolution

What other important clients did i forget about?
If you can, it would be great if you could provide the http user-agent sent by this application.

[KwadroNaut]

Clients that do natively support it: iOS, macOS, Fantastical, DavDroid

From DavDroids website: "Since DAVdroid 1.8, DAVdroid can detect CalDAV-advertised Webcal feeds and integrate them using ICSdroid." That's not native, that's after having installed ICSdroid, otherwise it's CalDAV only.

[georgehrke]

From DavDroids website: "Since DAVdroid 1.8, DAVdroid can detect CalDAV-advertised Webcal feeds and integrate them using ICSdroid." That's not native, that's after having installed ICSdroid, otherwise it's CalDAV only.

As far as I know ICSDroid is integrated into DavDroid. (both are GPL licensed apps from the same developer)

Can someone verify that?

(I wouldn't mind enabling the caching for DavDroid as well though, because they don't support creating subscriptions, they can only read them)

[simonspa]

It's not "integrated" but a separate app. The subscription from Nextcloud ICS via DAVDroid however works flawlessly.

[moritz31]

What other important clients did i forget about?
If you can, it would be great if you could provide the http user-agent sent by this application.

I use gnome calendar on my desktop :) can try to send you the user-agent later

[Peque]

Otherwise I would take over so we get this into Nextcloud 14

@georgehrke As you are already rolling out the beta, I guess this is not going to be the case?

PS: I would like to point out that this issue is currently the one with the highest number of :+1: and :heart: reactions. Maybe you realized already, but just in case you did not and @nextcloud organization wants to take that into account. :innocent:

[W-Hamra]

You all can help resolve this issue by telling me what clients don't support subscriptions natively

KOrganiser does not support refreshing a web based ical resource. It can refresh from a local ical file only. If given a web address, it will download it and use it locally (Though that always causes akonadi to crash on my system)

[georgehrke]

I use gnome calendar on my desktop :) can try to send you the user-agent later

ping @moritz31 :)

[schaten]

Davdroid seems to use this useragent: "DAVdroid/2.0.7-ose (2018/12/23; dav4android; okhttp/3.12.0) Android/8.1.0"

[nickvergessen]

As per https://gitlab.com/bitfireAT/davx5-ose/commit/7ce739c271f9b4f9b3cd7c16c526a774a7d99bdc it is now:
DAVx5/2.0.7-ose (2018/12/23; dav4android; okhttp/3.12.0) Android/8.1.0
and it might change to DAVx5/2.0.7-ose (2018/12/23; dav4jvm; okhttp/3.12.0) Android/8.1.0 soon.

[Programie]

I've some Ubuntu installations with KOrganizer and Evolution/GNOME Calendar which also don't support CalDAV Subscriptions.

KOrganizer: Mozilla/5.0 (X11; Linux x86_64) KHTML/5.50.0 (like Gecko) Konqueror/5 KIO/5.50
Evolution and GNOME Calendar: Evolution/3.30.1

Adding /^Mozilla\/5\.0 \(X11; Linux x86_64\) KHTML\/([0-9\.]+) \(like Gecko\) Konqueror\/5 KIO\/([0-9\.]+)$/ as regex to "ENABLE_FOR_CLIENTS" in the WebcalCaching plugin works for me in KOrganizer. But the regex should probably extended to be more generic. For Evolution it works by just using /^Evolution\/.*$/.

Edit: For Evolution/GNOME Calendar it only works if adding the CalDAV account as a collection account in Evolution instead of the GNOME Online Accounts. The difference: Configuring it as GNOME Online Account uses remote.php/caldav instead of remote.php/dav which seems to not using the WebcalCaching plugin.

[army1349]

Guys, did anyone test what will happen with modified ENABLE_FOR_CLIENTS during upgrade?

[Programie]

@army1349: The updater will overwrite the changes, so you have to edit the file again.

[army1349]

@army1349: The updater will overwrite the changes, so you have to edit the file again.

That's what I was afraid of. Kind of weird.

[redtux]

It says that this issue has been fixed by #10059, so I just tried to get sync my imported ical on my smartphone via DAVx, and it worked. :+1:

BUT: I already had imported the same calendar before directly in ICSx, so I have two calendars now using exact the same external ical link. I am afraid this not precisely what @grafenhofer was referring to when opening this issue. :smirk_cat:

Nextcloud now still is not a "proxy service" for ical, there is no "increased privacy", and the load on the servers providing the calendar is the same as before. Also inside Nextcloud, when trying to download the calendar, you get no Nextcloud link, but are simply redirected to the original ical link. Apart from this, I cannot share those calendars inside Nextcloud.

This has been tested on Nextcloud Server 16.0.4, and Nextcloud Android App 3.7.2, DAVx (DAVdroid) 2.5.4.1-ose (299) & ICSx 1.8.4-standard.

[kesselb]

@redtux please create a new issue.

[PopeRigby]

Should I go ahead and make a new issue for this? I'd like this functionality.

[georgehrke]

Please see the discussion above about ENABLE_FOR_CLIENTS. We haven't enabled it for any clients so far.

Some people contributed user-agents for various clients. Thanks for that!
So we can enable it for those clients with Nextcloud 18. But the list is still rather limited.