Calendar: Calendar Import, numerous failures

[ll3N1GmAll]

Steps to reproduce

1. export .ICS file from owncloud
2. Import .ICS file to Nextcloud calendar

Expected behaviour

Import successful

Actual behaviour

a small percentage of items import successfully after dozens of attempts to import the ICS file. Now I get 0 success, a large number failed and a small number skipped because they are duplicates.

Server configuration

Operating system:
ubuntu 18.04
Web server:
nginx
Database:
mysql / mariadb
PHP version:
7.2
Server version: (see your admin page)
13.0.4
Calendar version: (see the apps page)
1.6.1
Updated from an older installed version or fresh install:
fresh install
Signing status (ownCloud/Nextcloud 9.0 and above):
Not sure what this is asking for. I entered the Nextcloud version 13.0.4 above.

Login as admin user into your cloud and access
http://example.com/index.php/settings/integrity/failed
paste the results here.

No errors have been found.

List of activated apps:

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your instance's installation folder

Enabled:

• activity: 2.6.1
• audioplayer: 2.3.1
• bruteforcesettings: 1.1.0
• calendar: 1.6.1
• comments: 1.3.0
• contacts: 2.1.5
• dav: 1.4.7
• federatedfilesharing: 1.3.1
• federation: 1.3.0
• files: 1.8.0
• files_sharing: 1.5.0
• files_texteditor: 2.5.1
• files_trashbin: 1.3.0
• files_versions: 1.6.0
• files_videoplayer: 1.2.0
• firstrunwizard: 2.2.1
• gallery: 18.0.0
• logreader: 2.0.0
• lookup_server_connector: 1.1.0
• nextcloud_announcements: 1.2.0
• notifications: 2.1.2
• oauth2: 1.1.1
• onlyoffice: 1.3.0
• password_policy: 1.3.0
• provisioning_api: 1.3.0
• serverinfo: 1.3.0
• sharebymail: 1.3.0
• spreed: 3.2.4
• survey_client: 1.1.0
• systemtags: 1.3.0
• tasks: 0.9.6
• theming: 1.4.5
• twofactor_backupcodes: 1.2.3
• twofactor_totp: 1.4.1
• updatenotification: 1.3.0
• workflowengine: 1.3.0
  Disabled:
• admin_audit
• encryption
• files_external
• files_pdfviewer
• user_external
• user_ldap

Nextcloud configuration:

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your instance's installation folder
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "www.MYDOMAIN.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/www.MYDOMAIN.com",
        "dbtype": "mysql",
        "version": "13.0.4.0",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true
    }
}

or

Insert your config.php content here
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

Are you using external storage, if yes which one: local/smb/sftp/...
no
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no

LDAP configuration (delete this part if not used)

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your instance's installation folder


  [Symfony\Component\Console\Exception\CommandNotFoundException]  
  There are no commands defined in the "ldap" namespace.          


Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Be sure to replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:
Firefox (linux & windows), Chromium (linux)
Operating system:
Ubuntu & Debian
CalDAV-clients:
thunderbird and davdroid

Logs

Web server error log

Insert your webserver log here

Log file (data/nextcloud.log)

Insert your nextcloud.log file here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

I have tried to split the ICS file into many smaller files, even one entry per file without success. I am getting 1 failed or 1 duplicate as a response. Even with small files that are a portion of the original ICS it shows 0 success. When this failure began I was able to run the import over and over again and the result would be that a few or a few dozen calendar items would import successfully. Now none import successfully; even when trying from different computers (windows and linux) and after clearing all browser data out.

Error text: "Imported 0 out of 1238, 948 failures, skipped 290 duplicates"
My ICS file is 1 MB (but as I said before, files with only one entry are even failing at this point)

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[georgehrke]

Please provide the log file and browser log as requested. Otherwise we are just digging in the dark ...

[ll3N1GmAll]

Thank you for your response. Sorry about that.
Here is the console log when reproducing the issue:
Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src”). Source: (function (NAVIGATOR, OBJECT) {

OBJ.... calendar:1

Content Security Policy: The page’s settings blocked the loading of a resource at self (“script-src”). Source: (function (ERROR) {

const V8_STACK_.... calendar:1

JQMIGRATE: Migrate is installed, version 1.4.0 core.js:7:542
Deprecation warning: use moment.updateLocale(localeName, config) to change an existing locale. moment.defineLocale(localeName, config) should only be used for creating a new locale See http://momentjs.com/guides/#/warnings/define-locale/ for more info. core.js:1563:2834
Attempting to call a FullCalendar method on an element with no calendar.
vendor.min.js:10:21215
Source map error: TypeError: NetworkError when attempting to fetch resource.
Resource URL: https://www.MYDOMAIN.com/core/vendor/core.js?v=797fc0dc-3
Source Map URL: purify.min.js.map[Learn More]
Source map error: TypeError: NetworkError when attempting to fetch resource.
Resource URL: https://www.MYDOMAIN.com/apps/calendar/js/public/vendor.min.js?v=797fc0dc-3
Source Map URL: vendor.min.js.map[Learn More]

Here are the nextcloud.log contents:
{"reqId":"mypN7F5m1KMQT0u7qkBz","level":4,"time":"2018-07-16T23:34:58+00:00","remoteAddr":"REMOTE_IP","user":"user","app":"webdav","method":"PUT","url":"\/remote.php\/dav\/calendars\/usere\/personal\/Nextcloud-RAPPG6X24BJFMAHGWN9DE.ics","message":"Exception: {\"Exception\":\"Sabre\\DAV\\Exception\\BadRequest\",\"Message\":\"Calendar object with uid already exists in this calendar collection.\",\"Code\":0,\"Trace\":\"#0 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/CalDAV\\/Calendar.php(201): OCA\\DAV\\CalDAV\\CalDavBackend->createCalendarObject('1', 'Nextcloud-RAPPG...', 'BEGIN:VCALENDAR...')\n#1 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(1096): Sabre\\CalDAV\\Calendar->createFile('Nextcloud-RAPPG...', 'BEGIN:VCALENDAR...')\n#2 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/CorePlugin.php(529): Sabre\\DAV\\Server->createFile('calendars\\/user...', 'BEGIN:VCALENDAR...', NULL)\n#3 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#4 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/event\\/lib\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#5 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#6 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#7 \\/usr\\/share\\/nginx\\/nextcloud\\/apps\\/dav\\/lib\\/Server.php(287): Sabre\\DAV\\Server->exec()\n#8 \\/usr\\/share\\/nginx\\/nextcloud\\/apps\\/dav\\/appinfo\\/v2\\/remote.php(35): OCA\\DAV\\Server->exec()\n#9 \\/usr\\/share\\/nginx\\/nextcloud\\/remote.php(164): require_once('\\/usr\\/share\\/ngin...')\n#10 {main}\",\"File\":\"\\/usr\\/share\\/nginx\\/nextcloud\\/apps\\/dav\\/lib\\/CalDAV\\/CalDavBackend.php\",\"Line\":988}","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"13.0.4.0"}
{"reqId":"8c9dlH04V2FA1QIPctlc","level":4,"time":"2018-07-16T23:35:00+00:00","remoteAddr":"REMOTE_IP","user":"user","app":"webdav","method":"PUT","url":"\/remote.php\/dav\/calendars\/usere\/personal\/Nextcloud-U990J2YAZ1GIFI37AEQ4U.ics","message":"Exception: {\"Exception\":\"Sabre\\DAV\\Exception\\BadRequest\",\"Message\":\"Calendar object with uid already exists in this calendar collection.\",\"Code\":0,\"Trace\":\"#0 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/CalDAV\\/Calendar.php(201): OCA\\DAV\\CalDAV\\CalDavBackend->createCalendarObject('1', 'Nextcloud-U990J...', 'BEGIN:VCALENDAR...')\n#1 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(1096): Sabre\\CalDAV\\Calendar->createFile('Nextcloud-U990J...', 'BEGIN:VCALENDAR...')\n#2 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/CorePlugin.php(529): Sabre\\DAV\\Server->createFile('calendars\\/user...', 'BEGIN:VCALENDAR...', NULL)\n#3 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#4 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/event\\/lib\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#5 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#6 \\/usr\\/share\\/nginx\\/nextcloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#7 \\/usr\\/share\\/nginx\\/nextcloud\\/apps\\/dav\\/lib\\/Server.php(287): Sabre\\DAV\\Server->exec()\n#8 \\/usr\\/share\\/nginx\\/nextcloud\\/apps\\/dav\\/appinfo\\/v2\\/remote.php(35): OCA\\DAV\\Server->exec()\n#9 \\/usr\\/share\\/nginx\\/nextcloud\\/remote.php(164): require_once('\\/usr\\/share\\/ngin...')\n#10 {main}\",\"File\":\"\\/usr\\/share\\/nginx\\/nextcloud\\/apps\\/dav\\/lib\\/CalDAV\\/CalDavBackend.php\",\"Line\":988}","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko\/20100101 Firefox\/61.0","version":"13.0.4.0"}

[TheBigFatTony]

I experienced the same issue on NC15. This is my workaround:

I simply imported the same calendar file a few times in a row into the same NC-calendar and copied the message to check whether all events were imported.

This is how it looked like:

Imported 849 out of 1762, 913 failures
Imported 867 out of 1762, 68 failures, skipped 827 duplicates
Imported 5 out of 1762, 1435 failures, skipped 322 duplicates
Imported 38 out of 1762, 171 failures, skipped 1553 duplicates
Imported 2 out of 1762, 124 failures, skipped 1636 duplicates
Imported 1 out of 1762, 98 failures, skipped 1663 duplicates

TOTAL: 849+867+5+38+2+1 = 1762

It's exploiting the ability of NC to detect duplicates.

Needless to say, this is ridiculous - but it works. Hope this helps anyone.

[rponline]

I experienced the same issue on NC15. This is my workaround:

I simply imported the same calendar file a few times in a row into the same NC-calendar and copied the message to check whether all events were imported.

This is how it looked like:

Imported 849 out of 1762, 913 failures
Imported 867 out of 1762, 68 failures, skipped 827 duplicates
Imported 5 out of 1762, 1435 failures, skipped 322 duplicates
Imported 38 out of 1762, 171 failures, skipped 1553 duplicates
Imported 2 out of 1762, 124 failures, skipped 1636 duplicates
Imported 1 out of 1762, 98 failures, skipped 1663 duplicates
TOTAL: 849+867+5+38+2+1 = 1762

It's exploiting the ability of NC to detect duplicates.

Needless to say, this is ridiculous - but it works. Hope this helps anyone.

I found a - also ridiculous - but still more comfortable solution:

The problem is, that the client sends a huge amout of PUT requests to the server. At some point the webserver refuses to handle the requests in a proper manner.

So, I basically used the connection throtteling feature of Firefox to send these http-requests very slowly, ..... so the server can handle the load (GPRS speed ;))

The "good solution" would be, to send the ics to the server in a whole piece and let the server do the work.

The easy fix would be, to send the put-requests after one another.

[danst0]

Yes please. I am now uploading my ical file for the 20th or so time.

[rponline]

@georgehrke You can remove the _needs info_ tag since it's pretty clear now what's the problem here:
Every event in an uploaded ics file sends a PUT request to the server.
Many (small) servers can't handle that many PUT requests in this short amount of time.

There are two possible solutions:

• Upload the ics file as a whole to the server and let the backend handle the events.
  (... this would be the faster solution)

• Or adding some kind of rate limiting to the client side script, so the server can handle all the requets. Which would get more conform to the caldav featureset, but since we are talking about an integrated webclient for nextcloud, this seems to be the more hacky solution to me.

[georgehrke]

Duplicate of #445

[CuriousCocainist]

Hi,
It looks like an old closed issue , nevertheless I experience similar issue with importing .ics calendars in Nextcloud 17... There is basically no reaction, and console prints:

user changed files calendar.js:2:2268413

I use Nextcloudpi on SBC (ARM). Files were exported from Calendar ver 1.6.6 (NC 15.0.14) and I tried to import them into its ver 2.0.1 (NC 17.0.2)

[tcitworld]

@CuriousCocainist You probably have https://github.com/nextcloud/calendar/issues/1898

[CuriousCocainist]

@tcitworld
You may be right, I'm still not sure... nevertheless import problem is gone after updating NC to version 18.0.1

[warsix4]

18.0.1 here and unable to Import an .ics file at all. No response from the UI after selecting a file. Nothing at all happens in browser console.

[georgehrke]

This is a multiple years old issue that has been resolved long time ago.

If you have trouble importing ics files, check if it's a duplicate of #1898 or open a new issue __using the issue template__ please.