Core: keeping webinterface in sync

[jancborchardt]

Currently the webinterface doesn’t poll for changes, except when reloading, changing apps or folders. This can lead to the displayed data being heavily outdated: for example when you have the web app open, sync some files using the desktop client (or the web app open on another computer) these changes do not appear in the originally open web app.
As this is a problem with not only Files but all apps, it should be a core component.

Thoughts: Websockets, API, post-upload hooks. cc @DeepDiver1975 @karlitschek

[DeepDiver1975]

FYI:
php web sockets:
http://socketo.me/docs/hello-world

web sockets with angular:
http://www.html5rocks.com/en/tutorials/frameworks/angular-websockets/

[DeepDiver1975]

Why the angular link above?

We need to move all display related code to the client/javascript - doing this without a javascript framework is just crazy. Some of us already have good experience in this area.

Would be great to see a rapid prototype on this - maybe the files app would be a great show case.

(rapid prototype means we will throw it away! ;-) )

[jancborchardt]

Calling in @Raydiation and @butonic for Angular.

[BernhardPosselt]

I have experience with AngularJS and websockets and I can highly recommend it. If people want to, I could provide an example in the advanced example. Im planning on implementing an model parentclass anyway, which does all the caching for you

PS: The linked example broadcasts the received messages, I ran into problems with this approach. I created a publish/subscriber object where the models registerered themselves for certain messages. The websocket then just publishes the messages over that object.

[DeepDiver1975]

@Raydiation looking forward to see the demo. As stated above files app would be a good candidate.

@bartv2 please share your thoughts on this. THX

[bartv2]

Yes, using websockets looks good.
Maybe use the cache for communicating between different request (webdav->files app). This is already used for the import progress in calendar and contacts

[karlitschek]

Please keep in mind that websockets are not fully supported by all browsers that we have to support. Another challenge with websockets is that connection to the webserver are keept open which is a problem for the scalability of big installations. so please imlement a config switch in config.php to turn it of optionally

[BernhardPosselt]

I'm not sure how big that impact will be, in general it will reduce a lot of bandwidth I think, since we dont have to poll every 30-60 seconds. We could also transition gracefully to a fallback if websockets are not supported by having the fallback implement the same interface.

PS: theres a problem with firefox, when you hit ESC it will close the websocket (known, but low priority bug)

[karlitschek]

Polling is actually something that can be handled well with most server setups. open connections are the problem

[jancborchardt]

If websockets are not supported in a browser – then well, automatic updating simply doesn’t work. That’s ok because currently ownCloud does that in _no_ browser. Plus, this ability should be automatically detected via feature detection and not set by a hard config switch.

Postponing to milestone ownCloud 7.

[MorrisJobke]

Postponing to milestone ownCloud 8

[brantje]

@MorrisJobke Maybe postponing this to OC9? Don't seem to be much progress here.
Websockets are awesome, and OC can use them for sure.
Browser support won't be any problem, http://caniuse.com/#feat=websockets
Only IE9 does not support it, but we can solve that.

Edit: Stupid close and comment button -_-'

[PVince81]

@brantje AFAIK php doesn't support web sockets. You'd need to setup a separate Node JS server or any other server that supports it, and then make that one poll the ownCloud server for changes or make the server push changes to that server.

And in general requiring users to setup a separate server goes against the "simply setup" philosophy.

[DeepDiver1975]

another option would be to go for a polling approach similar to the desktop client.
Performing a PROPFIND on the current folder (or root ?) to detect changes based on the etag change.

In order not to DDOS the server the polling should be disabled as soon as the browser window/tab is not active - we have something like this already in place for the free space check - see https://github.com/owncloud/core/blob/master/apps/files/js/files.js#L270

[DeepDiver1975]

@PVince81 ^ does that make sense?

[PVince81]

Yes, makes sense.

For this the list.php endpoint should also return the current folder's fileinfo (including etag).
If we switch to WebDAV at some point that information will be included automatically.

[DeepDiver1975]

If we switch to WebDAV at some point that information will be included automatically.

sounds like time to switch :wink:

There is one remaining issue to be solved: pagination - which is not part of WebDAV.
I was thinking about introducing owncloud specific headers for this purpose like

OC-Page: 5
OC-PageSize: 20

[PVince81]

Providing pagination in list.php and/or WebDAV is another topic, let's move it here: https://github.com/owncloud/core/issues/13915

[oparoz]

The problem with polling is that you get it wrong half the time, so users will still hit refresh after having uploaded files to be sure the view picked it up.The net result is a lot of effort to build something which won't be really useful imho.

I don't think there is a way around introducing a new requirement. I vote for using a php redis extension to push events via eventsource.

[icewind1991]

Implemented this over the weekend: https://github.com/icewind1991/files_live_reload

Using eventsource in combination with a memcache based buffer

[oparoz]

Cool :)

[DeepDiver1975]

Implemented this over the weekend: https://github.com/icewind1991/files_live_reload

Using eventsource in combination with a memcache based buffer

Really nice - but I think this is not the right way to do it, because this introduces yet another code path into the files app and we are trying to move the files app purely on webdav.

We need to implement some light-weight sync algo in js within the files app - once we did move over to webdav ...

[oparoz]

You can wait a long time before WebDAV does push though:
https://tools.ietf.org/html/draft-pot-webdav-notifications-00

So that means polling, which isn't great.

As a plugin, as designed in files_live_reload, the Files app keeps using the endpoints it currently uses, but knows in real-time when changes are happening.

[DeepDiver1975]

Push is anyhow not an option for us as multiple times discussed.

Any push approach requires a long living php process which is a nightmare in big installations.

So the only thing we can do is polling .. which is by the way what all mechanisms use (caldav, carddav, our desktop client ......)

[oparoz]

Push is anyhow not an option for us as multiple times discussed.

Ah, sorry, I missed that discussion. Seems the only way to do push would be to use a daemon available on another port or something similar...

Any push approach requires a long living php process which is a nightmare in big installations.

I was wondering how it would scale. It would require a machine with a lot of RAM...

which is by the way what all mechanisms use (caldav, carddav, our desktop client ......)

What about mobile clients?

[PVince81]

9.0 now uses webdav.

Here is an idea:

• poll using a PROPFIND with Depth: 0 that requests the etag on the current directory
• compare etag with old one to see whether there were changes
• optional: get rid of "getstoragestats.php" that already polls and add the "quota-available-bytes" and "quota-used-bytes" properties to the above PROPFIND (kill two bird with one stone)

That's it for update detection.
Regarding the UI, I'd still vote for simply displaying an orange notification "The contents of the folder has changed, click here to refresh" instead of changing UI elements while the user has their pointer over them.

Moving to 9.1

[oparoz]

Regarding the UI, I'd still vote for simply displaying an orange notification "The contents of the folder has changed, click here to refresh" instead of changing UI elements while the user has their pointer over them.

The problem is that if someone is uploading lots of files, the user in front of the GUI will keep getting that message after every refresh, unless we can get realtime info about the upload and print a "upload in progress" message until we can print "upload completed, you can refresh now".

[PVince81]

You mean uploading outside of the web UI ? If uploading in the web UI, we can disable the refresh message and simply update the directory's known etag.

If they're uploading outside of the web UI, for example with the sync client, then they should get the message only once. Then it's their fault if they click it repeatedly :wink:

[oparoz]

If they're uploading outside of the web UI, for example with the sync client, then they should get the message only once. Then it's their fault if they click it repeatedly :wink:

But you'd have to let the user know that new files have been uploaded.

1. User logs in
2. Upload starts
3. Message is shown
4. User refreshes
5. Files are still being uploaded, so new files are now hidden until the user hits refresh again
6. Show message
7. User refreshes
8. More files are uploaded, message required
   etc.

[PVince81]

Side note: this is what KDE Dolphin does when new files appear.
This is why I prefer having a simple message "Folder has updated, refresh ?" instead of refreshing instantaneously or periodically and confusing the user:

[PVince81]

hint: I'm not scrolling, but a copy is being done from another computer into this folder

[michaelstingl]

superseded by owncloud/phoenix ?

[brunnomoreira]

Hi guys, I implemented this file's polling as a simple plugin, which I've tested and seems to work smooth.
Is that strategy according to the idea that you guys were proposing to that?

https://gitlab.com/anolisti/files_polling