Client: LAN Sync [Migrated from owncloud/core]

[LukasReschke]

Dear all,

I am trying owncloud and it looks really great, but there is one feature DropBox has and that I cannot find in owncloud: the Lan Sync, i.e. the possibility for clients under the same Lan to synchronize directly between themselves, which limit the bandwidth.

For organizations working in remote areas with slow connection, it really makes a difference and I was wondering if the feature does exist and if not, if it would be possible to implement it.

Thanks and best regards,
Pierre

Migrated from https://github.com/owncloud/core/issues/304

[RandolfCarter]

Isn't this the same as https://github.com/owncloud/mirall/issues/203 ?

[LukasReschke]

I don't think so. This issue seems to be about the LAN sync between clients like Dropbox has it.

https://www.dropbox.com/help/137/en

[RandolfCarter]

Ah yeah, this one is between two clients in the same LAN, while the other is for connecting differently between server and client according to current location of the client! Thanks for pointing it out!

[zatricky]

This feature's power is especially compelling when you have remote sites with poor connectivity. I've been using my laptop to get ease the bandwidth requirements with Dropbox - would give me less reason to keep Dropbox around if this was implemented in ownCloud. :)

I'm not sure how Dropbox implemented the feature. The KISS method I can think of is to report the private network and masks to the server - if there's a match, a simple cryptographic between clients, coordinated by the server, can confirm local connectivity.

[zatricky]

https://en.wikipedia.org/wiki/Local_Peer_Discovery (References refers to http://forum.utorrent.com/viewtopic.php?id=63567 which mentions some implementation detail used there)

The above is also a KISS approach - a simple cryptographic would "seal the deal".

[nloui]

FWIW - We setup an ownCloud server in-house and then a second one in the cloud where the data is sync'd via Bit Sync (a little convoluted) but achieves something similar. DNS within the network points to our in-house ownCloud and outside the network, it goes to the cloud-based server.

[etiess]

Hello,

I live and work in Africa, where bandwith is low and expensive: it would be a great point to have this "LAN sync". Is it part of the plans for owncloud?

Thanks!

Etienne

[danimo]

@etiess As you can see, this issue is not scheduled, so it's not on the near-term list.

[etiess]

Thank you @danimo for your feedback. We'll be patient!

[menelic]

+1 for LAN sync, this would really be an important feature in contexts of low bandwidth, high connection costs but not only there. In US, UK and GER, where internet is comparably cheap and fast, sync on LAN is still attractive for multiple device scenarios, eg a music or movies folder synced between laptops, tablets and phones.

As OwnCloud positions itself as a collaborative suite with teh documents app, sync on LAN becomes an important feature in that regard too...please consider giving it more priority

[Herve5]

+1 for LAN sync; for me it'd become a reason to switch an entire research lab here…
(indeed, it's currently the reason they refuse to at this moment…)

[jancborchardt]

Is this related to the 1.6 »Sync Performance« milestone? @MTRichards @dragotin @danimo

[MTRichards]

It is likely far more complicated than we can put into 1.6. Peer to peer sync is of course interesting, but for me a feature request much further out. And, interestingly, there are those who don't want this enabled, so it has to be configurable.

[willie68]

+1 for LAN sync, too;
I work with PC's, Netbook, Laptop, sometimes all within the same network. So a quick sync of a client over the LAN would be desirable. Actually i'm just waiting on my new laptop to sync all data. Just 7GB, but i think it will take 32hours... (The server has only a 384kbit upload rate) My netbook with all data in sync is just 20cm away.

[benedikt-buchert]

This seems to be an interesting solution to do this in case you are using your own server: http://adammatthews.co.uk/2013/05/owncloud-bit-torrent-sync-dropbox-clone/

[zatricky]

This brings across an interesting point: People have opted to bypass ownCloud's builtin "sync" entirely because it is inadequate? :-(

FWIW, I'm not experimenting with ownCloud any more. I realise ownCloud has uses _other_ than file sync - but btSync has had me sorted for a while already.

[Asuquo12]

+1 LAN Sync, This is very key for my organisation and for Owncloud as well. LAN Sync is the single reason we are looking at Dropbox. Please lets get this pushed up the priority ladder.

[burton1982]

+1 LAN Sync!!

[muetze-online]

Me too. Mostly my two synced Macs are at different locations, but often they are in the same network. If I sync 2 GB of new files, they will be uploaded to the internet - that must be - but then they will be downloaded for my second machine from the internet, allthough the data is present via a gigabit ethernet. When dropbox offered this feature, I was very happy. Now I'm on ownCloud - far beyond dropbox with some needed features (but better in an overall privacy relationship). You get the users with speed and simpleness.

[Herve5]

Muetze, I hope what we write here will be read (we have little else to ask since we are not developing ourselves indeed) but for regularly checking this feed, what I understand is, a guy name Deepdiver closed the action here referring to another thread, that he also closed the same day.
He seems to behave like he really don't like the issue.

Will Owncloud die by lack of simplicity I don't know. I definitely continue to use it since it is, well, the only open alternative… The world is not so large after all :-(

[muetze-online]

This issue is still open - see on the top. The other issue is closed, because it is a duplicate of this issue.

[tflidd]

There is even an open-source alternative to bittorrent-sync: http://syncthing.net/

[step4wd]

+1 for LAN Sync but is there any news/update on the roadmap?

[Herve5]

Still definitely interested in LanSync -for my younger contacts that's definitely the thing preventing them to abandon Dropbox, for instance.
Step4wd the last large update is quite recent IMHO (months), I didn't even switch myself. Let's hope :-)

[ghost]

Didn't see an answer here yet about the Dropbox way of LAN sync: Dropbox regularly sends broadcast packets to the network, announcing that it is running. What I don't know is how they communicate who has the most recent version of files... didn't inspect that yet, I only saw the announcement packets when wiresharking my network. LAN sync would be great!

[strugee]

Didn't see an answer here yet about the Dropbox way of LAN sync: Dropbox regularly sends broadcast packets to the network, announcing that it is running.

What kind of answer did you expect?

[zatricky]

@zeero4ever - from my comment on 27 Mar 2013:

https://en.wikipedia.org/wiki/Local_Peer_Discovery (References refers to http://forum.utorrent.com/viewtopic.php?id=63567 which mentions some implementation detail used there)

This isn't necessarily how Dropbox does it - but it is the most likely method since it is the simplest.

[oxivanisher]

Another question is, if the clients should transfer files without the knowledge of the server, or if the server even should orchestrate it. So that it tells clients A and B to transfer the file X between them. The fear a hacked client which can fetch all files from clients in the LAN could be ruled out this way.

[benedikt-buchert]

I think it would be nice to have peer to peer sync included into own cloud in general not just for lan. This could also result in a big speed increase for a folder shared with a group. If this could be switched off on client and server side would be amazing. I know that there is torrent sync to achieve this. But own cloud has some other advantages which i don’t want to be miss.

Am 08.01.2015 um 10:44 schrieb Marc Urben [email protected]:

Another question is, if the clients should transfer files without the knowledge of the server, or if the server even should orchestrate it. So that it tells clients A and B to transfer the file X between them. The fear a hacked client which can fetch all files from clients in the LAN could be ruled out this way.

—
Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/230#issuecomment-69156829.

[danimo]

@oxivanisher We want the server to have full control. That's why we can't implement this now. The server has to be prepared for it.

@eundd While pure peer-to-peer sync might be nice, it is not what we envision for ownCloud, as it puts the server operator out of control of the data. You may want to try bittorrent-sync or other solutions instead.

[Herve5]

The issue is not really peer-to-peer: it's the fact that, when two sync'd machines appear to be on the same LAN, Owncloud in this case only appears pathetic when compared to Dropbox, and this case happens very, very often both in research and industry.
I know personally a dozen people that'd obviously chose Owncloud against Dropbox and renounced just because of that very feature. It's to the extent I don't dare advertising for us, or at least very prudently now :-(

[sharadmittal]

I have a suggestion to implement this in a relatively easy way:
1) the current csync based approach does not allow for secure peer-peer chunk transfers.
2) How about we add a 2nd download option based on bittorrent protocol. libtorrent (C++) can be used for this.
3) libtorrent supports SSL cert based auth for the tracker, web seed and peers.
4) The owncloud server can act like the tracker and web seed. It can also issue peer certs to every client.
5) In libtorrent, we can add an extension (great support for that) which exchanges a token after the SSL handshake is complete. This token could be issued by the server to each peer for authenticating that the peer has access to the particular file id.
6) This way server has control over who gets the file while the peers can exchange the pieces among themselves in a secure authenticated way.

Reference - http://blog.libtorrent.org/2012/01/bittorrent-over-ssl/

Overall, this will drastically reduce the load on the server in terms of cpu and bandwidth, and make owncloud much more useful in low internet connectivity situations.

In terms of code complexity, we could integrate with libtorrent and add an extension which validates a small token. I don't imagine it being a huge effort.

Any comments/questions on the approach?

[danimo]

@sharadmittal This conflicts with the requirement that the server needs to have the last word on which client is allowed to received a certain file.

[Herve5]

@sharadmittal I'm 100% favorable to this idea; in case the issue described by danimo is a concern one could perhaps add a specific setting for enabling/disabling this
-but I must say I don't really understand that concern indeed, as long as the server does remain a central hub, to quote sharadmittal: "this way the server has control over who gets the file…"?

[sharadmittal]

Thanks @danimo for your feedback. How about the peers make a quick check with the server whether the connecting peer has access to that file (using the authenticated peer id)?

Even the server has no control after it gives out the file to a client. With what I'm suggesting, every client will check with the server before starting the file transfer to a peer.

[shoeper]

The client could send a list with the owncloud users in it's network (or just id's) with each download request. The server could then say ask client id x and use token y. The client requests the file from oc user x using token y via some encrypted channel. Before sending the file oc user x asks the server if the token is valid for the request file and if it was requested by the user wanting to have the file.

To get better speeds there could also be bulk mechanism where the client gets a token to receive 50 files and oc user x can verify that the requesting user has the permissions for those files with just one request.

And there could be a token valid for a day being used for folders being shared between two users.

[danimo]

The basic principle is clear, what's missing is an authorization protocol in the server. @LukasReschke and @DeepDiver1975 have a concept for that I think, but it's not there yet.

[sharadmittal]

It'll be great if @LukasReschke or @DeepDiver1975 could present/discuss their strategy here.

I've another simple way this could be done. Server encrypts the file once which then gets downloaded using a bittorrent like protocol. Clients have to get the encryption key from the server directly (using regular https authenticated route). This way the server gets to have the last word.
A side effect of this approach vs the original approach i suggested is that the peers have to keep 2 copies - one encrypted for sharing and other decrypted for local consumption. This issue can be reduced by keeping a time limit on how long a client needs to "seed".

This approach is also mentioned in here .. http://blog.libtorrent.org/2012/01/bittorrent-over-ssl/

[jalcine]

Realistically, ownCloud's designed afaik to have the server be the authorative source of all files. LAN Sync breaks this since there's no querying of the server about the status of the files updated. Now, if it was possible to ask the server what files have been changed from the last provided timestamp and use that to determine between local clients what to sync, that'd be dope. But again, authorization would have to be in the hands of the server. I'm assuming that a time-based key, not necessarily OATH but something like this can be issued to each client and used to identity each client to one another, thus (kind of) solving the issue of determining who's who. For the paranoid, an option can be provided to have the client get a new local sync token after a period of inactivity of local sync.

It's not an easy problem :wink:

[lordloh]

Drop box tan sync does need the cloud. So, the cloud is still the authorative source.

[sharadmittal]

@jalcine - I have covered the server authority in both the proposed solutions. At no point will a peer provide file bits before checking with the server or the bits being clear text.. Hence the server is practically and technically the first and final authority.

Regarding the file change, the peer sync only works for a particular file version... The clients will find the current version number/identification from the server and will only look for pieces of that version .. The time based token mentioned by you is definitely one approach .. The other approach is having the peers verify the connecting peer's cert/token every time with the server to be bulletproof...

[step4wd]

So in short this feature will never be introduced in ownCloud?

On Fri, Mar 6, 2015 at 12:12 PM, sharadmittal [email protected]
wrote:

@jalcine https://github.com/jalcine - I have covered the server
authority in both the proposed solutions. At no point will a peer provide
file bits before checking with the server or the bits being clear text..
Hence the server is practically and technically the first and final
authority.

—
Reply to this email directly or view it on GitHub
https://github.com/owncloud/client/issues/230#issuecomment-77516741.

[shoeper]

Where did you read that?

[jalcine]

So in short this feature will never be introduced in ownCloud?

That’s a hell of a stretch, don’t you think? This issue is more for
brainstorming an implementation strategy.

[dragotin]

Once we have checksums about file contents we can think on how to implement this. How is pretty straight forward: By doing the discovery against the server, the client gets to know which files it needs. With that information it can do a broadcast along the line "who can deliver me file with checksum xyz?". If it gets a positive answer, it might be able to download it from a different client rather than from the server.

On a general note, about the _"will never be implemented..."_ comment: ownCloud is a open source software project, so you can never say that. Somebody might step up and simply do it.

[sharadmittal]

Wanted to mention that libtorrent could potentially be used to have clients use the torrent protocol (with extensions). It supports SSL, client-server certificates and other stuff.

[jalcine]

Re: @dragotin

Once we have checksums about file contents we can think on how to implement this. How is pretty > straight forward: By doing the discovery against the server, the client gets to know which files it
needs. With that information it can do a broadcast along the line "who can deliver me file with
checksum xyz?". If it gets a positive answer, it might be able to download it from a different client
rather than from the server.

This sounds like a good idea, strategy wise to me, and actually does kind of sound like the act of torrenting, tbh. So this would require the server to accept a route that a client could send with the most recently changed checksums of files that the client has and determine which ones are out of sync then initiate that torrenting action.

I genuinely think something like this wouldn't be _insane_ to prototype, and thanks to the current flexibility of ownCloud, one could make a separate app and client app to test if this is functionality that could be made main-line.

[wizdude]

As a side note, Windows 10 has implemented "WUDO" - Windows Update Delivery Optimisation. This uses a bit-torrent style peer to peer delivery of updates between machines.

More and more product delivery and sync style products are now utilising local peer to peer technology. It would be great for Owncloud to consider the same.

Perhaps one solution would be for the server to have an index of all files that exist for each sync client. Some form of hash could be created for these that also incorporates the node IP address. This would be provided to the sync client at the same point where a file download would be occurring from the OC server. it could then try and transfer the file locally (with WAN failover as well). This index could be populated by the clients, perhaps by the use of a modified bittorrent style tracker.

This method, while obviously not the most optimal, would give the OC server the final say in all of this.

I'm personally a big fan of the way bitsync works. the server runs a tracker so each node knows where other nodes are. when content is synced it comes from LAN, WAN or wherever to give the best possible transfer rate.

[danimo]

@wizdude See https://github.com/owncloud/client/issues/230#issuecomment-74153667

[phil-davis]

+1 from me for something along these lines. Adding my use case here as it will be different from the "1st world" use cases.
In our use case we have very remote offices where we currently have a small server, pfSense router and OpenVPN back to main office. The remote office internet is really crud - comes and goes and when it is there it nearly always has 5-20% packet loss. Nepal Telecom is the only provider in those areas, so apart from crazy expensive satellite, some internet must be better than none, maybe.
In some towns they have local hydro power. We have had the power station or dam or... broken for weeks at a time. The local Nepal Telecom does not have enough solar/battery to run the phone system plus internet services, so the internet is off for a few weeks. Enough background - you get the idea.
It is proving very difficult to support any sort of "modern network" in these remote places - power troubles, server troubles, router troubles, cables pulled out by random staff... So we are thinking to simplify and give these offices just a basic internet device with WiFi (ADSL/WiFi/4-port LAN) combo device and no server...
If we can give them ownCloud to share files and have LAN sync, then a file changed by 1 user in a remote office can be uploaded once but then also update all the other laptops in the office locally.

And for us it would be really nice if local LAN sync could somehow happen when the internet is down. Somehow 2 clients can know which folders they have shared in common, and hold some authentication information that lets them trust that the other client is an authenticated client of the same server... If that would not be possible to build into the base ownCloud Client in a secure way, then an extra app would also be fine.

I am happy to be involved in helping with anything along these lines.

[bcbounders]

+1 I'm interested in this feature. I am mobile ( a lot! ) with bandwidth usage being a serious consideration. The fact that DropBox clients can sync between each other over a local LAN network means that they don't use up precious bandwidth. Lack of this feature in the OwnCloud clients is, at the moment, the only thing keeping me from making the switch.

[hvisage]

+1

;(
@danimo keeps hampering on the "Server must be authoritative" but that is an easy solve: the server gives the key to the files on the remote LAN client to fetch.

The protocol could be as simple as a broadcast (on UDP) that the other clients pick up, and report to the server those it saw. Packet could/should be encrypted/signed with the user's password and the servers certificate thus it could throw away those not trusted or related to the owncloud server instance.

Client ask for list of files to download from server, server provides list of other lan-clients (and their IPs) with the keys to the files to attempt to download from those lan-client, and if the lan-client doesn't respond/have the file, the client then fetch it from the server.

Q.E.D. imho, and still "KISS"...
with a torrent thingy, it would be the server that gives the client a key to download, and those keys could also be time based.

[danimo]

@hvisage And I do not disagree at all. Please see https://github.com/owncloud/client/issues/230#issuecomment-74153667.

[q2dg]

Suggestion: Why not add an option to Owncloud server that allows it to communicate with a specific (local and of choice) Torrent server and thus generate a magnet file for every content of an specific folder? (I'm thinking in distributing isos, for instance). The link to that magnet file could be displayed along the direct download link on web Owncloud serverś interface and then customers (if they select the magnet link instead of the direct link) could use any Torrent client to achieve that magnet, and, therefore, to download the associated content at the same time they share it via Torrent with other Torrent clients...(in other words: to do LAN Sync). Yes, it would be necessary a Torrent server and a Torrent client and Owncloud server would act simply as a "easy and web" intermediary...what maybe it's the most straightforward solution. What do you think?

[zephyrus00jp]

+1
It will be great to have this LAN sync option. I had to produce a presentation video from a power point file on a desktop PC and notice that it generates almost 1GB file.
This has to be synced with my home PC server. I understand that this can be slow. OK, I accept that.

However, then I realized my note PC next to it needs to sync it via the Internet: 1GB upload from my desktop and then 1GB download from the home server.
(If this is a _SINGLE_ file, I may simply copy the file into USB memory and later copy it to the Notebook
in a directory NOT under control of owncloud so that I won't have the chance of collision/corruption.
But in my use case, I often found a bunch of PDF files each measuring a few MBs from someone at once, and there are MANY OTHER REAL WOLRD USE CASES.)

As others have noted, Dropbox simply handles this by local LAN connection and I am glad for that.
Syncing with my home server can wait.

I am not entirely sure what Dropbox does (especially the communication with its server: it is encrypted using SSL, I think.), but as someone mentioned that there are peer-discovery and communication.
I found out about it when I checked the LAN activity for UPnP and other low-level lan activity.
Wireshark is your friend.

Its payload seems to contain the file list and other information that tells, based on my educated guess,
with server's PKI signature (to avoid forgery) [that is the client asks for the server to
create such a list with the server signature], that the client has the
files (timestamp, checksum) and the authority to hand out the files.
When the other client checks with the server and finds that it needs to download a set of files, it can
ask the local LAN first to see if other PCs have such files, and when one PC answers yes with the above list (and it is signed with server's key), it can begin copying the file using SSL using a key based on RSA/DH or whatever.
Of course, there may be a few more version-related information, etc. to make sure
the server can keep track of what version files were placed into each client with the LAN sync feature
and that the proper syncing will continue.

I hope the implementation of such a feature will be someone's new year's resolution :-)
With this LAN sync, I believe ownCloud will become more valuable: I have been using it share and sync about 30GB of data now without the fear of Dropbox forced to divulge part of its file storage by government pressure, etc. (After Snowden, nobody calls me a paranoia. Right?)

Thank you for sharing the great package.

[strugee]

Can someone please lock this issue to prevent people from +1ing?

[guruz]

Discussion now on https://central.owncloud.org/t/gsoc-2017-fast-lan-sync/6271

[PVince81]

We had a discussion with @ogoffart, @IljaN and @felixheidecke today about bittorrent sync. This was without reading the contents of this ticket.

Here is a dump of the topics we looked at for bittorrent download for clients:

• some research done, not exhaustive
  
  
  
  • seems possible to provide bittorrent tracker functionality over HTTP which could remove the need for a dedicated service
  
  
  • https://github.com/SuprDewd/simpletracker
  
  
  • bittorrent supports encryption
  
  
  • research needed: does bittorrent encryption prevents MITM between peers to prevent finding out the hashes
  
  
  • research needed: is there a way for hackers to download disallowed files by knowing their ids/hashes ?
  
  
• server
  
  
  
  • can provide HTTP endpoints to implement bittorrent tracker functionality
    
    
    
    
    
    • provides list of peers and a way for peers to register for a specific OC file
    
    
    
    • checks access permission
    
    
    
    • optionally: only let peers of a specific OC group sync between themselves (if that even makes sense security-wise)
    
    
    
  
  
  • provides Webdav property "oc:torrent-id" on every file/folder
  
  
  • possibly set a min file size limit for files to be torrented instead of any file
  
  
  • seeds original file:
    
    
    
    
    
    • should server also implement a torrent client to be able to seed ? (might need external service)
    
    
    
    • can server advertise itself as web link (with auth) through tracker instead ?
    
    
    
  
  
  • more research: authentication/security
  
  
• desktop client: implementation using Qt (@ogoffart please insert here)
  
  
  
  • read "oc:torrent-id" from PROPFIND
  
  
  • connect to tracker for download
  
  
  • else fallback to regular download
  
  
• LAN sync: likely offline without server/tracker ? might be another topic... if yes, need to separate bittorrent sync discussion to a separate ticket

Now maybe some questions were answered in the old thread:

• [ ] consolidate points list with discussion from the thread

[guruz]

• [ ] Look why this bittorrent makes more sense than extending on the rsync/zsync based way we planned for not-syncing-whole-file-from-or-to-server.

[individual-it]

What about synching protocol https://github.com/syncthing ?

[c0fe]

Anything new on this?

[volker-raschek]

Hello,
I can't understand why such an essential function, which was asked for 5 years ago, is not finally being implemented.

Instead, it implements every nifty gimmick, but the client's essential function is completely neglected.

In order to continue to stand up to other service providers, such a function should be implemented slowly.

[c0fe]

@volker-raschek i am not sure if the dev team will be working on this themselves. I can start to work on it but I am concerned that the work will be disregarded and it all will be for naught.

[michaelstingl]

@volker-raschek @c0fe did you check the Syncthing approach?

[c0fe]

@michaelstingl i have heard of it but haven't done all that much research into it yet.

[michaelstingl]

I'd love to hear feedback if Syncthing could be used here. Maybe basic integration isn't that hard…

[c0fe]

@michaelstingl I would assume this would require the installation of Syncthing in addition to Webserver PHP and Nextcloud and management would go through some sort of hooks.

The question is what interface does Syncthing supports and if it is possible to do this through straight PHP or would require something else in addition.

[michaelstingl]

I'm not sure if Syncthing on the server is really needed. Syncthing comes with a REST API which shouldn't be too hard too connect.

[c0fe]

@michaelstingl if they have a REST API then it shouldn't be difficult at all.

[EstebanSalpeter]

hello there, what about letting the client act as a proxy server, taking local copies as cached files?

[biva]

@MorrisJobke the development looks not active here. Do you still collaborate between owncloud and nextcloud? Would you still consider this development? In Africa, where internet is slow and data is expensive (illimited plans are very rare), it would be very helpful. Thanks a lot for your consideration!

[biva]

As I did not see any update about this LAN sync issue here, I created another issue on the Nextcloud side: https://github.com/nextcloud/desktop/issues/2010

Is there any update on your side? Thanks @michaelstingl for your involvement :)