Syncthing-android: Status Update: SD Card Support?

[TaoK]

Given SyncThing-android's behavior/capability of synchronizing entire folders/shares (and not partials), the ability to store synchronized shares on the SD card (in these days of mobile SDXC support) appears to be very commonly desired, even Core to the use of syncthing for many people.

Given the desire to actually use the files that are synchronized, having the synchronized folder be "private" to SyncThing (which is the only option today, for read/write shares) is not sufficient.

The current state and workarounds are noted in the FAQ (https://github.com/syncthing/syncthing-android/wiki/Frequently-Asked-Questions), but the text there could be substantuially improved on two counts:

• The statement "This is not yet supported by the Go language", while true, is misleading:
  
  
  
  • It implies that it will eventually be supported by the language - which, reading through the issues, appears to be be extremely unlikely
  
  
  • The Go team and the SyncThing team appear to have agreed on a way that this could/should be made to work without language-level support, instead having a small SyncThing-defined Java library being invoked from the Go code for the necessary write authorization intent interactions.
  
  
• The links to the Issues are a bit frustrating, given that the Go issue is closed, and the SyncThing-android issue cannot be commented on (since a year ago) and has seen no updates, despite the apparently-productive outcome of discussion with the Go team. It would be kind to have an accurate summary in the FAQ rather than inviting folks to read huge reams of mostly irrelevant comments.

Could we get an update in the FAQ and in Issue #29 reflecting the actual/current status please?

• Does the syncthing team intend to ever add support for synchronizing folders (two-way) on arbitrary locations on the SD card?
• What's the status, how can third parties contribute if at all?
  
  
  
  • It sounds like there are a few moving parts (an Android Java library that needs to be invoked from the Go codebase under certain circumstances? Would that change be in syncthing/syncthing-android or syncthing/syncthing?)
  
  
  • Are there any aspects of this that the team would want help with? (I'm not offering, not qualified, just frustrated with the stalled communication here)
  
  

[AudriusButkevicius]

There were no productive outcomes.

The Go team and the SyncThing team appear to have agreed on a way that this could/should be made to work without language-level support, instead having a small SyncThing-defined Java library being invoked from the Go code for the necessary write authorization intent interactions.

This is not true, as I figured out, syncthing has to become an android java application with a go library, not the other way around. JVM needs to invoke us, as we can't invoke the jvm ourselves.

So I don't think this will ever happen, as even with cgo, android hides access to jvm directly and can only be used by the operating system.

[Nutomic]

There was also this idea on the forum. But I don't think anyone currently has plans to implement this.

As another option, it would be possible to implement 2-way-sync functionality into syncthing-java, but that would probably be even more complicated to do correctly.

[TaoK]

This is not true, as I figured out, syncthing has to become an android java application with a go library, not the other way around. JVM needs to invoke us, as we can't invoke the jvm ourselves.

OK, sorry for the confusion. Can we update golang/go#10588 to reflect this outcome? I may be missing something, but it looks to me like the go team was left with the impression that there is no need for them to implement explicit support, given the workaround possibility.

So I don't think this will ever happen

Again, I think it would be kind to reflect that understanding in the FAQ

There was also this idea on the forum. But I don't think anyone currently has plans to implement this

This is not linked to at all from the forum, so it might warrant a link there (or in #29), to help drive interested eyes to the right places

it would be possible to implement 2-way-sync functionality into syncthing-java, but that would probably be even more complicated to do correctly

Completely off-topic note on that: I can't get a-sync-browser to transfer files at all (see davide-imbriaco/a-sync-browser#23) - as someone who's worked with the codebase, do you know anything about why that might be? (for context, my syncthing connectivity between devices appears to be going through relays and be super-shaky...)
UPDATE: Never mind, I didn't realize you also had a fork of the front-end piece at https://github.com/Nutomic/syncthing-lite/, I'll check that out, thx.

[Nutomic]

I've updated the wiki and added a comment in #29.

[madduck]

@Catfriend1 @TaoK The setting suggested in #1017 does not apply. Unfortunately, there seems to be no solution thus far, and not even running Syncthing as root works, because it then can't read its own config file anymore.

[Catfriend1]

@madduck which version of the app did you try? Please export your config and clear app data.

[madduck]

I'm using v0.14.45+1-gbf165d68 from F-Droid.

The setting suggested in #1017 is about installing apps to the external SD card, not about letting apps read data there.

I've cleared app data and reimported the configuration, but the behaviour hasn't changed.

[Catfriend1]

@madduck: Thanks for posting additional information. So I guess you're on app version 0.10.6 or 0.10.7. Can you please install the "Catlog" app and grab a Verbose-Log after a) force-closing syncthing app b) starting syncthing app and wait for it to initialize for 60 seconds using root rights. Which ROM/Android-Ver are you on?

[madduck]

Sorry, app version 0.10.7 is right. I'm running the so-called "Superman ROM" with Android 7.0.

I have "Catlog" installed, but I don't really know how to use it to grab logs. Can I just use adb logcat instead?

[Catfriend1]

@madduck: adb should be fine,too. Catlog has a three dot menu>log level>verbose. Menu>record. Do stuff. Menu> End record. Then, youll find catlogs log on sdcard.

[madduck]

My Catlog does not have a three-dot menu…

Anyway, here's the output of logcat, and the problem is exemplified by the fact that I created Camera/bla in the dcim-ext repository on another host:

% adb logcat -s SyncthingRunnable:v SyncthingNativeCode:v ConfigXml:v
03-30 16:24:14.999  3044  3076 I SyncthingNativeCode: [EOTEM] INFO: Puller (folder "DCIM ext" (m103s-n0bt6), file "Camera/bla"): finisher: dst create: open /storage/ADC0-14CB/DCIM/Camera/.syncthing.bla.tmp: permission denied
03-30 16:24:15.012  3044  3076 I SyncthingNativeCode: [EOTEM] INFO: Folder "DCIM ext" (m103s-n0bt6) isn't making progress. Pausing puller for 4m0s.
03-30 16:24:25.347  7427  7443 D ConfigXml: Trying to read '/data/user/0/com.nutomic.syncthingandroid/files/config.xml'
03-30 16:24:25.354  7427  7443 I ConfigXml: Loaded Syncthing config file
03-30 16:24:25.354  7427  7443 I ConfigXml: Checking for needed config updates
03-30 16:24:25.646  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: syncthing v0.14.45+1-gbf165d68 "Dysprosium Dragonfly" (go1.9.4 android-arm64) vagrant@jessie64 2018-03-13 18:08:09 UTC [noupgrade]
03-30 16:24:25.648  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: My ID: EOTEMOR-FEIPQ2E-NZ47LFU-GM7D56J-O7QDAMX-D72LAP2-JIWQITG-A2UDXQW
03-30 16:24:25.654  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Disabling weak hash
03-30 16:24:25.692  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Starting deadlock detector with 20m0s timeout
03-30 16:24:25.694  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Ready to synchronize "DCIM int" (o3y5s-oaoye) (readwrite)
03-30 16:24:25.698  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Ready to synchronize "DCIM ext" (m103s-n0bt6) (readwrite)
03-30 16:24:25.700  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Ready to synchronize "Downloads" (2nqco-mckz6) (readwrite)
03-30 16:24:25.701  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Send rate is unlimited, receive rate is unlimited
03-30 16:24:25.702  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Rate limits do not apply to LAN connections
03-30 16:24:25.706  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: TCP listener ([::]:22000) starting
03-30 16:24:25.762  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Completed initial scan of readwrite folder "DCIM ext" (m103s-n0bt6)
03-30 16:24:25.793  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Completed initial scan of readwrite folder "Downloads" (2nqco-mckz6)
03-30 16:24:26.029  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Device EOTEMOR-FEIPQ2E-NZ47LFU-GM7D56J-O7QDAMX-D72LAP2-JIWQITG-A2UDXQW is "madduck's sgs7e" at [dynamic]
03-30 16:24:26.031  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Device EYQXPWN-5ZQD3MI-LU4QTRL-WTTD54N-2JRPHIU-MR2PD7I-ZI4U3DI-KOGEVQ4 is "fishbowl" at [tcp://fishbowl]
03-30 16:24:26.033  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Device E2S6AJG-T3RWPWR-WB4YWOG-V53LIUV-7RRZHQ5-YYMPMO7-5OUKGET-EQ33LQO is "albatross" at [dynamic tcp://albatross.lehel.madduck.net]
03-30 16:24:26.034  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: No automatic upgrades; STNOUPGRADE environment variable defined.
03-30 16:24:26.042  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: GUI and API listening on 127.0.0.1:8384
03-30 16:24:26.044  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
03-30 16:24:26.306  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Completed initial scan of readwrite folder "DCIM int" (o3y5s-oaoye)
03-30 16:24:26.623  7427  7480 W SyncthingNativeCode: ionice: exec 7456: No such file or directory
03-30 16:24:35.799  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: New NAT port mapping: external TCP address 122.61.218.151:44814 to local address 0.0.0.0:22000.
03-30 16:24:35.803  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Detected 1 NAT service
03-30 16:24:46.627  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Established secure connection to EYQXPWN-5ZQD3MI-LU4QTRL-WTTD54N-2JRPHIU-MR2PD7I-ZI4U3DI-KOGEVQ4 at [fe80::8ef5:a3ff:fe5c:f9a9%wlan0]:22000-[fe80::b8a0:61ff:fe4a:5e02%wlan0]:41550/tcp-server (TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305)
03-30 16:24:46.629  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Device EYQXPWN-5ZQD3MI-LU4QTRL-WTTD54N-2JRPHIU-MR2PD7I-ZI4U3DI-KOGEVQ4 client is "syncthing v0.14.44-ds1" named "fishbowl" at [fe80::8ef5:a3ff:fe5c:f9a9%wlan0]:22000-[fe80::b8a0:61ff:fe4a:5e02%wlan0]:41550/tcp-server
03-30 16:24:46.667  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Puller (folder "DCIM ext" (m103s-n0bt6), file "Camera/bla"): finisher: dst create: open /storage/ADC0-14CB/DCIM/Camera/.syncthing.bla.tmp: permission denied
03-30 16:24:46.705  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Folder "DCIM ext" (m103s-n0bt6) isn't making progress. Pausing puller for 1m0s.
03-30 16:24:46.849  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Failed to exchange Hello messages with EYQXPWN-5ZQD3MI-LU4QTRL-WTTD54N-2JRPHIU-MR2PD7I-ZI4U3DI-KOGEVQ4 at 192.168.1.6:22000-192.168.1.8:59796/tcp-server: EOF
03-30 16:25:46.717  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Puller (folder "DCIM ext" (m103s-n0bt6), file "Camera/bla"): finisher: dst create: open /storage/ADC0-14CB/DCIM/Camera/.syncthing.bla.tmp: permission denied
03-30 16:25:46.738  7427  7457 I SyncthingNativeCode: [EOTEM] INFO: Folder "DCIM ext" (m103s-n0bt6) isn't making progress. Pausing puller for 1m0s.

[Catfriend1]

@madduck: try setting the experimental option use root privileges in the app. Version 0.10.8 is recommended for that test. In Catlog, you have the possibilty to filter for Syncthing and su as a keyword.

[bjaglin]

Commenting here as https://github.com/syncthing/syncthing-android/issues/29 was closed as "fixed". Even though it's nice to have 2 workarounds, I believe there is still a strong use case for normal people (not rooting their phone nor patching the system) to be able to use sd card as storage in SyncThing. This remains what personally prevents me from moving away from Resilio Sync for sharing pictures with my family members.

[Catfriend1]

@bjaglin This is currently not possible as syncthing core is a go application and android will/cannot grant file system write rights to this at the current state. A downloader for syncthing folders having native android support is syncthing-lite @github.

[bjaglin]

@Catfriend1 thanks for the prompt and very useful answer - my bad for not seeing it in the comments above. For other readers skimming through this threads after a Google search: check out https://github.com/syncthing/syncthing-lite.

[Catfriend1]

Lets close this as answered instead of solved by implementation.

[jistr]

I found another interesting workaround-solution, wanted to mention it in #29 but it's locked.

Creating a folder under Android/data/com.nutomic.syncthingandroid works but there's a caveat - if you want to sync media files (music), then it will not get scanned by Android's Media Storage service, because Android/data/.nomedia file is present (and removing it is not a good idea, considering many other apps could have their data under that directory).

However, it is possible to create Android/media/com.nutomic.syncthingandroid (note the media instead of data in the path) on the SD card, and Syncthing seems to be able to write into that directory too, and it does get scanned by Media Storage, so it's ideal for music. (I've only tried this on Android 9 Pie.)

It seems like a very underdocumented feature of Android, i hope this info solves a music syncing issue for someone else too :). I've added it into the wiki:

https://github.com/syncthing/syncthing-android/wiki/Frequently-Asked-Questions#what-about-sd-card-support

[Catfriend1]

Fyi: Next year, Google will enforce all apps to be compliant with the new Android Q scoped view permissions to read write data on internal/external storage. See https://developer.android.com/preview/privacy/scoped-storage#filtered-view
I could only wish for a localhost tcp Connection from the wrapper to SyncthingNative where the java code can provide file descriptors with rw permissions ... Just dreaming,not actually saying someone must do this.

[imsodin]

What's the difference to now - we already can't access the external SD card directly? And what's the relevance for the API connection?

[AudriusButkevicius]

It seems that we'll loose access to internal storage too.
You can't send file descriptors over a socket, as it's a piece of state associated with a process.

@imsodin did some work to turn syncthing into a library, so perhaps we can run native code inside the java process and implement a "filesystem" that has the SAF abstraction in it.

[Catfriend1]

From reading the part on Android Q scoped view it says :
"An app that has a filtered view always has read/write access to the files that it creates, both inside and outside its app-specific directory. Your app doesn't need to declare any storage permissions to access these files."
This seems to me like a new selinux policy where files are "owned by apps". I conclude that then other apps would have trouble to access files that were synced to the mobile depending on their permissions or need a fileprovider-like handover.
Files that other apps created (and we should pickup to sync them) could also be inaccessible for us.
Especially this part makes me fear as beforehand we had pretty good read access with the read storage permission but natives are restricted more:
"Note: Apps that have a filtered view into external storage don't have direct kernel access to paths like /sdcard/DCIM/IMG1024.JPG. To access such a file, apps must use the MediaStore, calling methods like openFile().0"
If it comes to that point I would know how to code that cumbersome mediastore stuff I think but couldn't pass anything on to the native.
Moreover, Nutomics old routine is still in Eventprocessor.java and takes care of adding files synced by Syncthing to the mobile into the MediaStore. I guess this would only work in the special paths if the native will prior get to write a file which I doubt from my reading. So two essential parts of ST Android may be affected: the native's read write access and the MediaStore handling of new files must be adjusted.

[Catfriend1]

Hey ho let's go, sign the petition?! https://www.change.org/p/google-inc-stop-google-from-implementing-scoped-storage-in-android

[imsodin]

It seems that we'll loose access to internal storage too.

I only find references to "external storage" in there, nothing about internal storage?
Edit: From what I read from non-google sources this does indeed apply to internal storage as well. Meaning when google says "external storage" they mean "any storage"? That's confusing as **.

As I understand it this would break read-only sd-card access only.

And this will only affect us if we target API level >28, so it seems it's only a problem in the long run.

@imsodin did some work to turn syncthing into a library, so perhaps we can run native code inside the java process and implement a "filesystem" that has the SAF abstraction in it.

Looks like I don't really have a clue about what I am doing there: With the library it might be possible to create java language bindings, to run it directly instead of spawning a process. However a filesystem still had to access androids SAF from within go, which I thought was determined as not possible? Or is the idea to somehow pass a filesystem implementation from java to go when invoking the library?

[Catfriend1]

Or is the idea to somehow pass a filesystem implementation from java to go when invoking the library?

Yes , gomobile supports Reverse Binding.

[Catfriend1]

Hi,

I've linked an interesting issue above about Syncthing on Android q beta 5 not working in most paths under scoped storage. Curious if that's user or OS fault?!

[codethief]

@imsodin

I only find references to "external storage" in there, nothing about internal storage?
Edit: From what I read from non-google sources this does indeed apply to internal storage as well. Meaning when google says "external storage" they mean "any storage"? That's confusing as **.

That's correct, "external storage" no longer means "actually external storage" as in "SD card storage". Rather, it just refers to /storage/emulated/0 (aka /sdcard) which is typically part of the built-in storage – as opposed to /data which is, by definition, the internal storage. If there is an actual external SD card present, there will be multiple "external" storage locations. See also https://developer.android.com/training/data-storage/files and https://developer.android.com/training/data-storage/files/external .

[Catfriend1]

The current discussion about Android 11 being expected to be the first version since 4.4 supporting writing to sdcard again is here: https://forum.syncthing.net/t/android-11-all-files-access-for-the-syncthing-app

[AudriusButkevicius]

I would refrain from mentioning it, to save people from being disappointed when it does not work outside the emulator.