Termux-packages: Can we add Unison and Duplicity?

[gerroon]

Hi

It would be great if we have Unison and Duplicity in the repo. This will allow good backup options on Android.

[justincase2006]

Yes

[fornwall]

The package suggestions have been added to https://termux.com/package-suggestions.html

[Neo-Oli]

Hi, I am reopening this issue. Package requests will now be labeled and tracked as an issue.

[ghost]

+1 for Unison.

[tristan-k]

+1 for Unison

[ghost]

+1 for unison

[ghost]

Just to help a bit

Jens Wagner has compiled a static ARM binary for unison 2.40.102. He is hosting it, as well as instruction for compilation, on his web site http://www.crutzi.info/unison/binary/armel.

[sunziping2016]

+1 for unison

[SapphicCode]

+1 for Unison

[vovcacik]

I spend whole day on getting unison into my android phone. I've got some notes more or less relevant to this issue.

Getting debugging toolchain for Android:

The SDK Manager and USB driver

• for the USB driver get SDK Manager
• run sdkmanager.bat extras;google;usb_driver

The adb.exe and fastboot.exe

• you can get just the debug tools from SDK Platform Tools, without the unnecessary stuff
• list devices with adb devices -l
• get shell with adb shell

Unison binaries

Linux repositories

• it might have been stupid idea, but don't bother with unison binaries from
  
  
  
  • Arch linux for ARM (I tried arm and arm7h platforms)
  
  
  • Unison for Android app (it features 9 versions of unison, including stable 2.48.4). Unpacking this app gave me non-executable binary files like unison-armeabi-2.48.4.enc. By the extension I think it is encrypted or UUencoded.
  
  
  • TODO it might be worth trying Debian ARM package repo
  
  

Crutzi build

• the only one that worked for me, unfortunately the version is old
• his build seems to be the easiest to reproduce
  
  
  
  • OCaml does support Linux on ARM, but not crosscompilation to it (there are patches that might make it possible), you might want to build natively instead as follows:
  
  
  • there are QEMU prebuild images for ARM arch here
  
  

Running Unison in Android

• for some reason my ROM requires the binary to be located in /system/bin; I believe it is permission problem - I've got the basic ACL right, but SELinux was probably the culprit
• you can easily alter your ROM with mount -o remount,rw /system. If the device is busy, go to recovery environment and do it from there - the actual command will look more like this: mount -t ext4 -o rw,noatime,data=ordered /dev/block/platform/msm_sdcc.1/by-name/system /rom (to get parameters relevant to you run mount | grep /system from OS). Make sure to read full howto.
• running unpriviledged unison in my ROM produced quite a few problems:
  
  
  
  • PATH env var: because /sdcard/ is mounted with noexec and because other locations like /data/local wont let me execute binary (SELinux...) I had to put it in /system/bin, see beginning of this section
  
  
  • broken su --preserve-environment -c ...: I tried four ssh servers, but only one worked (see section below)
  
  
  • overriding HOME env var for Unison: my ROM had HOME environmental variable set to HOME=/ which is readonly. I solved this with changing the ROM: a) /system/etc/mkshrc and b) /system/etc/profile. The latter may be unnecessary when you have properly working su --preserve-environment -c .... For details see this. Also this can be probably simplified with termux, because it provides OpenSSH (and not just Dropbear as other apps) using ~/.ssh/environment; not sure though. Other OpenSSH options include SendEnv/AcceptEnv.
  
  

broken su --preserve-environment -c ...

• this whole section is about simple problem:
  
  
  
  • SuperSu is not preserving environmental variables for direct commands over ssh, unless you specify --preserve-environment parameter. For direct commands versus interactive ssh session see section below. Unison is using direct commands when you are using ssh method (see unison docs for ssh method vs socket method). With this setup you need to run unison as su --preserve-environment -c unison which preserves proper $HOME env var. However some ssh servers will interpret this command as interactive session, and hence is equivalent to su --preserve-environment -l, which wont work at all.
  
  
• the combination of SuperSu app (it provides /su/bin/su) and 3 ssh servers: Servers Ultimate, SSHDroid and SimpleSSHD had this problem (I think all three are using Dropbear)
• only the combination of termux which provides both the $PREFIX/bin/su and $PREFIX/bin/sshd (OpenSSH) worked for me. SuperSu and termux sshd works fine together.

Direct ssh commands vs interactive ssh sessions

• suppose you have configured your environmental variables in ~/.profile and/or ~/.bashrc or ~/.mkshrc etc.
• when you are using interactive session (ssh admin@host or putty...) all is fine, your shell will find and execute the rc files
• when you are using non-interactive session commands are sent directly (plink admin@host "unison -server for example) and thats exactly what unison does - the environment may get resetted if you configured the ssh server to use su shell for clients or you sent command like plink admin@host "su --preserve-environment -c unison -server. This is bad when the unison on server depends on properly set $HOME variable

Unison caveats

• unison is sensitive about versions - you should run at least the same MAJOR.MINOR, even better MAJOR.MINOR.PATCH (definition).
• unison is sensitive about what version of OCaml you used during compilation even with same sources
  
  
  
  • due to some internal changes in OCaml, unison may throw Uncaught exception Failure("input_value: bad bigarray kind"). Google or here.
  
  
  • another example is easy to find in official windows builds
  
  

[vovcacik]

Android binaries in https://github.com/vovcacik/unison-build-scripts. Can anyone package it?

[vovcacik]

Unison usage notes for Android here https://github.com/vovcacik/unison-build-scripts/blob/master/ANDROID.md

[YtvwlD]

duplicity is already installable:

• Install gnupg, librsync-dev, clang, python2 and python2-dev via apt.
• Install duplicity via pip2.
• Edit ~/usr/bin/duplicity with your favourite text editor and set the shebang to the output of which python2.

(This is just a fancy "me too".)

[niklasl]

Until Unison becomes available, I use a crude fallback method:

• rsync local files to a "landing directory" on the server;
• use unison to consolidate the landing dir with the real sync dir locally on the server;
• rsync back the results.

Not sure about how clever that is, but perhaps it'll help someone.

[gerroon]

@niklasl you can also install syncthing in termux. It was added recently. It is a better syncing solution then bi directional rysnc.

I use rsync to back my whole sd card btw.

[YtvwlD]

You can also install Syncthing as a normal Android app.

[gerroon]

@YtvwlD

I do not recommend the regular syncthing android app, it is super buggy and crashy especially with large shares. The app regularly crashes the syncthing binary and you will never know about it.

Running it under termux much more flexible.

[niklasl]

@gerroon Thanks for the tip! It seems syncthing is configurable in a different manner than Unison, but it seems like an interesting alternative.

[SDRausty]

It appears that both Duplicity and Unison are currently available in Termux through PRoot Arch Linux. This script is designed to install Arch Linux in just a few keystrokes. Tapping this link https://sdrausty.github.io/TermuxArch/setupTermuxArch.sh will transfer setupTermuxArch.sh to your device. Simply run bash ~/storage/downloads/setupTermuxArch.sh to install Arch Linux in Termux. To install Duplicity use pacman -S duplicity. Running pacman -S unison will install Unison on device.

[markbil]

@sdrausty i like your idea. I installed the unison package in termuxarch via "pacman -S unison". now i would like to ssh from my pc (in the same local network) directly into termuxarch to sync files via unison. however "ssh -p 8022 [email protected]>" connects me to the termux environment not termuxarch, so unison commands in termuxarch are not available directly via ssh (e.g. "ssh -p 8022 [email protected] unison -version" results in "command not found"). "ssh -p 22 [email protected]" results into "connection refused" error, even though I have set up an open ssh server in termuxarch. any ideas how I can ssh directly into termuxarch?

[SDRausty]

@markbil I am glad to hear you like the idea. Use a higher numbered port, something like 8222 in TermuxArch for instance.

[alexozer]

I've tried Unison in Arch, but when using -repeat watch to sync instantly on local file changes, after a while of running it seems to stop syncing changes on the phone eventually.

At some point I'll modify a file, and unison will print "Looking for changes" and then just freeze until restarted, when it will then sync the change. No idea how to proceed.

[vovcacik]

@alexozer that's off topic for this thread. I've got some ideas, start new issue so we can discuss it there.

EDIT:
see #2412

[alexozer]

@vovcacik Sorry about that, don't know why I thought it was relevant to this thread at the time.

[QGB]

"termux"+ setfacl getfacl

[natethern]

@niklasl I was doing the same thing as you (rsync, then unison, then rsync back).
Then I switched to using sshfs to create a local mount of the directory (I don't know if that can be done on windoze - I use debian, among others). My process when I come home is (was)

1. on android: open termux and start sshd
2. on linux: run sshfs ... to mount the directory on my android device
3. on linux: run unison ... to sync
4. on linux: run fusermount -u ... to unmount

I also wrote it up as an bash script. As an example, for my one 3-way sync the script does this:

1. Initiate the sshfs mount on device A
2. Initiate the sshfs mount on device B
3. Sync device A with the master using unison
4. Sync device B with the master using unison (propagates any updates from A into B)
5. Sync device A with the master using unison (propagates any updates from B into A)
6. unmount A & B

The script checks for empty directories at the mount points (meaning the mount failed) and skips unison if appropriate.

Recently I switched to SimpleSSHD which automatically keeps an sshd server alive on each of my android devices. This, of course, is a non-termux solution.

[bitcoinmeetups]

+1 for Duplicity.

Please just add support for Duplicity, it would be so cool.

[hellas2000]

I'm going to buy an android phone, but I'm waiting for unison to work in termux, it's very important to me. Please let an expert to help us :)

[tulakalbeyo]

+1 for unison

[SDRausty]

@tulakalbeyo unison is available in Termux through TermuxArch:

$ startarch s user yay unison
Warning: your password will expire in 0 days.
8 aur/unison-232-compat 2.32.52-2 (+0 0.00) (Orphaned)
    File-synchronization tool (legacy 2.32 version). Compatible with parallel installation to other versions of unison binaries (but pay attention to the local user configurations!)
7 aur/unison-git 2.51.3.r257.g22a05df-1 (+1 0.00)
    File-synchronization tool
6 aur/csync2-git 2.0.r22.gce67c55-1 (+1 0.00)
    Asynchronous cluster syncronisation tool based on librsync and inspired by Unison
5 aur/ucm-bin M1m-1 (+2 0.39)
    Unison language code manager
4 aur/unison-240-compat 2.40.128-1 (+3 0.00) (Orphaned)
    File-synchronization tool
3 aur/csync2 2.0-4 (+10 0.00)
    Asynchronous cluster syncronisation tool based on librsync and inspired by Unison
2 aur/soundfont-unison 1.00-3 (+32 0.00)
    A lean and clean GM/GS soundbank
1 extra/unison 2.51.2-9 (1.9 MiB 8.8 MiB)
    File-synchronization tool
==> Packages to install (eg: 1 2 3, 1-3 or ^4)

going to buy an android phone, but I'm waiting for unison to work in termux

[user15:30~]$ unison
Usage: unison [options]
    or unison root1 root2 [options]
    or unison profilename [options]

For a list of options, type "unison -help".
For a tutorial on basic usage, type "unison -doc tutorial".
For other documentation, type "unison -doc topics".

just add support for duplicity, it would be so cool

[user15:34~]$ duplicity -V
duplicity 0.8.15

@bitcoinmeetups both give the impression of working in Termux, and they do need proot.

[tulakalbeyo]

@SDRausty

@tulakalbeyo unison is available in Termux through TermuxArch:

i installed unison in a ubuntu proot-distro. but how do you run only unison with proot?

i tried unset LD_PRELOAD && proot -r /data/data/com.termux/files/usr/var/lib/proot-distro/installed-rootfs/ubuntu/ unison, similar to how it's explained in the official proot documentation here and adding some of what it says on the termux proot wiki page, but no dice:

$
$ unset LD_PRELOAD && proot -r /data/data/com.termux/files/usr/var/lib/proot-distro/installed-rootfs/ubuntu/ unison
proot warning: can't chdir("/data/data/com.termux/files/home/./.") in the guest rootfs: No such file or directory
proot info: default working directory is now "/"
proot error: 'unison' not found (root = /data/data/com.termux/files/usr/var/lib/proot-distro/installed-rootfs/ubuntu, cwd = /, $PATH=/data/data/com.termux/files/usr/bin:/data/data/com.termux/files/usr/bin/applets)
fatal error: see `proot --help`.
$

which of all of these parameters do you need to run it?

[xeffyr]

which of all of these parameters do you need to run it?

All of them. Termux is not GNU/Linux system so you need to pass some arguments to proot to get mounts & environment be configured correctly.

[xeffyr]

Well, you can omit some of them, but anyway proot command line should look like

proot \
    --link2symlink \
    --rootfs="/path/to/rootfs" \
    --root-id \
    --cwd=/root \
    --bind=$HOME:/root \
    --bind=/dev \
    --bind=/proc \
    --bind=/sys \
    /usr/bin/env -i \
        "HOME=/root" \
        "LANG=C.UTF-8" \
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \
        "TERM=$TERM" \
        "TMPDIR=/tmp" \
        "unison"

[tulakalbeyo]

IT WORKED!!! :laughing::grin::+1::+1::sparkles:

[SDRausty]

IT WORKED!!! 😆😁👍👍✨

@tulakalbeyo I am very glad to hear that it is working for you!

how do you run only unison with proot

You might be surprised to hear program unison, along with the slew of other programs are new to me, and I have not really used many of them. When I announced myself to This Crew a couple of years ago I kept hearing people crying, "Help me, help me please!" for programs that people wanted to use, but could not! These crying people included me. That is why I announced myself.

This doesn't work! That doesn't work! Oh no!

I'm very glad to hear that we all have made progress with unison and many other programs that did not used to work in our Android smartphones, tablets and smart TVs, and now do work as expected. Can you please share some good noob primers about unison with us here please?

EDIT

good noob primers

What are the next couple of steps with unison? I know that checking for version and checking help does test if the program really works. What are the next couple of steps with unison to see if it really works; whether I can get some basic functionality out of this program which is at present new to me; Can you share this in just a couple of lines of code please?

[tulakalbeyo]

hey, @SDRausty !

tests

i am just waking up here, and will for the next few hours run all the pertinent unison tests for running it this way.

primer

i will be more than happy to write a primer / howto on unison, at least for my specific usecase. perhaps i will publish it on the termux subreddit. and link it from here.

description

to put it in ordinary language, unison is basically just a dropbox.com for your local directories, in the way it can do bidirectional sync, instead of just one-way sync like rsync does. since it can work through openssh, it will also do bidirectional sync between remote machines.

try it

unison is pretty powerful, and has a wide range of options for all kinds of usecases, but you can start with a simple case of:

unison /testdir1 /testdir2

just plop different files, and/or different versions of the same files into each directory and see how it behaves.

documentation

i think it's important to update the proot wiki article to exhibit and clarify how to run single commands / applications with proot, the way @xeffyr just demonstrated.

[tulakalbeyo]

well. i managed to progress considerably, but i find it difficult to continue, as i am at a loss as to how to troubleshoot.

tldr

1. a line of commands, containing unison, with all its necessary proot parameters, gets cat'ed into ~/bin/unison and given _execute_ permissions.
2. a symlink is created in /data/data/com.termux/files/usr/bin/unison that points to this executable ~/bin/unison file.
3. ~/bin is added to PATH in both .bashrc and .bash_profile.
4. a symlink is created in ~/ that points to the actual directory to be synchronized, since it is normally not-accessible to unison.
5. some tests are run.
   
   
   
   1. the local tests are successful ✅.
   
   
   2. the remote test is unsuccessful 🚫.
   
   

how it went down

#==================================================
# proot unison
#==================================================
#   - - - - - - - - - - - - - - - - - - - - - -
#   NOTE
#   - - - - - - - - - - - - - - - - - - - - - -
<<.
#   .bashrc     = for local non-login shells
#   .bash_profile   = for local or remote login shells
#       ref: https://unix.stackexchange.com/questions/332531/why-does-remote-bash-source-bash-profile-instead-of-bashrc
.

#   - - - - - - - - - - - - - - - - - - - - - -
#   add home/bin to path
#   - - - - - - - - - - - - - - - - - - - - - -
cat >> temp <<'HEREDOC'
    # Allows Executable Files in "~/bin" to be Run Directly
    export PATH="$PATH:$HOME/bin"
    #
HEREDOC
#

#           : : : : : : : : : : : : : :
#           add "temp" to
#           ".bashrc" & ".bash_profile"
#           : : : : : : : : : : : : : :
cat temp >> .bashrc &&
cat temp >> .bash_profile &&
rm temp &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   create home/bin
#   - - - - - - - - - - - - - - - - - - - - - -
mkdir bin &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   proot unison with all required arguments
#   -- cat'ed into home/bin
#   - - - - - - - - - - - - - - - - - - - - - -
cat > bin/unison <<'HEREDOC'
    #
    unset LD_PRELOAD &&
    proot \
        --link2symlink \
        --rootfs="/data/data/com.termux/files/usr/var/lib/proot-distro/installed-rootfs/ubuntu" \
        --root-id \
        --cwd=/root \
        --bind=$HOME:/root \
        --bind=/dev \
        --bind=/proc \
        --bind=/sys \
        /usr/bin/env -i \
            "HOME=/root" \
            "LANG=C.UTF-8" \
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:" \
            "TERM=$TERM" \
            "TMPDIR=/tmp" \
            \
            \
            "unison" \
            "$@" \
    #
HEREDOC
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   make bin/unison executable
#   - - - - - - - - - - - - - - - - - - - - - -
sudo chmod +x bin/unison &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   link home/bin/unison to /usr/bin/unison
#   (where remote clients expects it)
#   - - - - - - - - - - - - - - - - - - - - - -
#sudo \
ln -fs "/data/data/com.termux/files/home/bin/unison" "/data/data/com.termux/files/usr/bin/unison"
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   link /storage/emulated/0/Desktop/ to ~/
#   (since unison can't write to it)
#   - - - - - - - - - - - - - - - - - - - - - -
ln -fs /storage/emulated/0/Desktop/ ~/Desktop &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   local tests
#   - - - - - - - - - - - - - - - - - - - - - -
mkdir foo bar && touch foo/foo bar/bar &&
tree foo bar &&
unison -batch foo bar &&
tree foo bar;
rm -R .unison &&
rm -R foo bar &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   test from client
#   - - - - - - - - - - - - - - - - - - - - - -
unison -testServer ~/Desktop ssh://river//storage/emulated/0/Desktop &&
:
#

test results

the local tests run excellently!! 🙂👍✨
but the test from client shows this 🙄😤

tulakal@linux:~$
tulakal@linux:~$ unison -batch ~/Desktop/ ssh://river//Desktop                                                             
Contacting server...
X11 forwarding request failed on channel 0
Connected [//linux//home/tulakal/Desktop -> //localhost//Desktop]
Looking for changes
  Waiting for changes from server
Reconciling changes
new dir  ---->            /
local        : new dir            modified on 2020-08-24 at 11:25:53  size 191026917 rwxr-xr-x
localhost    : absent
Propagating updates
UNISON 2.48.4 started propagating changes at 13:04:15.92 on 27 Aug 2020
[BGN] Copying  from /home/tulakal/Desktop to //localhost//Desktop
Failed: Fatal error during unmarshaling (input_value: ill-formed message),
possibly because client and server have been compiled with differentversions of the OCaml compiler.
Fatal error: Fatal error during unmarshaling (input_value: ill-formed message),
possibly because client and server have been compiled with differentversions of the OCaml compiler.
tulakal@linux:~$

to eliminate the possibility that the symlink to /storage/emulated/0/Desktop/ could be the problem, i also created a test directory on termux's home (mkdir test) and ran unison -batch ~/Desktop/ ssh://river//test. it still resulted in the same Failed: Fatal error during unmarshaling (input_value: ill-formed message) error.

possible solution?

recompile unison with the same OCaml compiler? how to even tell which OCaml compiler was used?

how everything is setup: termux after-installation runbook:

#==================================================
# remote administration
#==================================================
#   - - - - - - - - - - - - - - - - - - - - - -
#   preliminary packages
#   - - - - - - - - - - - - - - - - - - - - - -
pkg upgrade &&
pkg install termux-auth openssh &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   set password
#   - - - - - - - - - - - - - - - - - - - - - -
passwd &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   openssh
#   - - - - - - - - - - - - - - - - - - - - - -
#           : : : : : : : : : : : : : :
#           termux: run ssh daemon now
#           : : : : : : : : : : : : : :
sshd &&
:
#

#           : : : : : : : : : : : : : :
#           client: alias
#           : : : : : : : : : : : : : :
cat >> ~/.ssh/config <<'HEREDOC'
#
#
        host termux
        port 8022
        user root
        hostname 192.168.X.X
        protocol 2
        #ciphers blowfish
HEREDOC
#

#           : : : : : : : : : : : : : :
#           termux: run ssh daemon on termux startup
#           : : : : : : : : : : : : : :
cat >> .bashrc <<'HEREDOC'
    #
    # Run OpenSSH daemon on Termux launch
    sshd
    #
HEREDOC
#

#==================================================
# gain acess to external storage
#==================================================
# your milage may vary, but this worked like a charm the last time
termux-setup-storage &&
:
#

#==================================================
# additional packages
#==================================================
pkg upgrade &&
pkg install tsu htop tree micro rsync proot-distro &&
:
#

#==================================================
# grant root privileges
#==================================================
sudo echo &&
:
#

#==================================================
# proot ubuntu & packages
#==================================================
#proot-distro list &&

#   - - - - - - - - - - - - - - - - - - - - - -
#   install ubuntu
#   - - - - - - - - - - - - - - - - - - - - - -
#proot-distro reset ubuntu && # clean re-install
proot-distro install ubuntu &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   login to ubuntu
#   - - - - - - - - - - - - - - - - - - - - - -
proot-distro login ubuntu &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   packages
#   - - - - - - - - - - - - - - - - - - - - - -
#           : : : : : : : : : : : : : :
#           update
#           : : : : : : : : : : : : : :
apt update &&
:
#

#           : : : : : : : : : : : : : :
#           install
#           : : : : : : : : : : : : : :
apt-get install -y unison &&
:
#

#   - - - - - - - - - - - - - - - - - - - - - -
#   back to termux
#   - - - - - - - - - - - - - - - - - - - - - -
exit &&
:
#

#==================================================
# ascertain storage layout
#==================================================
ls storage &&
:
#

[sergey1369]

Hi, guys.

May be I do something wrong, but proot --link2symlink doesn't play well with unison.

Here is unison-static:
http://gael-varoquaux.info/misc/unison-248-binaries-for-arm.html

It works great under proot (synced d1/f1), but not proot --link2symlink (failed d1/f2).

UPDATE:
Unison works on K10 with manually compiled proot,
But this proot fails on newer droids ("bad syscall" errors).
No reason to play with static version.

# Walkaround
proot-distro install unison-18.04
proot-distro login ubuntu-18.04
apt update
apt install unison
# works fine for me