Termux-packages: Discussion: dropping android 5&6 support

[Grimler91]

So, lets officially start the discussion about dropping android 5 support.
This concerns both the apps and termux-packages but lets gather the discussion here.

What about all the packages?

I suggest that we create an "android 5" branch which won't receive updates but might be useful for people stuck on android 5. We could then also set up a separate android 5 repo or (preferably I guess) let someone who's actually using android 5 host and maintain it.

If we create a separate android 5 repo, how do we inform all android 5 users and simplify the transition between? Posts in all community groups? A temporary new motd message?

What about the apps?

Should we create android 5 branches for the apps as well? Would it be possible to have termux-app as well as termux-app5 on playstore (or would it somehow conflict since both apps are installed into com.termux)?

Do we even want to bump the minimum android version for the apps as well or is it enough to bump the API_LEVEL in build-package.sh?

What should we so when we've bumped the required android version?

We have at least one if version == lollipop that can be removed.
There's probably some patches for packages that can be dropped (identifying which ones might be tedious though).

Most of the cool changes, like the possibility to stop using LD_LIBRARY_PATH will have to wait until we drop android 6 support.
@monoidic did some experiments with this in https://github.com/termux/termux-packages/issues/2429.

What other changes/benefits have I not thought about?

[hyperpallium]

First, what benefit are there to dropping android 5? It seems most of the benefits would come from dropping 5, 5.1 and 6? Why not defer all the trauma until then, when it's more worth it?

BTW You can have distinct names on playstore (e.g. termux-app and termux-app5). That's how people do "lite" versions.
I have com.termuz on the same device (using aapt .... --rename-manifest-package com.termuz ....) - though there's a bunch of other stuff you need to change too, if you want to install both on the same device.

[fornwall]

I think we should target Android 7 directly.

We could create some kind of old-android branch of termux-packages (for supporting android 6&7) which is only maintained for critical bug fixes and as best effort.

For the package repo at termux.net perhaps the best thing is to leave the current one as is, to avoid people from updating and breaking their environment, and add a new repo alongside it. This would be in combination with updating the Termux app so that new installations get that repo automatically, and it could automatically replace the old repo with the new in etc/apt/sources.list if the device is running Android 7 or later.

For the apps, I think we should bump the required api version to avoid having Android 5 and 6 users installing the app and get a subpar experience with non-supported packages. Those knowledgeable enough can always work around it by finding an earlier app version on e.g. F-Droid if they really need it.

Thoughts?

What other changes/benefits have I not thought about?

One small thing that can be dropped is the libutil package, since the functions implemented there are built into bionic as of Android 6.0.

[tomty89]

I think we should target Android 7 directly.

Couldn't agree more. Android 6 seems to be on half way of many stuff. Doesn't seem worth bumping if that's our target.

[vishalbiswas]

I'm curious about third party repos.
@its-pointless @xeffyr How will you guys handle the API Level change?

[ghost]

Perhaps y'all could handle them apt-level, like:

deb https://termux.net/lollipop stable main
deb https://termux.net/marshmellow stable main
deb https://termux.net/nougat stable main
deb https://termux.net/oreo stable main
deb https://termux.net/pie stable main

or

deb https://termux.net/l stable main
deb https://termux.net/m stable main
deb https://termux.net/n stable main
deb https://termux.net/o stable main
deb https://termux.net/p stable main

[hyperpallium]

leave the current one as is, to avoid people from updating and breaking their environment, and add a new repo alongside it.

Seems the right approach.

earlier app version on e.g. F-Droid

Seems risky... I'm not sure people on older android versions are more likely to be across the android ecosystem...

How to check that https://f-droid.org/en/packages/com.termux/ has not been altered?

Is there some other way to keep the present app version available?

One solution is to leave the app as it is, and create a new version with a new name (com.termux2). Of course, this loses the ratings and download stats - and maybe people on 7, 8 etc wouldn't know to download the new one. But won't the play store only allow compatible downloads, according to the manifest? The description can note the other version.

Or, download the ols version directly from termux.net? (since it's already trusted for packages).

[Grimler91]

Targeting android 7 straight away sounds good to me!

@fornwall how is the current user distribution? How many percent are using android 5 or 6?

I have com.termuz on the same device (using aapt .... --rename-manifest-package com.termuz ....)

@hyperpallium Cool, thanks for that information. I guess all packages need to be recompiled with this new PREFIX if it is to be usuable.

For the package repo at termux.net perhaps the best thing is to leave the current one as is, to avoid people from updating and breaking their environment, and add a new repo alongside it. This would be in combination with updating the Termux app so that new installations get that repo automatically, and it could automatically replace the old repo with the new in etc/apt/sources.list if the device is running Android 7 or later.

I think the other way around sounds more reasonable. We can update apt and have a script replace "termux.net stable main" with "termux.net android-old main" if the android version is 5 or 6. We then wait a few months and hope that the wast majority of old android users have had their repo changed.
Feels more reasonable to have the main termux.net repo up to date, targeting newer android versions.

One solution is to leave the app as it is, and create a new version with a new name (com.termux2). Of course, this loses the ratings and download stats - and maybe people on 7, 8 etc wouldn't know to download the new one. But won't the play store only allow compatible downloads, according to the manifest? The description can note the other version.

Same here then really, feels like the main com.termux app should target up-to-date devices, but we could perhaps have another app targeting android 5 & 6.

Perhaps y'all could handle them apt-level, like:

@aiisuika The differences between the android versions aren't that big, so this might be slightly overkill, taking up 5x as much diskspace on the server. Also, people might not bother to build/test for different android versions, it would require quite a lot of extra time spent in emulators, it's a neat idea though!

[xeffyr]

I'm curious about third party repos.
@its-pointless @xeffyr How will you guys handle the API Level change?

@vishalbiswas I can handle it in this way:

1. Android 5.0-compatible repository will be accessible from default URL, so sources list will not be changed:

deb https://termux-x11.ml x11 main

1. Android 7.0 will require a different sources list line:

deb https://termux-x11.ml x11-nougat main

[hyperpallium]

@Grimler91 I guess all packages need to be recompiled with this new PREFIX if it is to be usuable.

Yes (it's also hardcoded in the app).

A key advance of termux over other terminals is packages modified for android's directory structure. But they are now compiled to termux's directory structure (literally, /data/data/com.termux/ etc). This could be an opportunity for all packages to dynamically looku the currrent $PREFIX instead of hard-coding it.

This enables different termux's to use the same repository without recompilation. It also enables "self-hosting", in the sense of developing termux within termux itself. The version being developed having a different name (and $PREFIX) from the termux doing the developing... so it can be installed without replacing the first termux, and all packages can be installed too!

[xeffyr]

This could be an opportunity for all packages to dynamically looku the currrent $PREFIX instead of hard-coding it.

@hyperpallium This is task is not easy and may have some bad effects (at least if PREFIX is long). See sample for bash how this is done: https://github.com/xeffyr/Alpine-Term/blob/master/env-packages/packages/bash/bash-4.4_config-top.h.patch.

It also enables "self-hosting", in the sense of developing termux within termux itself.

Unfortunately, it does not enable "self-hosting". At least because of difference between NDK's clang and one provided in Termux. It also not possible to compile big packages on device (out of ram at least on LLVM).
However if you want to just compile packages on device, take a look on https://github.com/xeffyr/termux-makepkg.

[hyperpallium]

dynamically looku the currrent $PREFIX instead of hard-coding it.

@hyperpallium This is task is not easy and may have some bad effects (at least if PREFIX is long). See sample for bash how this is done.

@xeffyr There's many packages to change, but... it seems to me the places needing the $PREFIX are already identified, and the additional snippet (to lookup and default if not found) would be the same for each. It seems feasible to even write a sed script to transform the present patch scripts, to automate this, based on the occurance of @TERMUX_PREFIX@. Some cases would be tricky. And such automatic transforms are problem-prone and thus more than a little scary!

There could be a test for the length of $PREFIX, in the termux-app, and an informative error message.

It also enables "self-hosting", in the sense of developing termux within termux itself.

Unfortunately, it does not enable "self-hosting".

I beg to differ, as I have compiled and installed the termux app, in termux (no PC). For the app, there's only a little C, and clang worked fine (invoked as gcc). I used this to experiment with horizontal scrolling. That's what I meant by in the sense of developing termux within termux itself.

The main work was a bash script to replace the gradle files. There's still some rough edges, but it basically works e.g. vim and curl are installed and work, after s///g the hardcoded $PREFIX. I agree you probably can't compile every package in the repository.

[fornwall]

@fornwall how is the current user distribution? How many percent are using android 5 or 6?

This is the user distribution of active installs of the Termux app from the Play store:

| Android version | Installations (%) |
| ------------- | ------------- |
| 8.0 | 20.0 |
| 7.0 | 18.4 |
| 6.0 | 18.1 |
| 7.1 | 14.3 |
| 8.1 | 12.0 |
| 5.1 | 9.8 |
| 5.0 | 3.8 |

Here active means: Number of Android devices that have been active in the previous 30 days on which your app is installed.

[fornwall]

@xeffyr There's many packages to change, but... it seems to me the places needing the $PREFIX are already identified, and the additional snippet (to lookup and default if not found) would be the same for each.

Actually, we only patch cases where the path is hard-coded into source files directly. Often paths end up in binaries after configuration from the build system (as in --prefix=$TERMUX_PREFIX for configure), so having dynamic prefix would probably require more patches.

That said, dynamic prefix would be interesting, I've created #2902 for that!

[hyperpallium]

@fornwall strikingly close to 1/3 each to 5/6, 7 and 8. A third is a lot...

@Grimler91 My first question would have been better as: What are the benefits to users?

[ Making development easier is important. But, like businesses that start placing operations ahead of customers, it can be the beginning of the end (unless it's a monopoly). Maybe there are great benefits to users, e.g. maybe fullscreen mode? ]

[ghost]

Inb4 libtermuxglibbionicc.so :^)

[Dmole]

...user distribution...

My (google, oneplus, LineageOS) devices are all on 9, is the % of Android 9 users unknown?
Those stats are only from people who opted in to reporting to Google, and did not install from f-droid.

I'd drop support for anything older than 6, because LineageOS is a free upgrade and supporting botnets on insecure devices is not cool;
Android 8 (from 2017-08-21) is still supported by Google.
Android 7,6 (from 2015-08-05) "security updates only" (no bug fixing)
Android 5 (from 2014-09-12) is EOL
And I'd drop anything older than 8 is some old Android bug consumed any development time.

[monoidic]

LineageOS is a free upgrade

One that is not an option on all devices and for some other reasons.

I'd drop anything older than 8

So only around ⅕ of the current users can keep using Termux...?

Don't get me wrong, I'm all for getting rid of the LD_{LIBRARY_PATH,PRELOAD} annoyances, being able to actually use patchelf and whatever else Android 7 brings, but "screw 'em" probably isn't the way to go about this.

[xeffyr]

I'd drop anything older than 8

Only Android 5/6 may be 'dropped'. There also plans for creating separate repositories for Android 5 and 7. Android 5 will receive only critical updates for packages, i.e. bug fixes but nothing new.

[ghost]

what bout an "emulated" libc.so that's actually proot with some changes

that'd also allow for users, groups, binfmt, ect...

[xeffyr]

what bout an "emulated" libc.so that's actually proot with some changes

@Bakaika

1. Proot is not "emulated" libc.
2. Usage of proot will at least degrade performance of the environment. Why not to use a QEMU then ?

[ghost]

i mean, libc.so where i/o operations are replaced with translated ones, faking of users, mounting, ect... like proot.
degration of performace is a big thing... perhaps a custom dpkg could modify packages as they're installed?
a qemu-proot deb compiler?

[xeffyr]

libc.so where i/o operations are replaced with translated ones,

@Bakaika No, this won't work here as it is impossible to intercept system calls with this way. You have to use ptrace() here so program like proot is needed.

mounting, ect... like proot.

proot does not do mounting and I'm not sure that there are people that can implement a userspace EXT4 driver usable with proot (even in read-only mode).

degration of performace is a big thing... perhaps a custom dpkg could modify packages as they're installed?
a qemu-proot deb compiler?

QEMU is not proot and I'm not talking about user-mode emulation. I'm about system emulation which actually gives much more than proot: full Linux distribution with real root permissions (on VM level) will provide multiuser environment, mounting and everything else but only on VM level.

The main point of Termux is to provide an environment similar to Linux distributions. It does not provide root, faking of users, mounting, but command line tools. Everything else is secondary: want root - then root your device, need classical unix fs layout - use termux-chroot (from package 'proot'), need something else - use linux chroot (proot) or QEMU (from my x11-packages repo).

[hyperpallium]

Turning it around, there are benefits for 5/6, for having a separate app, as changes have already been made for 7 and 8 that adversely affect 5/6. Examples I know of (I expect there are more):

1. fullscreen mode removed
2. termux-elf-cleaner leaving things that cause warnings
3. ecj needing java 8 features (ecj4.6 workaround)

[ghost]

Fine suggestion because support for things like DT_RPATH can be added. Thus I think stuff like LinuxBrew and third party relocatable binaries will work.

Whatever, keep a termux-old on fdroid.

[joakim-noah]

I don't think this would be a good idea at all: plenty of people still use old devices that stopped getting Android updates years ago, as @fornwall's stats show. Those devices are plenty fast enough to use Termux: I actually build LLVM _on_ a 2014-vintage 32-bit Android/ARM device with 3 GB of RAM and that's stuck on Android 6, without a problem.

The benefits discussed here are fairly small and could probably be done now by changing the Termux build and scripts to be more customized for the different Android versions. Of course, I don't support these old Android versions for Termux, and you may be sick of doing it: I don't know how much work it is. But I think it'd be a mistake to cut the userbase like this anytime soon.

[easyaspi314]

@joakim-noah I'm gonna guess LG G3? If so, I have it too. But check out LineageOS, you can get up to 8.1. The only real bug is that you need to move /system/bin/crash_dump32 to something else and make a shell script wrapper in its place that resets LD_LIBRARY_PATH and unsets LD_PRELOAD, or if anything segfaults, the crash_dump32 will crash, calling itself until it runs out of RAM.

Anyways, what about if we provide libc itself? Or make a polyfill for the different Android versions, kind of like Gnulib? That way, we can have a common ground on APIs.

I mean, at least on my G3, libc is only a few kB, and the combined size of libc, libc++, libm, ld-android, and libdl is 1.5 MB, which is a very reasonable size.

[xeffyr]

I mean, at least on my G3, libc is only a few kB, and the combined size of libc, libc++, libm, ld-android, and libdl is 1.5 MB, which is a very reasonable size.

It's bionic, libc for embedded systems. That's why it is small. GNU libc much bigger.
To use a custom libc we need to build own cross compiler for that - while it is fairly easy to compile, the good question will it work correctly. Many distributions use a patched GCC.

---

I have tried libc from different Android version - binaries just hang without any output. So using a custom bionic libc may be not a good idea.

[joakim-noah]

No, Samsung Tab S 8.4, the first one. When my Windows 7 Asus Zenbook died in late 2015, I bought a cheap $4 tablet stand and a $20 bluetooth keyboard and used this 8.4" tablet as my "desktop" for the last couple years, similar to how this guy used his Nexus 10. I picked up a 5.5" Android/AArch64 8.0 smartphone in February and have been using it with the same stand/keyboard instead: I'm typing this comment up on it now.

I'm not going to bother upgrading that old tablet to a newer Android, as it's four years old and I've never installed a custom ROM. I mostly use it for web reading and youtube these days, most of my software development has moved to the newer and much faster smartphone. I only build stuff on or for the tablet to make sure code's still working on 32-bit or older Android.

I have no problem with trying a newer libc or other libraries on older Android, assuming it can be made to work like @xeffyr says. I just question dropping or moving into another slower repo about a third of the userbase.

[hyperpallium]

@joakim-noah most of my software development has moved to the newer and much faster smartphone

BTW a setup I'm thinking of: ssh into the new phone from the tablet. The faster device, on the larger display, and a double-headed development environment. tips:
I found wifi latency unbearable but bluetooth instant.
tmux groups allows different windows on different devices (which you can switch to, to type into, from the tablet).

[joakim-noah]

I found wifi latency unbearable

Yeah, this is why I don't do this much, only for short sessions the other way, when I need to test some code on the tablet.

but bluetooth instant

Not sure what this means, you run ssh over Bluetooth?

Never tried solutions like tmux or mosh, will get around to it one day.

I was actually planning on picking up a laptop shell like the Sentio Superbook and using it with my smartphone, but that project has been delayed for years.

In the meantime, I just used the smartphone itself and found it worked fine, with its 500+ PPI display. I go anywhere I want with my backpack, take out my foldable tablet stand and Rapoo Bluetooth keyboard, and can start typing away over the 4G connection from my phone, just like the last photo from this post that same guy posted later.

Interestingly, someone showed me their Macbook Air a couple months ago and my instinctual response was to try and put my hands on the display, to wipe away all the grease making the text so blurry. But there was none: its resolution was just so low, 128 PPI, that it looked horrible compared to the high-PPI mobile displays I'd gotten used to.

That's why mobile has killed off the PC for people like me, eventually for everyone.

[hyperpallium]

you can share internet over bluetooth (i.e. a bluetooth hotspot). In 5.1, it's:

Settings -> (wireless and networking) more -> Tethering & portable hotspot -> bluetooth tethering

On the other device's bluetooth, mark the first device for internet access.

Of course, if you can already see enough text on your phone, no need to lug around a tablet too. I'm finding my 5" phone is too small, considering a mi max 2 (6.4") or 7" tablet. We probably should shift this to another thread.

[joakim-noah]

Oh yeah, didn't people do that before WiFi hotspots? I'd forgotten about that, as I'd never tried it, didn't know it was still around.

[samoht0]

Well, putting 1/3 of the users (maybe more, as F-Droid user distribution is unknown) into some legacy branch really needs consideration.
Don't think, staying with current packages is an issue because of missing feature enhancements. But "critical bug fixes" include security fixes for me. And the easiest security fix is often to upgrade to the current upstream version. Back-ports can be a pain and would likely require some serious cve tracking. Just a few projects (like ffmpeg) maintain older major version (for a limited time).
There also should be a clear statement about the planed minimum duration of the legacy support. Maybe at least until the Android 5/6 general user base (according to playstore statistics) drops below 10 percent or something.
Termux app isn't exactly a big install package. Would it be an option to have main and legacy application in one package?
I'm not against the change, but for me, this needs a clear concept to convince the affected termux user base (including me).

[xeffyr]

But "critical bug fixes" include security fixes for me.

@samoht0 If you care about security, why not to update your Android then ? Android provides much more security issues than one application (Termux in this case). Termux is not OS and has little surface for remote attacks.

And the easiest security fix is often to upgrade to the current upstream version. Back-ports can be a pain and would likely require some serious cve tracking. Just a few projects (like ffmpeg) maintain older major version (for a limited time).

I think it should't be a problem to update packages, but new packages won't be added.

[samoht0]

If you care about security, why not to update your Android then? Android provides much more security issues than one application (Termux in this case). Termux is not OS and has little surface for remote attacks.

There's no working Android 7 build for the device. Slim6 or LOS 13 still get security fixes. Even LOS 11. So I don't see a problem, if there's a maintained official/unofficial build of an older version.

Using ffmpeg/curl/wget etc. to process remote data is relevant security matter in termux.

I think it should't be a problem to update packages, but new packages won't be added.

That sounds good. But needs to be clarified.

[miniBill]

As @hyperpallium said: what are the advantages to _users_ if dropping 5/6 support?

[xeffyr]

@miniBill

1. Porting packages & maintenance will be less difficult as Android 7 and higher brings more features to libc.

2. Higher Android versions have less bugs in libc.

3. Ability to use DT_RUNPATH so we can drop LD_LIBRARY_PATH and workarounds like done for package 'mpv': https://github.com/termux/termux-packages/blob/master/packages/mpv/build.sh#L59.

[Dmole]

As @hyperpallium said: what are the advantages to _users_ if dropping 5/6 support?

The same as some app dropping support for Windows 3.1: nothing; other than incentivizing them to upgrade their OS to something that sucks less.

[xeffyr]

@Dmole Of course, no one will support old OSes forever. Just progress always goes forward.

[Grimler91]

As @hyperpallium said: what are the advantages to users if dropping 5/6 support?

We could sort of spin it around as well. Users that are using newer android version would get a better experience using termux (no more LD_LIBRARY_PATH etc) and more packages can be ported since newer android version have a more complete libc.

So users with newer android versions (the majority) benefits from this upgrade.

[easyaspi314]

@Dmole The same as some app dropping support for Windows 3.1: nothing; other than incentivizing them to upgrade their OS to something that sucks less.

Except upgrading an Android version is nothing like upgrading a Windows version.

On Windows, you can just run the installer and it usually works fine.

For Android, many devices never see a newer Android version. They either need

• A flagship with guaranteed upgrades (but even promises don't get fulfilled -- my S II Skyrocket was promised 4.2.2 but never got past 4.1.2)
• A lucky update. That rarely happens on most of the low-end devices, which never get a single software update.
• An unofficial upgrade which needs tons of work and a root exploit.

That is why I think we should make a standalone build of libc, the linker, etc. It might also let us hardwire libtermux-exec into our linker/libc to make things faster, potentially fixing #2416, and also make crash dumps easier to access.

[joakim-noah]

I agree about shipping a Termux libc, I don't think the Android devs ever imagined an app that would run as much native code as Termux does.

[xeffyr]

GNU libc should be optimal solution for custom libc. It will allow to drop many compatibility patches as well as LD_LIBRARY_PATH environment variable. Localization of cli tools will be possible too.

We can run Linux chroots on Android device, so why custom libc be used ?

However, adding a custom libc will be the same major upgrade as like 'dropping android 6/7' as we need custom cross-compiler and deal with bugs that may occur.

[miniBill]

@xeffyr w.r.t. using a non-bionic libc, one problem I foresee is the whole seccomp syscall restriction of late Androids: see https://android.googlesource.com/platform/bionic/+/master/libc/ for the whitelist/blacklist files

plus, maintaining a libc is HUGE work.

@Grimler91 so the advantage is more stable packages (less libc bugs) and more packages in general (less libc bugs, easier porting and less kludges that might break packages)

@Dmole not everyone has the ability to "upgrade their OS to something that sucks less"

[robertvalik]

I have a three years old Lenovo Yoga 2 tablet. Still runs fine (because almost everything I use is running in Termux - nginx, php, mysql, vim, mutt; except browser). And I have no option to upgrade to higher Android version - still running 5.1 (yup, thanks, Lenovo) :-(

I understand that progress is inevitable but I dont see upgrading my tablet for next two years. Will there be a way to install older Termux and older packages if you decide to drop support for everything lower than 7?

[monoidic]

@easyaspi314

That is why I think we should make a standalone build of libc, the linker, etc. It might also let us hardwire libtermux-exec into our linker/libc to make things faster, potentially fixing #2416, and also make crash dumps easier to access.

As far as I can tell, simply dropping LD_LIBRARY_PATH (which isn't really possible with Android 5 & 6) would also fix #2416 without introducing the vast plethora of bugs and issues that could pop up from jumping to another libc/toolchain.

[joakim-noah]

I say stick with Bionic, but maybe start shipping Bionic 7 or 8 on Termux for Android 5/6. I also don't like the LGPL license used by the GNU libc, much prefer the BSD license of Bionic (for the same reason, looking forward to when google switches out GPL linux for non-GPL Fuchsia).

[hyperpallium]

@Grimler91 As @hyperpallium said: what are the advantages to users if dropping 5/6 support?
We could sort of spin it around as well. Users that are using newer android version...

No need to spin: "users" was not qualified and refers to all users.

So users with newer android versions (the majority) benefits from this upgrade

It's also how much benefit and how much detriment, not just how many. A great thing about software is a chance to avoid zero-sum games.

@xeffyr Of course, no one will support old OSes forever.

I submit that it's more effective to stick to concrete, specific facts, because abstract and general opinions are trivial to counter. e.g. vim still supports amigaOS

Just progress always goes forward.

e.g. "forward" depends on whose definition... Many people preferred Windows XP.. Android versions are increasingly google-integrated. Oracle increasingly restricts their licensing of Java. All three are "progress" from the vendor's perspective.

@Dmole nothing; other than incentivizing them to upgrade their OS to something that sucks less.

i.e. spending $100's on a new device. Termux is the only reason I consider android versions - benefits for users are minimal. And even if I invest in 8.1, what's its lifespan before termux moves on again?

However, termux already routinely makes changes for 7/8 users, at a cost to 5/6 users (mainly because the development devices have later versions). A separate 5/6 version would benefit 5/6 users. A great example is the sorely missed fullscreen/immersive mode, removed because of later android versions (though in termux-app, not termux-packages):

Remove the fullscreen setting
Trying to use fullscreen doesn't work well in a multi-windowed world... https://github.com/termux/termux-app/commit/de4f334e24a0676aa21d577f229dfaa4f595f9a0

But maybe I shouldn't have responded to all the above - I don't want to distract from the later discussions of libc etc, as they are much more technical, more interesting, more co-operative, and more constructive.

[vishalbiswas]

Shipping a newer version of bionic may not be the perfect answer.
A different libc than the system one may disable us from using any other system provided library because of any potential incompatibilities.

[xeffyr]

Shipping a newer version of bionic may not be the perfect answer.

Newer Bionic should be compatible with older Android versions but not vice versa.

I will try to use a libc from Android 8 on Android 6 to see if it will work in Termux.

A different libc than the system one may disable us from using any other system provided library because of any potential incompatibilities.

There only one major incompatibility - ABI.

[xeffyr]

Did a few experiments with linker & libc, libm, libdl from Android 8 on Android 6. Generally, Termux's utilities worked good with custom linker and libc, but as @vishalbiswas wrote above:

A different libc than the system one may disable us from using any other system provided library because of any potential incompatibilities.

custom libc caused problems with tools from /system/bin - they all segfaulted, possibly due to linker mismatch.

So, custom libc is not an option for Termux.

[samoht0]

So, custom libc is not an option for Termux.

Thanks for testing this option.
Back to topic, what happens on the legacy 5/6 branch, right?

[easyaspi314]

@xeffyr Are you sure that isn't a path issue? I know Bionic hardcodes some paths. And can I have commands to test?

Side note: I was able to get musl to work, and with some modifications to musl, I was able to get some existing libraries to work.

The main problems are the __sF array and the fortify functions.

__sF is a mess. I'll let this snippet explain it.

/* Before M the actual symbols for stdin and friends had different names. */
extern FILE __sF[] __REMOVED_IN(23);

#define stdin (&__sF[0])
#define stdout (&__sF[1])
#define stderr (&__sF[2])

Basically, this hack makes things complicated because musl's FILE struct is 124/232 bytes, and Bionic's is 84/152 bytes. While putting stdin, stdout, and stderr into an array is trivial, it won't work because the struct size won't line up.

I was able to put some of the items in another struct and point everything to it, but it is not fun to patch it each time unless we can agree to compatibility upstream.

I wasn't doing it on a git repo so I don't have a patch, but here is what I was toying with.
musl.tar.gz

I was able to get dash and nano (the source trees I had handy) to build against it, although zlib needs to be built from source.
The other things you need to do to build either nano or dash are:

• Symlink the following things in the musl prefix

$PREFIX/include/ -> $PREFIX/musl/include/
    curses.h
    histedit.h
    linux/
    magic.h
    ncurses.h
    ncurses_dll.h
    ncursesw/
$PREFIX/lib/ -> $PREFIX/musl/lib/
    libmagic.so*
    libedit.so*
    libncursesw.so*

• Make libdummy.so

echo | clang -x c -shared - -Oz -s -nostdlib -ffreestanding -o $PREFIX/musl/libdummy.so

Then symlink the following to it in $PREFIX/musl/lib:

libandroid-support.so
libm.so
libdl.so

[xeffyr]

@easyaspi314 Bionic & linker should be from the same ROM, otherwise 'strange' things may occur. This means if Termux will use own libc & linker, then you won't be able to use utilities from /system/bin, i.e. tools like 'logcat' won't work but Termux will work fine.

Statically compiled binaries are not affected by this problem as they don't use linker.

If you want to experiment with libc/linker from Android 8, get it here: oreo-libc.tar.gz (arm & aarch64).

To make program use a custom linker, you can patch it with patchelf. Example:

patchelf --set-interpreter $PREFIX/bin/linker busybox

---

Usage of custom libc may be possible if LD_LIBRARY_PATH will be removed, so tools from /system/bin will use Android's libraries and Termux will use own.

Maybe I should try to patch and build linker with libc from source, in that case it may work better...

[hyperpallium]

Indirectly related: what sort of compatibility problems have been found in the later versions of Android (7, 8, 9; Marshmellow, Nougat, Pie)? I get the impression it's:

• linux problems are basically solved (e.g. longer paths supported, so e.g. numpy works)

• java 8 problems are somewhat solved, but some things still don't work even in 9 Pie.

This could help anticipate the next level of support dropping... and to guide people buying new devices for termux (e.g. is 7 Marshmellow good enough, and for how long?)

[xeffyr]

linux problems are basically solved (e.g. longer paths supported, so e.g. numpy works)

This is libc problems, not kernel's.

java 8 problems are somewhat solved, but some things still don't work even in 9 Pie.

Android Pie introduces much more hidden problems for 'java', see https://github.com/termux/termux-packages/issues/3054.

This could help anticipate the next level of support dropping... and to guide people buying new devices for termux (e.g. is 7 Marshmellow good enough, and for how long?)

This actually depends on which features will be needed for Termux. Android 7 should be enough.

[hyperpallium]

linux problems are basically solved (e.g. longer paths supported, so e.g. numpy works)

@xeffyr This is libc problems, not kernel's.

Sorry, I don't understand. Are you saying that all the linux problems termux has to overcome are all libc problems? And that libc is not related to the version of Android?

Android 7 should be enough.

There's really no compatibility improvements in 8 or 9 that help termux packages? Wow, that would be great for me. I just want to really make sure before basing a buying decision on it.

[xeffyr]

@hyperpallium Kernel's specific problems are like missing support for swap, filesystem, namespace, etc... Libc-specific problems are like missing specific functions or linker issues, see 'fresh' example: https://github.com/termux/termux-packages/issues/3045#issuecomment-438679254.

There's really no compatibility improvements in 8 or 9 that help termux packages?

By looking at NDK headers, it seems that major changes happens each 2th API version, so in Android 9 (api 28) there are many changes. But I never seen a packages that require them.

Install ndk-sysroot silversearcher-ag, then see manually what has been added:

cd $PREFIX/include
ag '__INTRODUCED_IN\(28\)'

[benyaminl]

I'm curious about third party repos.
@its-pointless @xeffyr How will you guys handle the API Level change?

@vishalbiswas I can handle it in this way:

1. Android 5.0-compatible repository will be accessible from default URL, so sources list will not be changed:

deb https://termux-x11.ml x11 main

1. Android 7.0 will require a different sources list line:

deb https://termux-x11.ml x11-nougat main

It's not working without the key, could you add a tutorial for fixing the key? Thanks

[xeffyr]

@benyaminl

For 'tutorial', see https://github.com/termux/x11-packages#how-to-enable-this-repository.

P.S. Line deb https://termux-x11.ml x11-nougat main was example.

[benyaminl]

@xeffyr sorry to ask again, so this report isn't the mirror repo for lollipop, I mean to replace the main termux apt repo? Just the X11 package?

[xeffyr]

I mean to replace the main termux apt repo?

There no replacement for main repo. Lollipop support is not being dropped currently, it's just discussion.Just use Termux and don't change the main repo url.

Just the X11 package?

Package x11-repo enables APT access to the X11/gui packages only.

[twaik]

About libc: Termux can be wholy built with musl or uclibc and use Android tools/libc using libhybris's linker. It is not a problem. Libhybris can be built with all supported linkers and load device's one in runtime. When Android functions are unneded bionic is just not loaded.

[twaik]

About dropping Android 6&7 support.
We can implement some late Android features like ASharedMemory as it implemented there: https://github.com/xeffyr/libandroid-shmem-x11/blob/master/shmem.c#L619 . For now I am trying to implement AHardwareBuffer using this thing. It should support both Gralloc0 (7 and older) and Gralloc1 (8 and newer) devices. All linker stuff like loading greylisted libraries can be fixed in libhybris linker. Any other libraries can be built from sources.

[xeffyr]

It is not a problem.

There at least 2 problems:

1. Reliable custom toolchain. Solvable if there are people that can deal with compiler-related problems.

2. Entire incompatibility with libraries from /system/lib and /system/vendor. If Termux will use a custom libc, you can forget about hardware acceleration in x11-packages as usage of libGLES became impossible. The same goes for any other system/vendor library.

[twaik]

you can forget about hardware acceleration

Libhybris can provide an access to device's libGLESv2 and libEGL. Any other library can be accessed using libhybris too.

[twaik]

Solvable if there are people that can deal with compiler-related problems.

Can you provide some examples?

[xeffyr]

Can you provide some examples?

One time I tried to create an LFS-like environment for Termux which is based on musl-libc. I used instructions from the CLFS project (http://clfs.org/), but in addition applied patches for fixing paths so programs/libraries will be located and run in $PREFIX.

I built a GCC, Bash and set of basic utilities. Created environment looked fine until I decided to compile an additional program on device - some programs just randomly segfaulted and I was not able to identify problem.

Unfortunately, stuff of this experiment is no longer available.

---

I can do another attempt to create environment with alternate libc, but with patches from Alpine Linux project (https://github.com/alpinelinux/aports). With pushing scripts & target binaries to Github and cross-compiling environment will be pushed to docker hub.

However, I'm busy for now so don't expect quick progress with this.

[twaik]

@xeffyr ok.

[twaik]

I think we still can use last Bionic linker and avoid linker avoided bugs and use libhybris to link to device's libs :) We can fork bionic and apply these patches directly. Also we will no longer need android_support library.

[benyaminl]

Will it solve the problem on Android lollipop?

Sent from My Asus Pegasus X005

Pada tanggal Jum, 14 Des 2018 14.37 Twaik Yont <[email protected]
menulis:

I think we still can use last Bionic linker and avoid linker avoided bugs
and use libhybris to link to device's libs :) We can fork bionic and apply these
patches
https://github.com/termux/termux-packages/tree/master/ndk-patches
directly. Also we will no longer need android_support library.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/termux/termux-packages/issues/2874#issuecomment-447241102,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NFYtPvAlSrHqyQZUp7qRiIlFV8ZTks5u41UwgaJpZM4W068h
.

[xeffyr]

@twaik About newer linkers, see https://github.com/termux/termux-packages/issues/2874#issuecomment-431628816. Linker is part of Bionic and should be exactly same version as libc.

@benyaminl No. If you try to use Oreo's linker with Lollipop's libc - you will get segfault.

In other words, to have a newer linker, we should have a newer libc.

[twaik]

I decided that linker is the part of libc. We can use newer linker and libc and open device's libs with libhybris.

[benyaminl]

Then there will be no cure for lollipop and above?

Sent from My Asus Pegasus X005

Pada tanggal Jum, 14 Des 2018 15.14 Twaik Yont <[email protected]
menulis:

I decided that linker is the part of libc. We can use newer linker and
libc and open device's libs with libhybris.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/termux/termux-packages/issues/2874#issuecomment-447249122,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NE11Aj5TkB_aKqDzV46rGTZZtULxks5u413bgaJpZM4W068h
.

[twaik]

If Termux will have its own linker and libc we will be able to run it on any version of Android libhybris supports.

[xeffyr]

I can do another attempt to create environment with alternate libc, but with patches from Alpine Linux project (https://github.com/alpinelinux/aports). With pushing scripts & target binaries to Github and cross-compiling environment will be pushed to docker hub.

All stuff for making experimental musl-libc based environment will be in https://github.com/xeffyr/termux-stuff/tree/master/termux-musl.

[xeffyr]

I compiled a basic packages (specifically bootstrap-ones) with musl-libc and did some experiments to check features what it gives.

Termux app that uses a custom bootstrap archive can be found here: https://github.com/xeffyr/termux-stuff/releases/download/2019.01.18/Termux.Musl-v1.03.apk.

So, what features we may have from custom (musl) libc, but that's likely not a complete list:

• No more LD_LIBRARY_PATH. Termux and Android will automatically detect correct library paths.
• Libc API/ABI will be same on all Android versions, so we don't need patches that implement features missing on older Android version.
• Termux will have own "system-wide" /etc/resolv.conf and /etc/hosts, so resolver configuration will be same for all packages. This will likely deal with issues like https://github.com/termux/termux-packages/issues/3124 + adds possibility of configuring vhosts for nginx and similar.

Also, interesting if musl-based packages will work correctly on Android 5. If so, then it may be a solution for Android 5/6 devices. But that needs testing, as I have tried packages only on Android 8.

[tomty89]

I suppose it won't be possible to do (local) audio output with such setup? (In other words, we cannot make use of our own libraries and the ones in Android at the same time like now?)

[twaik]

@xeffyr Can you please make bootstrap & repo for armeabi-v7a arch? I have no aarch64 devices.

@tomty89 Audio support will be available after adding libhybris support.

[benyaminl]

I've lollipop aarch64 device. Can you send the APK for me to test?

Sent from My Asus Pegasus X005

Pada tanggal Rab, 19 Des 2018 19.05 Twaik Yont <[email protected]
menulis:

@xeffyr https://github.com/xeffyr Can you please make bootstrap & repo
for armeabi-v7a arch? I have no aarch64 devices.

@tomty89 https://github.com/tomty89 Audio support will be available
after adding libhybris support.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/termux/termux-packages/issues/2874#issuecomment-448571347,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NLAcA1n9rb_hnozujj1N1G6Mw5o9ks5u6iuZgaJpZM4W068h
.

[twaik]

@benyaminl Read it again: https://github.com/termux/termux-packages/issues/2874#issuecomment-448545709

[xeffyr]

repo for armeabi-v7a arch?

@twaik Now added support for ARM. Links are updated.

[ghost]

@xeffyr what is wrong with glibc then? Many packages are well supported on glibc than musl or others at the cost of being heavyweight.

I have run some stuff w/ glibc loader. Especially nano and patchelf appear to work. But there needs to be some more investigation on LD_LIBRARY_PATH. Perhaps all binaries should be just built / patched with custom rpath and loader ld-linux-armhf.so.3.

Interestingly it allows to use fakechroot as a preload library while running unmodified binaries from, say, debian repositories. Example I have run nano w/ /etc/nanorc in some custom base.

So glibc is interesting at least as a package.

Well, if all this experiment goes well we may see a musl branch, a glibc branch and bionic of course. :P

Also, I think gentoo prefix for android used their glibc in prefix dir. They probably failed because they didn't have a terminal nor built binaries for one explicitly (although there was a sourceforge repo for terminal ide). I am unsure how it is at the current state.

[twaik]

@extremenerd mussl is lightweight libc. That is important on mobile devices. Nothing in termux-packages changes: packages are still built from source using build.sh script and packed to debs. The only change is libc.

[xeffyr]

what is wrong with glibc then?

It's really hard to understand glibc's internals to make right changes in it.

Many packages are well supported on glibc than musl or others at the cost of being heavyweight.

Yes, many packages were developed relying on glibc as standard.

Glibc won't significantly reduce the amount of patches needed to compile software in Termux. It also won't allow you to normally use binaries copied from Linux distributions - don't forget that Termux's file system layout is far different.

Also, musl-based programs don't need rpath/runpath/ld_libary_path.

So glibc is interesting at least as a package.

Then patch & compile it yourself 😁.

a glibc branch

No, I won't do glibc branch.

Also, I think gentoo prefix for android used their glibc in prefix dir.

Gentoo prefix is broken by design. They even didn't changed the paths and some progs searched in /bin for some binaries but on Android this directory is not exist.

[benyaminl]

I compiled a basic packages (specifically bootstrap-ones) with musl-libc and did some experiments to check features what it gives.

Termux app that uses a custom bootstrap archive can be found here: https://github.com/xeffyr/termux-musl/releases/tag/app-v1.0-1.

So, what features we may have from custom (musl) libc, but that's likely not a complete list:

• No more LD_LIBRARY_PATH. Termux and Android will automatically detect correct library paths.
• Libc API/ABI will be same on all Android versions, so we don't need patches that implement features missing on older Android version.
• Termux will have own "system-wide" /etc/resolv.conf and /etc/hosts, so resolver configuration will be same for all packages. This will likely deal with issues like #3124 + adds possibility of configuring vhosts for nginx and similar.

Also, interesting if musl-based packages will work correctly on Android 5. If so, then it may be a solution for Android 5/6 devices. But that needs _testing_, as I have tried packages only on Android 8.

Seems your build working on my ASUS Pegasus X005 with Lollipop 5.1
Thanks God. Anyway how to port the rest of the termux package now? I see there's not a lot of it anyway. I also never gone into that stuff, but I want to learn and invest my time, any suggestion?

No warning error on Lollipop 5.1 ( For Android 5.0, I've one but it's not broken because the architecture is x86, dunno why)

One with Official Termux

And this's yours.

I just tried dumb thing, maybe another thing to test? Thanks

[Grimler91]

Let's perhaps take the bionic/musl/glibc discussion somewhere else (new issue?) as I believe it is only partly related to the android 5&6 drop. Such a transition would need way more testing before it can happen compared to a split of the repos into android 5&6 and android >=7 I guess.

Anyways, so roughly 35 % of users use android 5 or 6 which is a quite large part of the users. When would a android 5&6 drop feel justified? At 20 %? 10 %?

[benyaminl]

10% off them I think. Because until today many company still produce died
Android device. 😑

Sent from My Asus Pegasus X005

Pada tanggal Jum, 4 Jan 2019 22.07 Henrik Grimler <[email protected]
menulis:

Let's perhaps take the bionic/musl/glibc discussion somewhere else (new
issue?) as I believe it is only partly related to the android 5&6 drop.
Such a transition would need way more testing before it can happen compared
to a split of the repos into android 5&6 and android >=7 I guess.

Anyways, so roughly 35 % of users use android 5 or 6 which is a quite
large part of the users. When would a android 5&6 drop feel justified? At
20 %? 10 %?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/termux/termux-packages/issues/2874#issuecomment-451469177,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NA4DQ5ynr71a38Yh4-n-bd30b-jeks5u_24tgaJpZM4W068h
.

[samoht0]

Anyways, so roughly 35 % of users use android 5 or 6 which is a quite large part of the users. When would a android 5&6 drop feel justified? At 20 %? 10 %?

As I commented above, I think, that depends on the maintenance of the legacy branch.

If it's just a freeze of feature/packages with updates still provided, I wouldn't mind doing it even at current distribution level.
But if a "real drop" is in question (leaving packages unmaintained in a separate repo), I would consider 10 % as suitable mark.

[Wetitpig]

@fornwall how is the current user distribution? How many percent are using android 5 or 6?

This is the user distribution of active installs of the Termux app from the Play store:

Android version Installations (%)
8.0 20.0
7.0 18.4
6.0 18.1
7.1 14.3
8.1 12.0
5.1 9.8
5.0 3.8
Here active means: Number of Android devices that have been active in the previous 30 days on which your app is installed.

Is there any new update? There may be a number of users who have upgraded their Android version already.

[benyaminl]

Could be the option isn't dropping lollipop please?

[Wetitpig]

@benyaminl It is the problem with the timing rather than whether doing so. Anyway we need to drop Lollipop support later as Lollipop is already outdated.

[xeffyr]

@Wetitpig Once we will find a different hosting, we will create a separate repository. Then Android 5/6 drop. So question is not in "time".

[xeffyr]

Could be the option isn't dropping lollipop please?

No. Lollipop repo will continue exist some time.

[Grimler91]

Once we will find a different hosting, we will create a separate repository.

This was discussed in the termux development sessions, summaries of the discussions can be find on the wiki here: https://wiki.termux.com/wiki/Dev:Development_Sessions for those interested (full logs of conversations are found on gitter in the termux/dev channel and they are also sent to the termux-dev mailing list: https://groups.io/g/termux-dev).

[fornwall]

The master branch is now for Android 7.0 and later and is available for alpha testing on bintray. Join https://gitter.im/termux/dev or the mirrorred IRC channel #termux/development on freenode if you want to help testing (aarch64 devices only initially).

The termux.net repo will continue to be updated as previously, only that the packages will now come from the android-5 branch.