Termux-packages: build-package.sh renders termux in a broken state

[Lindydancer]

Hi!

I'm trying to build a local copy of the package "emacs" inside the termux environment, and I have runed into a number of problems along the way. The most important one is that after running the build-package.sh script, the entire "termux" installations becomes broken. The only way to recover is to delete the termux application from my device and reinstall it. For the minor problems, see the section "Steps to repeat" below.

Steps to repeat: (I hope I have remembered all the steps.)

pkg install termux-exec

Exit the shell and open a new shell.

pkg install git
pkg install curl
pkg install python
pkg install clang
git clone https://github.com/termux/termux-packages.git
cd termux-packages

Edit the script scripts/setup-android-sdk.sh. Add the line mkdir android-sdk before the unzip command.

Run the script scripts/setup-android-sdk.sh (ignore errors about java not being installed).

Patch the script build-package.sh:

• Uncomment the call to termux_step_handle_buildarch. (The function seems to retrieve settings from previous builds. Unfortunately, it stores information in the file /data/TERMUX_ARCH, something a normal termux session doesn't have write access to.)

• Remove three occurrences of --silent. The reason is that the patch command in termux doesn't support the --silent option.

Run build-package.sh emacs.

Now, Termux is left in an unusable state. The effect is that after the build-package.sh script, termux no longer finds commands like sh. If I start a new session window I get the following error message:

exec("/data/data/com.termux/files/usr/bin/login"): No such file or directory

If you need more information to find the cause of this, please let me know.

-- Anders Lindgren (Emacs developer)

[Grimler91]

Hi,

build-package.sh isn't suppose to work on device. Building with it should be done in ubuntu 17.04 (or the docker image).

About the missing option to patch, you are probably using the busybox version of patch (do pkg install patch).

The reason Termux is left in an unusable state (maybe/probably) is because the script unsets environmental variables (and termux needs LD_LIBRARY_PATH=/data/data/com.termux/files/usr/lib and probably PATH=/data/data/com.termux/files/usr/bin). If it happens in a new shell (that isn't a child shell), then that's interesting though, doesn't your termux work at all now?

If you truly want to compile emacs on device then it would be easier to start from scratch, sed all /bin/sh to /data/data/com.termux/files/usr/bin/sh and download&apply all the emacs patches that are available in termux-packages/packages/emacs and thereafter try to solve any errors that occur.

Another problem with emacs in particular is that it won't build with clang so you will need to install gcc from its-pointless repository, see instructions here.

May I ask why you want to compile it on device instead of cross-compiling it from a gnu/linux system (which probably would be easier)?

[Lindydancer]

Hi!

build-package.sh isn't suppose to work on device. Building with it should
be done in ubuntu 17.04 (or the docker image).

Ok. I was trying to follow the instructions in the README.md file, and I
got the impression that this should work. After re-reading it, I now
understand that I need an Ubuntu or Docker installation.

About the missing option to patch, you are probably using the busybox

version of patch (do pkg install patch).

I used the pre-installed ones.

The reason Termux is left in an unusable state (maybe/probably) is because

the script unsets environmental variables (and termux needs
LD_LIBRARY_PATH=/data/data/com.termux/files/usr/lib and probably
PATH=/data/data/com.termux/files/usr/bin). If it happens in a new shell
(that isn't a child shell), then that's interesting though, doesn't your
termux work at all now?

No, it's totally broken, as I can't open new shells. I need to delete the
Termux application and reinstall it and start with a new empty environment.
(Which I've done this a couple of time now...)

If you truly want to compile emacs on device then it would be easier to

start from scratch, sed all /bin/sh to /data/data/com.termux/files/usr/bin/sh
and download&apply all the emacs patches that are available in
termux-packages/packages/emacs
https://github.com/termux/termux-packages/tree/master/packages/emacs
and thereafter try to solve any errors that occur.

Another problem with emacs in particular is that it won't build with clang
so you will need to install gcc from its-pointless repository
https://github.com/its-pointless/gcc_termux, see instructions here
https://wiki.termux.com/wiki/Package_Management#its-pointless_.28live_the_dream.29
.

May I ask why you want to compile it on device instead of cross-compiling
it from a gnu/linux system (which probably would be easier)?

What I want is an Emacs where bug 29104 has
been fixed. Without this bug, local keymaps in the mode-line are ignored.
Effectively, this means that you can't click on things like the name of the
buffer to change buffers, like you can in the GUI version of Emacs. My plan
is to write a package to add features to the mode line which is useful for
touch devices (maybe even a simple keyboard), so that I don't have to open
up the real keyboard all of the time. The bug will not be fixed in the
upcoming Emacs 26 (as it was fixed too late), so the plan is to build Emacs
from the git archive.

I thought it should be straight forward to build stuff in Termux (like it
is in other distributions), but apparently it was't. I never touched
Docker, so I'll try to do this in Ubuntu.

Anyway, I understand if you haven't got the time to figure out why this
breaks the Termux installation, as the script wasn't intended to be used
inside Termux in the first place. However, one thing you could do is add a
test in the script and exit with an error message, in case it's executed in
Termux.

-- Anders

[Grimler91]

However, one thing you could do is add a test in the script and exit with an error message, in case it's executed in Termux.

That is a good idea. And we should update the README to be a bit more clear.

It should be pretty straight forward to build from emacs' git repo if there are tar files available. If you need to git clone then some manipulation of build-package.sh is needed (for example by overriding termux_step_extract package so that it does git clone instead of termux_download).

If you need to autoconf then you can add a termux_step_pre_configure function doing that (as in for example expect/build.sh).

My plan is to write a package to add features to the mode line which is useful for touch devices (maybe even a simple keyboard), so that I don't have to open up the real keyboard all of the time.

Sounds like a great feature that would be very useful for termux users (and others) :+1:.

Let's keep the issue open until build-package.sh and the readme has been updated.

[SDRausty]

Can build-package-on-device.sh be added to https://github.com/termux/termux-packages/tree/master/scripts @Grimler91?

[fornwall]

Thanks - the README has been clarified and a check has been added to build-package.sh!

[fornwall]

@Lindydancer I can update the emacs package to contain the patch in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=29104 (that is, what was committed in https://github.com/emacs-mirror/emacs/commit/51a5ccbe850da9fa5b58cee169fd5dbe8917f4e7#diff-6d3eacf6c7bb7f797e56679a36135944) if you want?

[fornwall]

There is now an updated emacs package at version 25.3-1 available, which contains the above mentioned commit backported.

[SDRausty]

@fornwall can you shorten the default startup scripts in emacs on Termux too? The startup time for emacs is excessive, and why is emacs loading all those languages at emacs boot in Termux?

[Lindydancer]

On Thu, Dec 7, 2017 at 3:45 AM, Fredrik Fornwall [e-mail]
wrote:

There is now an updated emacs package at version 25.3-1 available, which
contains the above mentioned commit backported.

Man får tacka och bocka! (i.e. "Thanks!")

I just tested it and it works perfectly!

-- Anders

[Grimler91]

@sdrausty you can shorten the loading time by:

1. Having a emacs --daemon running in the background that you can connect to with emacs --client. I start one in my .bashrc (if it isn't running already).
2. Starting emacs with emacs -Q (quick).

[SDRausty]

@Grimler91 Thank you for sharing this cool information. I followed your advice. How can this:

Loading language/chinese...
Loading language/chinese...done
Loading language/cyrillic...
Loading language/cyrillic...done
Loading language/indian...
Loading language/indian...done
Loading language/sinhala...
Loading language/sinhala...done
Loading language/english...
Loading language/english...done
Loading language/ethiopic...
Loading language/ethiopic...done
Loading language/european...
Loading language/european...done
Loading language/czech...
Loading language/czech...done
Loading language/slovak...
Loading language/slovak...done
Loading language/romanian...
Loading language/romanian...done
Loading language/greek...
Loading language/greek...done
Loading language/hebrew...
Loading language/hebrew...done
Loading international/cp51932...
Loading international/cp51932...done
Loading international/eucjp-ms...
Loading international/eucjp-ms...done
Loading language/japanese...
Loading language/japanese...done
Loading language/korean...
Loading language/korean...done
Loading language/lao...
Loading language/lao...done
Loading language/tai-viet...
Loading language/tai-viet...done
Loading language/thai...
Loading language/thai...done
Loading language/tibetan...
Loading language/tibetan...done
Loading language/vietnamese...
Loading language/vietnamese...done
Loading language/misc-lang...
Loading language/misc-lang...done
Loading language/utf-8-lang...
Loading language/utf-8-lang...done
Loading language/georgian...
Loading language/georgian...done
Loading language/khmer...
Loading language/khmer...done
Loading language/burmese...
Loading language/burmese...done
Loading language/cham...
Loading language/cham...done

Be suppressed? I don't want emacs loading all these languages except for English.

Emacs in Arch Linux in Termux PRoot does not seem to have this issue and loads lightening fast as it should.

[Lindydancer]

The fact that Emacs takes longer to start on Termux is because it's not
"dumped".

On other operating systems, when building Emacs, an Emacs process is
started and a lot of packages are loaded. The next step is to "dump" it,
which means that the executable and all its data is written to disk. When a
user later starts Emacs, this image is loaded, and with a lot of packages
including the language support files.

On Termux, on the other hand, Emacs is started from scratch which takes a
bit longer.

I'm new to Termux, so I have no real insight into why Emacs isn't dumped on
Termux, but I suspect it has to do with the fact that it's cross compiled
and thus can't be started on the build machine.

-- Anders

On Thu, Dec 7, 2017 at 6:11 PM, S D Rausty [e-mail] wrote:

@Grimler91 https://github.com/grimler91 Thank you for sharing this cool
information. I followed your advice. How can this:

Loading language/chinese...
Loading language/chinese...done
Loading language/cyrillic...
Loading language/cyrillic...done
Loading language/indian...
Loading language/indian...done
Loading language/sinhala...
Loading language/sinhala...done
Loading language/english...
Loading language/english...done
Loading language/ethiopic...
Loading language/ethiopic...done
Loading language/european...
Loading language/european...done
Loading language/czech...
Loading language/czech...done
Loading language/slovak...
Loading language/slovak...done
Loading language/romanian...
Loading language/romanian...done
Loading language/greek...
Loading language/greek...done
Loading language/hebrew...
Loading language/hebrew...done
Loading international/cp51932...
Loading international/cp51932...done
Loading international/eucjp-ms...
Loading international/eucjp-ms...done
Loading language/japanese...
Loading language/japanese...done
Loading language/korean...
Loading language/korean...done
Loading language/lao...
Loading language/lao...done
Loading language/tai-viet...
Loading language/tai-viet...done
Loading language/thai...
Loading language/thai...done
Loading language/tibetan...
Loading language/tibetan...done
Loading language/vietnamese...
Loading language/vietnamese...done
Loading language/misc-lang...
Loading language/misc-lang...done
Loading language/utf-8-lang...
Loading language/utf-8-lang...done
Loading language/georgian...
Loading language/georgian...done
Loading language/khmer...
Loading language/khmer...done
Loading language/burmese...
Loading language/burmese...done
Loading language/cham...
Loading language/cham...done

Be suppressed? I don't want emacs loading all these languages.
[image: screenshot_20171207-120620]
https://user-images.githubusercontent.com/27742457/33728000-0cb55960-db47-11e7-80de-9863a5d7c85a.png
[image: screenshot_20171207-120637]
https://user-images.githubusercontent.com/27742457/33728018-1a974fd4-db47-11e7-9d48-a27b23243ebc.png

Emacs in Arch Linux in Termux PRoot does not seem to have this issue and
loads lightening fast as it should.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/termux/termux-packages/issues/1910#issuecomment-350033097,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADBjQI5RfARJcVDps51S8ogZqi5NmKZbks5s-BwmgaJpZM4Q4BOZ
.

[SDRausty]

@Lindydancer Thank you for your response. It appears our mouths are watering while studying this article https://emacs.stackexchange.com/questions/37301/how-to-build-emacs-from-sources-on-termux which, when successfully compiled on device, should give emacs the desired speed increase. How to accomplish this in just a couple of keystrokes in Termux on device?

[Lindydancer]

Well, I'm the one that asked the question...

The answer, unfortunately, is that it's not possible to do that without a
major effort.

-- Anders Lindgren (I go under the name Lindydancer in the Emacs

comminuty)

On Thu, Dec 7, 2017 at 7:18 PM, S D Rausty [e-mail] wrote:

Thank you for your response. It appears our mouths are watering while
studying this article https://emacs.stackexchange.
com/questions/37301/how-to-build-emacs-from-sources-on-termux which, when
successfully compiled on device, should give emacs the desired speed
increase. How to accomplish this in just a couple of keystrokes?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/termux/termux-packages/issues/1910#issuecomment-350051248,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADBjQA1NNIxSssQa4SKnqg4cL6sFni9hks5s-CwOgaJpZM4Q4BOZ
.

[SDRausty]

Thank you for asking online.

[fornwall]

@Lindydancer

I'm new to Termux, so I have no real insight into why Emacs isn't dumped on
Termux, but I suspect it has to do with the fact that it's cross compiled
and thus can't be started on the build machine.

The reason that emacs is built undumped is that Android (as of version 5.0) only supports running position-independent executables (PIE) - and that is only possible undumped. See for e.g. http://www.openwall.com/lists/oss-security/2015/03/13/13 for more information.

Emacs in Arch Linux in Termux PRoot does not seem to have this issue and loads lightening fast as it should.

@sdrausty That's interesting! I would guess that the binary is undumped, since AFAIK dumped emacs does not work with PIE? Could you check by running file on the emacs binary, and check if it says LSB shared object (which means PIE) or LSB executable (which means non-pie)?

[SDRausty]

Reinstalling emacs in Arch Linux at Termux PRoot at present. Here is ours

[21:00 ~ ]$ file -b  /data/data/com.termux/files/usr/bin/emacs-25.3
ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/bin/linker64, stripped

Hopefully the result we are awaiting shall be forthwith. We are seeking results from this one emacs-25.3-1-aarch64. Possibly arm7 results forthcoming asap too. Emacs was never uninstalled. It must have gotten flushed in a purge. One of many bash setupTermuxArch.sh --purge purges on devices while debugging this script.

[SDRausty]

Here is our first successful answer:

[21:50 home ]$ file  -b /usr/bin/emacs-25.3
ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=87068aaba9a463f742fa9e3c9bd136dfd153ee26, stripped

To our question first question. Upon retesting: emacs loading time = lightening speed in Arch Linux in Termux PRoot on a smartphone in Android. @fornwall

[SDRausty]

Arm7 Android 5 emacs LSB results.

Community forum: https://termux.com/community
IRC channel: #termux on freenode
Gitter chat: https://gitter.im/termux/termux
Mailing list: [email protected]

Search packages: pkg search
Install a package: pkg install
Upgrade packages: pkg upgrade
Learn more: pkg help
$ which emacs
/data/data/com.termux/files/usr/bin/emacs
$ file -b /data/data/com.termux/files/usr/bin/emacs-25.3
ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, stripped
$ startarchi
Welcome to Arch Linux in Termux! Enjoy!

Chat: https://gitter.im/termux/termux/
Help: info and man
Portal: https://wiki.termux.com/wiki/Community

Install a package: pacman -S
More information: pacman [-D|F|Q|R|S|T|U] --help
Search packages: pacman -Ss
Upgrade packages: pacman -Syu
[04:26 home ]$ which emacs
/usr/bin/emacs
[04:30 home ]$ file -b /usr/bin/emacs-25.3
ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=e1fbb69cd46a4b42cb549509f261664d57a29eab, stripped

[Grimler91]

Feels like we should be able to dump emacs on device (for example in the postinst script) to speed it up for everyone, but it seems that our emacs doesn't have the dump-emacs function.

Following the steps at https://www.emacswiki.org/emacs/DumpingEmacs gives:
Symbol’s function definition is void: dump-emacs

I guess "CANNOT_DUMP=yes" disables the function from being included in emacs (and not only prevents dumping during cross compilation).

[Grimler91]

I played around a bit and tried compiling emacs with the dump-emacs function (but still without dumping during cross-compilation) but got some memory problem when trying to dump on device:

$ emacs -batch -execute '(dump-emacs "myemacs" "temacs")'
...
[LOTS OF LOADING]
...
Finding pointers to doc strings...
Finding pointers to doc strings...done
Pure-hashed: 22476 strings, 3615 vectors, 35131 conses, 3516 bytecodes, 99 others
**************************************************
Warning: Your system has a gap between BSS and the
heap (4564062 bytes).  This usually means that exec-shield
or something similar is in effect.  The dump may
fail because of this.  See the section about
exec-shield in etc/PROBLEMS for more information.
**************************************************
emacs: Can't allocate buffer for /data/data/com.termux/files/home/temacs: Out of memory

temacs has been copied from /home/builder/.termux-build/emacs/build/src/temacs to the device.

This reference manual page provides some help that might be useful.

Sources are at https://github.com/Grimler91/termux-packages/tree/emacs if anyone else wants to have a look.

[fornwall]

To summarise my understanding at this point:

1. Undumped emacs is considerably slower.
2. Dumping produces a non-PIE executable which the Android (as of version 5.0) dynamic linker does not allow.
3. Arch Linux uses a dumped, non-PIE emacs executable.
4. This works as running Arch Linux in Termux PRoot does not use the Android system dynamic linker, but the glibc one from the Arch Linux installation.

[Grimler91]

Dumping produces a non-PIE executable which the Android (as of version 5.0) dynamic linker does not allow.

I don't know very much about dumping. As I understand it, it basically saves the current state (in this case, the state where emacs + all the files that are loaded at start-up are in memory) into a file, which can then later be loaded.
If this is correct then I'm assuming that dumping on device would generate a PIE executable (since all the programs we run on device are compiled with -pie).

Correct me if I'm misunderstanding something.

Decreasing emacs start-up time would be nice, but as I mentioned earlier: there are already work arounds (emacs --daemon or emacs -Q).

[fornwall]

@Grimler91 My understanding is that a dump will contain raw pointer values, so for it to work the executable will need to be loaded at an exact address (=not position-independent). See

• http://www.openwall.com/lists/oss-security/2015/03/13/13
• https://lists.gnu.org/archive/html/emacs-devel/2016-10/msg00350.html
• https://en.wikipedia.org/wiki/Position-independent_code#Position-independent_executables
• https://dev.haiku-os.org/ticket/12430

[Grimler91]

@fornwall Alright, thanks, I guess trying to get a dumped emacs up and running would be futile then, for now at least.

[SDRausty]

Can emacs be built on device in Termux PRoot? Which distributions are best for this?

[Lindydancer]

Hi!

At the end of last year I wrote:

My plan is to write a package to add features to the mode line which is
useful for touch devices (maybe even a simple keyboard), so that I don't
have to open up the real keyboard all of the time.

Sounds like a great feature that would be very useful for termux users
(and others) 👍.

It turned out to be much, much harder that I initially thought to turn the
mode-line into a keyboard However, after a deep dive into how Emacs handle
keymaps I finally managed to put something together. Unfortunately, it
needs Emacs 26 (which hasn't been officially released yet) for complex key
sequences, but plain typing works in Emacs 25.

In anyone of you would like to try it out before I release it on Melpa, you
can get it from:

https://github.com/Lindydancer/mode-line-keyboard

-- Anders

[Grimler91]

@Lindydancer So only difference between 25 -- 26 is that key sequences doesn't work in 25?

I tried it a bit on my phone, looks nice!

It seems that there is a minimum width, otherwise the mode-line is just truncated:

It's a quite big distance between android's normal soft keyboard and the top mode-line (is mode-line the correct termiology?) on a phone, which probably decreases the practical usability of those keys.
Having the arrow keys and esc, home etc at the mode line is really nice:

[Lindydancer]

On Thu, Apr 5, 2018 at 8:44 AM, Henrik Grimler [e-mail]
wrote:

@Lindydancer https://github.com/Lindydancer So only difference between
25 -- 26 is that key sequences doesn't work in 25?

Yes. Key sequences trigger an error in the C parts of Emacs 25, which I
haven't been able to work around, so I decided to wait the problem out...

I tried it a bit on my phone, looks nice!
>
Thanks!

It seems that there is a minimum width, otherwise the mode-line is just

truncated:

I designed the lines to be approximately 80 characters wide, since that is
the width of your typical document or source code. On a phone this
typically means that you have to run it in landscape mode. However, on a
larger tab it would probably work in either portrait or landscape mode.

It's a quite big distance between android's normal soft keyboard and the

top mode-line (is mode-line the correct termiology?) on a phone, which
probably decreases the practical usability of those keys.

Having the arrow keys and esc, home etc at the mode line is really nice:

The top line is known as the "header-line" and the bottom "mode-line".

I hadn't thought about the use case where you have the normal android
keyboard visible at the same time as the mode-line keyboard, but now that
you mention it it make sense. In that case you could place other keys in
the mode-line than what I originally picked.

Again, thanks for testing it!

-- Anders