Termux-packages: GNU Parallel not working

[forresthopkinsa]

Problem description
GNU Parallel doesn't seem to work at all. I can't say if it used to or not, this is my first attempt.

Steps to reproduce

$ ls
a  b  c
$ parallel echo {} ::: *
Can't use string ("1") as a HASH ref while "strict refs" in use at /data/data/com.termux/files/usr/bin/parallel line 7210.

Expected behavior

$ ls
a  b  c
$ parallel echo {} ::: *
a
b
c

Additional information

$ termux-info
Updatable packages:
All packages up to date
Subscribed repositories:
https://dl.bintray.com/grimler/science-packages-24 science/stable
https://dl.bintray.com/grimler/game-packages-24 games/stable
https://dl.bintray.com/termux/termux-packages-24 stable/main
System information:
Linux localhost 4.14.83-perf+ #1 SMP PREEMPT Wed Aug 28 17:21:39 CST 2019 aarch64 Android
Termux-packages arch:
aarch64
Android version:
9
Device manufacturer:
OnePlus
Device model:
GM1917

$ parallel --version
GNU parallel 20190922
Copyright (C) 2007-2019 Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.

[Fernal73]

Same issue:

I tried to run the following command as per the parallel tutorial at
https://zenodo.org/record/1146014#.Xaqmnx5X40M:

seq 5 | parallel seq {} '>' example.{}

and receive the following message:

Can't use string ("1") as a HASH ref while "strict refs" in use at /data/data/com.termux/files/usr/bin/parallel line 7210.

[Grimler91]

Related: https://stackoverflow.com/questions/22517768/parallel-reading-of-input-file-with-parallelloops-module

[Fernal73]

@Grimler91 I haven't gone through your link as yet thoroughly but the above tutorial works as expected for Arch Linux on Termux.

The parallel version on Arch Linux is 20190722.

It's 20190922 on Termux.

The Perl versions are both: v5.30.0.

[xeffyr]

@Fernal73 Version 20190922 works fine on Arch. Problem is Termux specific and probably lives somewhere in perl package.

[Fernal73]

I'm unable to verify that version on Arch Linux. Do you know the mirror on
which that version is available?

[xeffyr]

I built it myself. Just ./configure --prefix=/tmp/parallel && make && make install then executed it from /tmp/parallel/bin/.

I didn't tried it with Termux patches. However, I'm not sure if they cause problems.

[Fernal73]

Ah. That's not something I want to do. Thanks for the update. I'd rather work with a stable release than build it from source. Are you expecting me to fix that for you?

[ole-tange]

I am pretty sure the problem is due to Perl somehow being more picky on Termux.

So I would expect you get the same issue if you install an earlier version of GNU Parallel, but not if you installed an earlier version of Perl.

I just tried compiling Perl-5.30.0 on my Ubuntu machine, and I do not get that error when runnning:

PERLLIB=lib ./perl  `which parallel` echo ::: 1

If you manage to reproduce the error on a (normal) GNU/Linux system, I may be able to fix it.

A simple work-around right now may be to remove the line 'use strict' (around line 32). That ought to make it stop complaining.

And now that I have your attention: I am really curious what you are using GNU Parallel for in Termux. So far I have seen it as "we have GNU Parallel in Termux just because we can", but I have the feeling you actually _use_ it for something. Will you be willing to tell us, what you use it for?

[Fernal73]

I do use gnu parallel but not your package on Termux.
It's used on Arch
Linux on Termux which has a different installation of Perl and Gnu Parallel.

I used it to run my build, static analysis and test scripts faster on my
phone.

https://github.com/Fernal73/LearnJava?files=1

Check the build scripts ending with 'all'.

Thanks for your help.

I encountered the error with Gnu Parallel (on Termux) when I decided to try the tutorial.
I'm currently running the tutorial on Arch Linux on Termux.

https://github.com/Fernal73/UnixWorkbench/tree/master/Parallel

Great to have a response to my query from the creator of GNU Parallel himself.

Yep, my scripts are not mission-critical!

[ole-tange]

I tested the workaround on my phone: It works.

So disable the 'use strict;' in line 32 and everything should work.

(Yes: GNU Parallel is a perl script, so no need to compile stuff or anything. Just edit /data/data/com.termux/files/usr/bin/parallel)

Until I can reproduce the error on GNU/Linux I suggest the Termux maintainers just remove that line: You lose no functionality by doing so.

[Fernal73]

What line 32 are you referring to?
In the Perl installation on Termux?
I'm inclined to wait for the next minor version than alter anything on my
installation.

[ole-tange]

'use strict;' is used to punish me if I write bad code style.

It is rarely updated when a new Perl version comes out, but it looks as if it as been updated in 5.30 for Termux. And right now I cannot see what the bad style is in line 7210, so that is why I need to be able to reproduce the error on a non-android system.

If you prefer waiting, it is up to you. But at least you now know a simple workaround.

[ole-tange]

Oh, the line 32 is 'use strict;' in the file: /data/data/com.termux/files/usr/bin/parallel

[Fernal73]

Thanks. That worked. I'm not familiar with Perl except for testing Regular Expressions.

[forresthopkinsa]

And now that I have your attention: I am really curious what you are using GNU Parallel for in Termux. So far I have seen it as "we have GNU Parallel in Termux just because we can", but I have the feeling you actually _use_ it for something. Will you be willing to tell us, what you use it for?

I have a Oneplus 7 Pro, which is capable of recording drop-dead-gorgeous 4k/60fps video, which I've gotten into the habit of using for everything.

Problem is, 4k60fps video takes a lot of space, which maxes out my Google Photos storage pretty much instantly.

I'm trying to use Tasker+Termux to:

1. Watch for new files in the Camera folder
2. Use ffprobe to see if they're 4k
3. Move 4k videos to a non-backed-up directory
4. Use ffmpeg to make a downscaled (1080p/24fps) copy
5. Put the 1080p version in the backed-up Camera folder
6. Once on wifi, upload the 4k file to private server

Weird use case, I know. But I'm pretty used to using Parallel for scripting, and it's particularly useful when running operations on a ton of files at once.

[ole-tange]

Found the problem.

sub sct_android($) {
    # Returns:                                                                                                        
    #   { 'sockets' => #sockets                                                                                       
    #     'cores' => #cores                                                                                           
    #     'threads' => #threads                                                                                       
    #     'active' => #taskset_threads }                                                                              
    # Use GNU/Linux                                                                                                   
    return sct_gnu_linux(@_);
}

Should be:

sub sct_android($) {
    # Returns:                                                                                                        
    #   { 'sockets' => #sockets                                                                                       
    #     'cores' => #cores                                                                                           
    #     'threads' => #threads                                                                                       
    #     'active' => #taskset_threads }                                                                              
    # Use GNU/Linux                                                                                                   
    return sct_gnu_linux($_[0]);
}

It will be part of version 20191122.

Thanks for finding the bug.

[Fernal73]

Great. Thanks, Ole! Now we can all sleep well knowing that all is well with GNU Parallel!

[Fernal73]

@ole-tange @Grimler91 @xeffyr

Are you aware that the semaphore and mutex examples from the tutorial don't work on Termux? Works on Arch Linux for Termux. Probably works on all PC variants on *Nix. That's for you to check. I'm only interested in the above two platform versions.

semaphore.sh

#!/usr/bin/env bash
sem --jobs 3 --id my_id -u 'echo Start 1; sleep 5; echo 1 done' && \
 sem --jobs 3 --id my_id -u 'echo Start 2; sleep 6; echo 2 done' && \
 sem --jobs 3 --id my_id -u 'echo Start 3; sleep 7; echo 3 done' && \
 sem --jobs 3 --id my_id -u 'echo Start 4; sleep 8; echo 4 done' && \
 sem --wait --id my_id

sem --id foo -u 'echo Slow started; sleep 10; echo Slow ended' && \
 sem --id foo --semaphoretimeout 1 'echo Forced running after 1 sec' && \
 sem --id foo --semaphoretimeout -2 'echo Give up after 2 secs'
sem --id foo --wait

mutex.sh

#!/usr/bin/env bash
sem 'sleep 10; echo The first finished' && \
echo The first is now running in the background && \
sem 'sleep 10; echo The second finished' && \
echo The second is now running in the background
sem --wait

sem --fg 'sleep 10; echo The first finished' && \
echo The first finished running in the foreground && \
sem --fg 'sleep 10; echo The second finished' && \
echo The second finished running in the foreground
sem --wait

sem --id my_id -u 'echo First started; sleep 10; echo First done' && \
sem --id my_id -u 'echo Second started; sleep 10; echo Second done' \ 
sem --id my_id -u 'echo all done;' 
sem --wait --id my_id

[Fernal73]

https://www.npmjs.com/package/parallelshell

There's also this nodejs command line version of gnu parallel which is not as advanced.

[ole-tange]

@ole-tange @Grimler91 @xeffyr

Are you aware that the semaphore and mutex examples from the tutorial don't work on Termux?

https://savannah.gnu.org/bugs/?57160