Ble.sh: ble modifies PS0

[tycho-kirchner]

ble version: 0.4.0-devel3+c257299
Bash version: 5.0.11(1)-release, x86_64-pc-linux-gnu

The title says it all: sourcing ble modifies bash's PS0-prompt, which some users use.

Thank you

Tycho

~
tycho@d:~$ PS0='$(echo ps0 >&2)'
tycho@d:~$ echo ok
ps0
ok
tycho@d:~$ source ~/tmp/ble.sh/ble.sh/out/ble.sh
ps0
tycho@d:~$ echo ok two
ok two
tycho@d:~$
~

[akinomyoga]

Thank you for the report! Hmm, actually I intentionally kill stderr in evaluating PS0. This is to suppress debug messages on set -x. May I ask why don't you just let the shell output the results?

PS0=$'ps0\n'

# or
PS0=$(echo ps0)$'\n'

# or, if the command doesn't necessarily output lines,
PS0=$(some-command;echo $'\r')

[tycho-kirchner]

Ah, ok, thanks for the quick response. Any way not to kill stderr? I'm running a custom setup function in PS0 which may fail and relies on stderr being available.

[akinomyoga]

I'll try to allow stderr in PS0 in the next commit. Nevertheless, I don't think it is a good idea to bypassing the mechanism of PS0. But, if you want to really output something to the terminal, you can actually redirect it to >/dev/tty. Something like

    PS0="$PS0"'$(_shournal_ps0 \# 2>/dev/tty)'
    PS1="$PS1"'$(_shournal_ps1 \# 2>/dev/tty)'

P.S. `～` is a deprecated form of command substitutions, which is discouraged by the POSIX standard. There are still many documents that use this older form, but I recommend you to completely switch to the form $( ) unless you target at the original Bourne shell (which is now only available as /bin/sh in Solaris as far as I know).

[tycho-kirchner]

Thanks.
And wow, you found my _shournal_ps0 really quick ^_^
What do you mean by "bypassing the mechanism of PS0"?

[akinomyoga]

What do you mean by "bypassing the mechanism of PS0"?

Ah, OK. I meant that the idea of PS0 is to output the expanded result of PS0 to the terminal. If one provides something to output as the expanded result of PS0, the shell will output it. But it seems that you try to output something directly to the terminal without passing it to the PS0 expanded result, which appeared to me like bypassing.

[akinomyoga]

OK! I pushed a commit 24a88ce! It was actually really a simply fix. The original reason that I suppressed stderr was to suppress debug messages by set -x when the user sets it, but actually the PS0 processing was just before restoring the user setting of set -x, so I could simply allow stderr!

Can you check the behavior? You can update ble.sh by running ble-update in a ble.sh session.

---

And wow, you found my _shournal_ps0 really quick ^_^

Yeah. Actually, it is very often that I cannot reproduce the issues from users just by looking at their reports, but sometimes I can reproduce the issue by using their dotfiles. So I have a habit of looking at the repositories of the users who reported issues...

---

By the way, if ble.sh is loaded, you could instead use

blehook PREEXEC+='_shournal_ps0 \#'
blehook PRECMD+='_shournal_ps1 \#'

without affecting PS0 and PS1. But, of course, you are always free to use PS0 and PS1 which can be used also for normal Bash.

[tycho-kirchner]

Nice. stderr now works as expected and in fact we're getting closer to the real issue.
ble seems to change the command-sequence counter \#:
test.sh:
~~~
ps0(){
echo "ps0 seq $1" >&2
}

ps1(){
echo "ps1 seq $1" >&2
}

PS0='$(ps0 #)'
PS1="$PS1"'$(ps1 #)'
~~~

---

~
tycho@d:~$ source test.sh
ps1 seq 2
tycho@d:~$ echo ok
ps0 seq 2
ok
ps1 seq 3
tycho@d:~$ echo ok2
ps0 seq 3
ok2
ps1 seq 4
tycho@d:~$ source ~/tmp/ble.sh/ble.sh/out/ble.sh
ps0 seq 4
ps1 seq 4
tycho@d:~$ echo ok3
ps0 seq 5
ok3
ps1 seq 5
tycho@d:~$
~

[akinomyoga]

Ah, good point. For this one, I need to rethink the structure of the execution model...

[akinomyoga]

I added a fix 8b0257e for \#, etc. Finally, I didn't need to change the execution model but just added a small constraints.

---

Details: ble.sh separates the command editing module (A) and the command execution module (B). When the user requests the execution of the command, (A) sends a command data to the queue of (B). (B) would run the received commands when ble.sh is idle. This means that (B) might run multiple commands at the same time when ble.sh is busy, where the execution might become

$ COMMAND1
$ COMMAND2
[PS0]
COMMAND1
[PS0]
COMMAND2
[PS1]
PROMPT$ 

I thought this can complicate the updates of \# if we try to update it after PS0, but I have changed so that (B) will try to run the command soon even when it is not idle. Now basically the above case will not happen (unless a user writes a custom editing function that sends multiple commands for one keystroke).

Actually, I noticed that the above [PS0]COMMAND1[PS0]COMMAND2[PS1] can happen even in the normal Bash when the user input multiline commands using C-vC-j, so I decided not to make too much efforts to completely exclude such cases.

I also noticed that the command execution count \# is the information only available in prompts, so I could easily change the update rule of \# without affecting other parts.

---

I'm not sure what shournal expects exactly, but if it expects that PS0 and PS1 are always paired around each command, blehook PREEXEC and blehook POSTEXEC are actually more clean and reliable for that purpose. Maybe you can instead use them when ble.sh is loaded.

[tycho-kirchner]

Great, shournal's integration tests now pass without errors, so at least the normal stuff works :+1: .

Actually, I noticed that the above [PS0]COMMAND1[PS0]COMMAND2[PS1] can happen even in the normal Bash

I was not able to verify this. Could you please describe in more detail, how to reproduce that?

I'm not sure what shournal expects exactly

It expects that PS0 is run immediately before the command-sequence is actually executed (but PS0 is not run on hitting Ctrl-c or ENTER). PS1 on the other hand is also run, if Ctrl-c is pressed.
shournal does expect that between two PS0 at least one PS1 was run.

[tycho-kirchner]

Ok, my daughter just closed the issue - sorry for that ^_^

[akinomyoga]

Actually, I noticed that the above [PS0]COMMAND1[PS0]COMMAND2[PS1] can happen even in the normal Bash

I was not able to verify this. Could you please describe in more detail, how to reproduce that?

You can insert a newline character in the command line by C-v C-j (press Ctrl-v and then Ctrl-j). See the following example (where user inputs are shown by boldface).

[murase@chatoyancy 1 ble.sh]$ bash --norcRET /* start plain Bash */
[murase@chatoyancy 0 ble.sh]$ PS0='[PS0]\n' PS1='[PS1]\n\$ 'RET
[PS1]
$ C-v C-j /* insert a newline in the command line string */
echo Command 1C-v C-j /* insert another newline */
echo Command 2RET /* run */
[PS0]
Command 1
[PS0]
Command 2
[PS1]
$

[tycho-kirchner]

Thanks for these detailed instructions, which I was able to reproduce. However, you seem to imply that PS1 is not executed between Command 1 and Command 2, however, what really happens is that the stdout of PS1 is supressed but it is still executed.:

PS0='$(echo ps0 out; echo ps0 err >&2)\n' PS1='$(echo ps1 out; echo ps1 err >&2)\n'RET
ps0 err
ps0 out
ps1 err
ps1 out
echo Command 1 C-v C-j
echo Command 2 RET
ps0 err
ps0 out
Command 1
ps1 err
ps0 err
ps0 out
Command 2
ps1 err
ps1 out

[akinomyoga]

however, what really happens is that the stdout of PS1 is supressed but it is still executed.:

Oh, I see! It is surprising to me that PS1 was actually evaluated in the middle of the execution of multiline commands. I didn't know that until now!

Anyway, ble.sh treats the multiline commands as a single execution as a whole, so it only evaluates PS0 once before all the commands and PS1 once after all the commands:

/* In a ble.sh session */
[murase@chatoyancy 0 ~]$ PS0='$(echo "[PS0]">&2)' PS1='$(echo "[PS1]">&2)\$ 'RET
[PS1]
$ C-q C-j
echo Command1RET
echo Command2C-j
[PS0]
Command1
Command2
[PS1]
$

In ble.sh

I think the current behavior of ble.sh is basically enough for shournal, but in theory, the following case might still happen in ble.sh (when the user inputs many commands in a row very fast but at some specific speed in very slow machiine), but the probability should be very low: [ Edit: I could reproduce this behavior by pasting multiline commands in « a terminal which doesn't support the bracketed paste mode (DECSET(2004)) » under the setting bleopt accept_line_threshold=-1. It does happen when multiline commands are pasted in « a terminal without the support of the bracketed paste mode » when « the number of characters in the multiline command except for the first line » is shorter than accept_line_threshold or when accept_line_threshold is negative ]

$ echo Command1RET
$ echo Command2RET
[PS0]
Command1
[PS0]
Command2
[PS1]
$

And also, ble.sh occasionally re-evaluates PS1 in other situations. For example, when it received SIGWINCH, when the editing mode is changed, when the prompt setting has been changed in the widget (editing function), when other plugins (such as fzf-marks) requested the update of the prompt, etc.

I will not change this behavior in order to make PS1 useful to detect the end of the command because it is the necessary calculation to update the prompt and also because the purpose of PS1 is to enable the customization of the prompt and not to provide the way to detect the end of the command. Using PS1 for the command-end detection is just a hack. The same for PS0. If you want a more reliable way to detect the beginning and end of the command execution in ble.sh, could you use the afore-mentioned PREEXEC and POSTEXEC in the presence of ble.sh? PREEXEC and POSTEXEC are exactly introduced for such kind of purpose and are ensured to be executed once for one command execution in any situation.

[murase@chatoyancy 0 ble.sh]$ blehook POSTEXEC+='echo POSTEXEC >&2'
POSTEXEC
[murase@chatoyancy 0 ble.sh]$ blehook PREEXEC+='echo PREEXEC >&2'
POSTEXEC
[murase@chatoyancy 0 ble.sh]$ echo yes
PREEXEC
yes
POSTEXEC

If you have any questions on the usage or any problems in using them, please feel free to ask me for the detail!

[akinomyoga]

@tycho-kirchner Hi, how are you doing? if you don't have further issues related to shournal, may I close this issue?

[tycho-kirchner]

Sorry, I did not dig furhter into this due to lack of time (homeschooling, etc). If you want, we can close this for now.
Once I have more time I'll take a closer look into ble's PREEXEC and POSTEXEC hooks. Basically shournal requires history 1 to be set to the current command-string in PS0 and the last exitcode $? in PS1. In fact shournal could even drop PS1 altogether and use the PROMPT_COMMAND instead, but I don't know how ble.sh deals with that one.
Anyway, thank you so far (:

[akinomyoga]

OK. Thank you for your reply!

Once I have more time I'll take a closer look into _ble_'s PREEXEC and POSTEXEC hooks. Basically _shournal_ requires history 1 to be set to the current command-string in PS0 and the last exitcode $? in PS1.

In ble.sh, history 1 cannot be reliably used to retrieve the current command inside PS0, but if you set a function name to PREEXEC, you can get the command string as a first argument of the function. You can access the exit code of the previous user command through $? in both POSTEXEC and PS1 in ble.sh (but of course $? needs to be saved by the first command inside the hook because it will be overwritten after the first command).

$ function preexec1 { echo "$FUNCNAME: command = $1"; }
$ function postexec1 { echo "$FUNCNAME: exit code = $?"; }
$ blehook PREEXEC+='preexec1' POSTEXEC+='postexec1'
$ (echo hello; exit 123)
preexec1: command = (echo hello; exit 123)
hello
postexec1: exit code = 123

In fact _shournal_ could even drop PS1 altogether and use the PROMPT_COMMAND instead, but I don't know how _ble.sh_ deals with that one.

The timing of PROMPT_COMMAND execution is the same as PS1 evaluation in ble.sh so is neither reliable in ble.sh for the detection of the end of user commands.

Anyway, thank you so far (:

You're welcome! Please feel free to create a new issue or reopen this issue when you are to try PREEXEC and POSTEXEC. I close the issue for now. Thank you again for the information on the behavior of PS0!