Dietpi: General | New screen clear method to reliably preserve command history

[MichaIng]

Sometimes I want to recheck the bash history, especially after dietpi-software installation/uninstallation runs or dietpi-config executions, but many of our scripts include a clear command, which removes the history completely.

I would like to replace clear completely with printf '\ec', which clears the current visible screen as well, but preserves the history and therefore allows scolling up.

Reference: https://stackoverflow.com/a/5367075

@Fourdee
What do you think, or is it explicitly wanted to clear bash history as well?

[Fourdee]

@MichaIng

I would like to replace clear completely with printf '\ec', which clears the current visible screen as well, but preserves the history and therefore allows scolling up.

Yep sounds good 👍
Originally clear was used to minimize any background onscreen text (and text flash at top of screen) during menu navigation.

[MichaIng]

PR is up: https://github.com/Fourdee/DietPi/pull/1629

Would be nice if someone could test this on SBC local terminal.

• e.g. "/DietPi/dietpi/dietpi-banner 1" should clear the current terminal screen but preserve scollback buffer without including whiptail dialogs.

[Fourdee]

Scroll local term RPi 3:

• 🈯️ Shift + Page up/down

@MichaIng

[MichaIng]

@Fourdee
Great, happy here as well to have the full scroll buffer chosen within PuTTY at all time 😃.

[Fourdee]

Completed, @MichaIng 👍

[rdslw]

Guys (@MichaIng @Fourdee ), currently change from 'clear' to 'print \ec' actually did the opposite and the bug is not fixed.

printf '\ec' sends terminal control code to CLEAR HISTORY/BUFFER and current screen. Opposite to what the intention was. BTW command 'clear' will not be good also.

Probably we should have: printf '\e[2J\e[r' or just printf '\e[2J'

How to check it:

• gnome-terminal, kde-terminal etc: do from bash "printf '\ec'" and observe results.

Additional explanations: https://askubuntu.com/questions/25077/how-to-really-clear-the-terminal
Terminal escape codes: http://www.termsys.demon.co.uk/vtansi.htm

[MichaIng]

@rdslw
Many thanks for your info. On all terminal I tested, printf '\ec' works fine to only clear the current view, not the history. But to be true this highly varies between different terminals. On PuTTY, while clear really clears all, printf '\ec' even just moves the current view up into scrollback, hence nothing is lost. This would be lost with printf '\e[2J\e[r'.

That printf '\ec' does on PuTTY is actually what we want in every case, moving current view up, to not loose any script output or cmd history. What should work most reliable is using stty size to estimate the current run scroll down that amounts of lines via printf '\eD'. Sadly this scrolls 1 line per call only. If anyone knows how to easily scroll all text out of current view into history, I'd love to implement it.

[rdslw]

@MichaIng
I tested (again, just now, just to be sure :) ):

• rxvt (ver 9.22)
• xterm (ver 351 )
• gnome-terminal (ver 3.34.2)

• macos terminal (os ver10.14)
• putty (ver 0.73)

Win and Macos are the ones which do not clear history on '\ec'. I'm not sure if this is correct, as according to terminal escape codes '\ec' is supposed to clear history (http://www.termsys.demon.co.uk/vtansi.htm) however I'm not sure if this url is correct source.

Linux ones do clear buffer :(

Defintely '\ec' is at least erratic and not implemented coherently?
I would reopen this issue and try to find sth else.

[MichaIng]

The following seems to work:

printf '\e[6n'
read -sdR line
line=${line#*[}
line=${line%;*}
printf "\e[$((line-1))S\e[H"
unset line

• First print is escape sequence to print current cursor position in format \e[<line>;<column>R.
• This only expends when printed to terminal, hence cannot be stored in variable directly. Since it has escape sequence format, printed to terminal, it shows bullshit since it is again parsed+expended... So the only way to use it is to read it back to variable directly.
• read -sdR line reads back the print, silent (read only, omit actual print), until delimiter "R" is reached and store in variable line.
• The following two string manipulations isolate the line number from string.
• print escape sequence to scroll up terminal by , so that all lines up from cursor are moved into history, out of screen. Second sequence then moves cursor to home position (top left).
• unset line variable, although I would safe the above steps into a function and initiate line as local variable instead.

Not beautiful, but:

• I could not find any simpler way to get current cursor position.
• There seems to be no sequence to scroll all content into history, which works reliable on all terminals.

@rdslw
If you can verify the above working successful on all terminals, I will add this as global function to use instead of printf '\ec'.

[rdslw]

I've tested and below are results:

• xterm: works ok.
• rxvt: work ok.
• gnome-terminal: works semi-ok (current screen is lost -> this is probably gnome-terminal problem.)
• macos terminal: works ok
• putty: works ok.

The more I deep into it the more non-standard behaviour of terminals Im observing - and this is in 2019 :) BTW I tried to check how they react to 'clear', and this was also hit and miss :)

Summary:
Function proposed by you is probably the best bet for now, unless we would like to resign from screen clearing which is also an option.

[MichaIng]

@rdslw
Okay, on gnome-terminal, what does the following do: printf '\e[5S'
It "should" scroll up terminal content by 5 lines by adding 5 new lines to the bottom. If current screen is lost by the whole sequence, then obviously this does not work reliably.

Or it does work, but:

printf '\e[6n'
read -sdR line
line=${line#*[}
line=${line%;*}
echo $line

does not print correctly the line of the cursor?

[rdslw]

first one, adds few empty lines (and scroll terminal accordingly). no history is lost. So looks like it works.

Second prints 30, which is correct number of rows in terminal i was testing it.

[MichaIng]

@rdslw
Interesting, then I am wondering which step of the whole sequence failed 🤔. Okay however, it is best we can do anyway. I'll add it with DietPi v6.28 as v6.27 shall be released soon.

[rdslw]

My guess would be (based on the result) that procedure cleared current view and moved cursor to top line. Last 30 lines of screen output was lost. I was testing it when just before executing the procedure, the cursor in the terminal was at the bottom line, while current view was full (and also there was some date in scrollback buffer).

[MichaIng]

@rdslw
Hmm, I'm still wondering what fails, as the first part result in correct cursor position and printf '\e[5S' adds empty lines to bottom correctly :thinking:. printf "\e[$((line-1))S\e[H" should then add correct amount of lines to to bottom first (to clear screen but move content up into history), before moving cursor up. Even moving the cursor up should not have an effect. Checking step by step would be an option to see what does not happen.

However all better as with printf '\ec' and I can run some tests + fine tuning on gnome-terminal at a later time. Will add is as function. Any suggestions what it should be name? G_CLEAR, G_TERM_CLEAR or something that doesn't sound like it clears "everything" :thinking: ...

[rdslw]

I check it and problem is in "printf "\e[$((line-1))S"

Starting situation:

• clean terminal, with 40 lines
• executed 'seq 1 100' to fill up screen and some buffer
• command prompt is at the very bottom after the seq

What happens after: printf "\e[20S"

• prompt is at very bottom
• there are 20 empty lines added above the prompt
• rest of screen looks ok (seq 83..100 visible)
• BUT if you scroll up, seq is missing 63..82 (inclusive) - those lines disappeared

What happens after printf "\e[$((line-1))S

• prompt is at the bottom
• reset of screen is clean
  *BUT if you scroll up, seq ends at 62, and this 62 was just above the empty screen

I would go with G_TERM_CLEAR or G_CLEAR_SCREEN.

[MichaIng]

@rdslw

reset of screen is clean

~And whole seq 1..100 is view-able when scrolling up?~
EDIT: Ah I think I misunderstood. So printf "\e[<integer>S" adds lines at bottom as intended but removes the same amount of lines at the top of the screen, instead of moving them into history. Damn thing...

*BUT if you scroll up, seq ends at 62, and this 62 was just above the empty screen

Hmm even when separating both sequences:

printf "\e[$((line-1))S"
printf '\e[H'

So the second sequence, which should move cursor to top left home position, then moves it somehow up more then wanted?

[MichaIng]

Hmm a different approach is to simply print the required amount of new lines explicitly:

printf '\n%.0s' $(eval "echo -n {1..$((line-1))}")

Not too beautiful with this eval echo construct, but I have no idea how to expend 1 to N in another way through this bash syntax, or any different method to concatenate a string/character N times without a loop.

Or do the loop:

str=
for ((i=1;i<$line;i++)); do str+='\n'; done
printf "$str"

Whole sequence:

printf '\e[6n'
read -sdR line
line=${line#*[}
line=${line%;*}
str=
for ((i=1;i<$line;i++)); do str+='\n'; done
printf "${str}\e[H"
unset line str

But both above only work when the cursor is at bottom position already. If its in the middle, newlines will not scroll the screen. So finally its the amount of lines of screen size, which works on all cursor positions, but requires external tput or stty size call.

[rdslw]

@MichaIng thank you kindly for tenacity!!

The loop solution works ok under gnome-terminal IF CURSOR is at the very bottom. screen is then clean, while everything is in the buffor, available for scroll-up.

Unfortunately it does not work properly (current view is not cleaned fully) if it was run when the cursor was in the middle of terminal (not at the bottom).

You can see it there: https://asciinema.org/a/XwFwG40cC2LE6PKnTbxYXQOYx

Let me know if you need more testing.

[MichaIng]

@rdslw
Jep, as I mentioned above 😉, however thanks for testing. The following should work:

lines=$(stty size)
lines=${lines% *}
str=
for ((i=1;i<$lines;i++)); do str+='\n'; done
printf "$str\e[H"

I don't like to rely on non-shell internal command stty, which makes this much slower, however I see now other method. Damn thing that gnome terminal "scrolls" be removing lines from top as well 🤔.

---

Tested on VirtualBox virtual "linux" terminal and "\e[S"-based scrolling does exactly nothing there. So this is generally not supported by all terminals.

Tested on LXTerminal and scrolling works but with same issue as gnome terminal: It removes as much lines from the top as it adds to the bottom.

[MichaIng]

PR merged to apply the above: https://github.com/MichaIng/DietPi/pull/3281
I hope I found all cases throughout our scripts to apply the new function.