Dietpi: rsync, scp & sftp not working for user dietpi

[AlbertEmil]

I am unable to transfer files from my local machine to a dietpi system (DietPi v6.6 running on Raspberry Pi Zero W) using rsync, scp or sftp.

scp:

bash$ scp foo.txt [email protected]:/home/dietpi
[email protected]'s password: 
tput: unknown terminal "unknown"
/DietPi/dietpi/func/dietpi-globals: line 297: ((: lines=(38+6)/ : syntax error: operand expected (error token is "/ ")
tput: unknown terminal "unknown"
\033[K\033[90m[\033[0m\033[32m  OK  \033[0m\033[90m]\033[0m Root access verified.

rsync:

bash$ rsync \
        --rsh="ssh -p 22" \
        --archive \
        --compress \
        --delete \
        --progress \
        --recursive \
        --verbose \
        . [email protected]:~/testfolder/
[email protected]'s password: 
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at /SourceCache/rsync/rsync-45/rsync/compat.c(61) [sender=2.6.9]
make: *** [push] Error 2

sftp (establishing connection failed):

bash$ sftp [email protected]
[email protected]'s password: 
Received message too long 219896633

Some further investigation showed (e.g. that serverfault question) that this issue might be related to the status output resp. banner messages after login. So I tried to disable the banner by renaming its file by

sudo mv /DietPi/dietpi/dietpi-banner /DietPi/dietpi/dietpi-banner_bak

However, the banner seems to be restored after reboot. So this did not have any effect.

As another solution I tried to disable the banner on Normal Login from inside the /DietPi/dietpi/login file which resulted in reduced output on login but does not improve anything besides that.

Might that issue be related to the banner/status output? If that is the case, how could be prevent any output on the (non-interactive) login shell?

[MichaIng]

@AlbertEmil
Thank you very much for your report. We will investigate.

Hmm, since when .bashrc output created issues like this? Quick test via dropbear client works well from one of mine DietPi to another.

# rsync --rsh="dbclient -p 22" --archive --compress --delete --progress --recursive --verbose . [email protected]:~/testfolder

Host '192.168.178.29' is not in the trusted hosts file.
(ecdsa-sha2-nistp521 fingerprint sha1!! ...)
Do you want to continue connecting? (y/n) y
[email protected]'s password:
sending incremental file list
created directory /root/testfolder
./
...

sent 8,030 bytes  received 332 bytes  576.69 bytes/sec
total size is 18,940  speedup is 2.27

According to your question:

• There is more output than just dietpi-banner. To test, please comment both DietPi lines within your ~/.bashrc:

#. /DietPi/dietpi/func/dietpi-globals
#/DietPi/dietpi/login

However, the banner seems to be restored after reboot. So this did not have any effect.

Jep, this is due to DietPi-RamDisk:

• Changed and new files will be synced to disk /boot/dietpi on shutdown/reboot, but deleted files will be not deleted on disk.
• Thus on startup, the still existing file from disk will be loaded into RamDisk /DietPi/dietpi.
• To avoid this, you would need to mv the file on both:

mv /DietPi/dietpi/dietpi-banner /DietPi/dietpi/dietpi-banner.bak
mv /boot/dietpi/dietpi-banner /boot/dietpi/dietpi-banner.bak
reboot
...
login as: root
[email protected]'s password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
[  OK  ] Root access verified.
/DietPi/dietpi/login: line 179: /DietPi/dietpi/dietpi-banner: No such file or directory
root@VM-Stretch:~#

• But as you can see, there is still output produced, one due to now missing file. Anyway reboot is not necessary to test, logout/login is enough.

=> Comment the lines in bashrc as mentioned above. Note that these will be added on reboot, as they are essential for DietPi to run properly. So comment them, just logout/login and try your file transfer. Reboot will bring everything back in place.

[AlbertEmil]

@MichaIng

After commenting both lines in user dietpi's .bashrc as suggested

There is more output than just dietpi-banner. To test, please comment both DietPi lines within your ~/.bashrc:

. /DietPi/dietpi/func/dietpi-globals

/DietPi/dietpi/login

the output is reduced to the bare minimum (which is more like the standard welcome message on Debian or Ubuntu systems) both scp, rsync and sftp are working as desired and files can be transferred.

However, I was able to reproduce this behavior by using either dropbear or openSSH as the SSH server on the dietpi machine and accessing the machine from Debian 9 as well as from Mac OS X 10.10.5.

[MichaIng]

@AlbertEmil
Ah, was able to reproduce the error when trying to send to user dietpi instead of root:

2018-04-14 18:04:32 root@micha:~/testfolder# rsync --rsh="dbclient -p 22" --archive --compress --delete --progress --recursive --verbose . [email protected]                                                                             :~/testfolder
[email protected]'s password:
protocol version mismatch -- is your shell clean?
(see the rsync man page for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(178) [sender=3.1.2]
2018-04-14 18:05:04 root@micha:~/testfolder# rsync --rsh="dbclient -p 22" --archive --compress --delete --progress --recursive --verbose . [email protected]:~                                                                             /testfolder
[email protected]'s password:
sending incremental file list
created directory /root/testfolder
./
testfile
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/2)

sent 108 bytes  received 77 bytes  28.46 bytes/sec
total size is 0  speedup is 0.00

The only difference is that /home/dietpi/.bashrc contains sudo /DietPi/dietpi/login, as there are root privileges necessary to gather some of the banner information (mainly network related, AFAIK). Quick test after removing just the sudo:

2018-04-14 18:08:41 root@micha:~/testfolder# rsync --rsh="dbclient -p 22" --archive --compress --delete --progress --recursive --verbose . [email protected]:~/testfolder
[email protected]'s password:
sending incremental file list
created directory /home/dietpi/testfolder
./
testfile
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=0/2)

sent 108 bytes  received 84 bytes  42.67 bytes/sec
total size is 0  speedup is 0.00

So finally, not the output from .bashrc is the issue, just the sudo inside. I moved it into /DietPi/dietpi/login to the only place, where it really seems needed sudo /DietPi/dietpi/func/obtain_network_details and removed root privileges check at beginning of /DietPi/dietpi/login. DietPi-Banner for user dietpi looks fine, script does not produce any error (although I don't use any autostart option, that is also done here) and rsync works as well.

@Fourdee
What do you think? Does one of our autostart options require root privileges? Games and media mainly, so should not, otherwise sudo Run_AutoStart would be needed as well 🤔, or just for the one where really needed...

[ragadeeshu]

Also unable to scp to the dietpi user, same error. scp to root as workaround.

[MichaIng]

@ragadeeshu
Did you try the workaround:

sed -i 's|/DietPi/dietpi/func/obtain_network_details|sudo /DietPi/dietpi/func/obtain_network_details|' /DietPi/dietpi/login
sed -i 's|sudo /DietPi/dietpi/login|/DietPi/dietpi/login' /home/dietpi/.bashrc

But it might lead to issues with DietPi autostart options, if you are not just booting/logging into console. We need to investigate/test it.

[Fourdee]

Notes:

• We need to exit .bashrc scripts when $TERM is unknown | tput: https://github.com/ArchiveTeam/warrior-dockerfile/issues/5#issuecomment-53665157
• ~We need to remove sudo from .bashrc and manually add sudo into login script as needed?~ Not the cause

---

Tests (will send commit once finished):

• 🈯️ SCP

root@DietPi:~# scp .bash_history [email protected]:/home/dietpi
[email protected]'s password:
.bash_history                                 100% 1004   472.3KB/s   00:00

• 🈯️ Rsync

root@DietPi:~# rsync \
>         --rsh="ssh -p 22" \
>         --archive \
>         --compress \
>         --delete \
>         --progress \
>         --recursive \
>         --verbose \
>         . [email protected]:~/testfolder/
[email protected]'s password:
sending incremental file list
created directory /home/dietpi/testfolder
./
.bash_history
          1,004 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=13/15)
.bashrc
          3,577 100%    1.14MB/s    0:00:00 (xfr#2, to-chk=12/15)
.profile
            148 100%   36.13kB/s    0:00:00 (xfr#3, to-chk=11/15)
.wget-hsts
            259 100%   42.15kB/s    0:00:00 (xfr#4, to-chk=10/15)
.cache/
.config/
.config/htop/
.config/htop/htoprc
            632 100%   61.72kB/s    0:00:00 (xfr#5, to-chk=2/15)
.config/pulse/
.config/pulse/15a3221b01944fef846859d61ce7789e-runtime -> /tmp/pulse-2L9K88eMlGn7
.local/
.nano/
.ssh/
.ssh/known_hosts
            222 100%   19.71kB/s    0:00:00 (xfr#6, to-chk=0/15)

sent 3,711 bytes  received 218 bytes  1,571.60 bytes/sec
total size is 5,865  speedup is 1.49

[MichaIng]

@Fourdee
Maybe we can combine this with: https://github.com/Fourdee/DietPi/issues/1477

• Move globals and login into /etc/profile.d/dietpi-login.sh?
• This makes somehow sense as all we do there is just needed for "login shells" and .bashrc is also for non-login shells, where our scripts again could cause issues, at least overhead for non-login connections (?).
• To have no valid $TERM may be also an issue for non-login connections?

Would be nice as well as we can leave .bashrc completely clean and have our own file to edit/replace in case of changes.

[Fourdee]

So yeah, we basically just needed to move the TERM + set dumb check to dietpi-globals which execs 1st.

[Fourdee]

@MichaIng

Would be nice as well as we can leave .bashrc completely clean and have our own file to edit/replace in case of changes.

Yep agree 👍

Ah, I see the issue with sudo now.

I'll mark this ticket as closed (now resolved for v6.8), switch back to the other: https://github.com/Fourdee/DietPi/issues/1477