Linux: Raspberry pi slow 3.5 Mbps upload speed on ethernet device after upgrade to debian 10 release of Raspbian

[unix4life]

I have 4 raspberry pi's running.
I recently had one upgraded to debian 10.0 release and a second one to debian 10.1.
And I have an old pi running debian 9.9.

Everything works fine on all devices, except for the upload speed of the ethernet interface.
Two of them are connected to the same switch (10.0 & 9.9), but the release 10.0 has low speed on upload.
One other on 10.1 is connected to the router/modem and has the same low speed.
Before the upgrade to release 10 these pi's had almost maximum speed of my ISP upload speed (25 Mbps).

Some screen output from the 3 pi's:

root@unix4life:~# speedtest-cli
Retrieving speedtest.net configuration...
Testing from Ziggo (83.82.72.134)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Qweb | Full-Service Hosting (Alblasserdam) [31.23 km]: 30.102 ms
Testing download speed................................................................................
Download: 91.87 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.57 Mbit/s
root@unix4life:~# uname -a
Linux unix4life 4.19.71-v7+ #1262 SMP Mon Sep 9 18:30:38 BST 2019 armv7l GNU/Linux
root@unix4life:~# cat /etc/debian_version
10.1
root@unix4life:~#
------------------------------------------------------------------------------------------------
root@debian10:~# speedtest-cli
Retrieving speedtest.net configuration...
Testing from Ziggo (83.82.72.134)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Qweb | Full-Service Hosting (Alblasserdam) [31.23 km]: 32.297 ms
Testing download speed................................................................................
Download: 92.01 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.55 Mbit/s
root@debian10:~# uname -a
Linux debian10 4.19.58-v7+ #1245 SMP Fri Jul 12 17:25:51 BST 2019 armv7l GNU/Linux
root@debian10:~# cat /etc/debian_version
10.0
root@debian10:~#
-----------------------------------------------------------------------------------------------
root@sandbox:~# speedtest-cli
Retrieving speedtest.net configuration...
Testing from Ziggo (83.82.72.134)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Qweb | Full-Service Hosting (Alblasserdam) [31.23 km]: 40.372 ms
Testing download speed................................................................................
Download: 41.60 Mbit/s
Testing upload speed....................................................................................................
Upload: 24.17 Mbit/s
root@sandbox:~# uname -a
Linux sandbox 4.19.46+ #1231 Mon Jun 3 17:28:02 BST 2019 armv6l GNU/Linux
root@sandbox:~# cat /etc/debian_version
9.9
root@sandbox:~#

I also have some logging which I collect every night. I upgraded the Unix4Life pi on Aug 21

Speedtest Unix4Life at Mon Aug 19 04:30:01 CEST 2019
Download: 89.02 Mbit/s
Upload: 24.74 Mbit/s
----------------------------------------------------------------------------
Speedtest tuinhuis at Mon Aug 19 04:30:27 CEST 2019
Download: 27.32 Mbit/s
Upload: 22.07 Mbit/s
----------------------------------------------------------------------------
============================================================================
Speedtest Unix4Life at Tue Aug 20 04:30:01 CEST 2019
Download: 91.38 Mbit/s
Upload: 25.16 Mbit/s
----------------------------------------------------------------------------
Speedtest tuinhuis at Tue Aug 20 04:30:28 CEST 2019
.Download: 26.54 Mbit/s
Upload: 22.01 Mbit/s
----------------------------------------------------------------------------
============================================================================
Speedtest Unix4Life at Wed Aug 21 04:30:01 CEST 2019
Download: 92.03 Mbit/s
Upload: 14.82 Mbit/s
----------------------------------------------------------------------------
Speedtest tuinhuis at Wed Aug 21 04:30:32 CEST 2019
.Download: 23.47 Mbit/s
Upload: 22.11 Mbit/s
WARNING: Speedtest for tuinhuis is LOW: down 23 MB/s and up 22 MB/s !!
----------------------------------------------------------------------------
============================================================================
Speedtest Unix4Life at Thu 22 Aug 2019 04:30:01 AM CEST
Download: 59.58 Mbit/s
Upload: 3.55 Mbit/s
WARNING: Speedtest for Unix4Life is LOW: down 59 MB/s and up 3 MB/s !!
----------------------------------------------------------------------------
Speedtest tuinhuis at Thu 22 Aug 2019 04:30:31 AM CEST
Download: 22.93 Mbit/s
Upload: 22.01 Mbit/s
WARNING: Speedtest for tuinhuis is LOW: down 22 MB/s and up 22 MB/s !!
----------------------------------------------------------------------------
============================================================================
Speedtest Unix4Life at Fri 23 Aug 2019 04:30:01 AM CEST
Download: 91.83 Mbit/s
Upload: 3.59 Mbit/s
WARNING: Speedtest for Unix4Life is LOW: down 91 MB/s and up 3 MB/s !!
----------------------------------------------------------------------------
Speedtest tuinhuis at Fri 23 Aug 2019 04:30:29 AM CEST
.Download: 24.46 Mbit/s
Upload: 22.27 Mbit/s
WARNING: Speedtest for tuinhuis is LOW: down 24 MB/s and up 22 MB/s !!
----------------------------------------------------------------------------
============================================================================
Speedtest Unix4Life at Sat 24 Aug 2019 04:30:01 AM CEST
Download: 92.15 Mbit/s
Upload: 2.76 Mbit/s
WARNING: Speedtest for Unix4Life is LOW: down 92 MB/s and up 2 MB/s !!
----------------------------------------------------------------------------
Speedtest tuinhuis at Sat 24 Aug 2019 04:30:33 AM CEST
Download: 20.87 Mbit/s
Upload: 22.03 Mbit/s
WARNING: Speedtest for tuinhuis is LOW: down 20 MB/s and up 22 MB/s !!
----------------------------------------------------------------------------
============================================================================
Speedtest Unix4Life at Sun 25 Aug 2019 04:30:01 AM CEST
Download: 92.22 Mbit/s
Upload: 3.58 Mbit/s
WARNING: Speedtest for Unix4Life is LOW: down 92 MB/s and up 3 MB/s !!
----------------------------------------------------------------------------

Please investigate.

Thanks in advance!
Best regards,
Ad Kuijpers

[unix4life]

Even tried forcing 100 full duplex and disable nautoneg...but no effect.
ethtool -s eth0 speed 100 duplex full autoneg off

[pelwell]

Please try a local test, e.g. iperf between some of your Pis: https://www.linode.com/docs/networking/diagnostics/install-iperf-to-diagnose-network-speed-in-linux/

[unix4life]

On pi unix4life / 192.168.178.2:

root@unix4life:~# iperf -c 192.168.178.5
------------------------------------------------------------
Client connecting to 192.168.178.5, TCP port 5001
TCP window size: 74.4 KByte (default)
------------------------------------------------------------
[  3] local 192.168.178.2 port 56188 connected with 192.168.178.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   103 MBytes  86.0 Mbits/sec
root@unix4life:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.178.2 port 5001 connected with 192.168.178.5 port 50532
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  91.0 MBytes  76.2 Mbits/sec
^Croot@unix4life:~#

=========================================================

On pi sandbox / 192.168.178.5:

root@sandbox:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.178.5 port 5001 connected with 192.168.178.2 port 56188
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.1 sec   103 MBytes  85.5 Mbits/sec
^Croot@sandbox:~#
root@sandbox:~# iperf -c 192.168.178.2
------------------------------------------------------------
Client connecting to 192.168.178.2, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.178.5 port 50532 connected with 192.168.178.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  91.0 MBytes  76.3 Mbits/sec
root@sandbox:~#
=========================================

What can you say about the results?

[unix4life]

root@unix4life:~# iperf -c 192.168.178.5 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.178.5, TCP port 5001
TCP window size: 74.4 KByte (default)
------------------------------------------------------------
[  5] local 192.168.178.2 port 56198 connected with 192.168.178.5 port 5001
[  4] local 192.168.178.2 port 5001 connected with 192.168.178.5 port 50534
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  65.6 MBytes  54.8 Mbits/sec
[  4]  0.0-10.1 sec  27.1 MBytes  22.6 Mbits/sec
root@unix4life:~#

[pelwell]

For the most meaningful results choose 3 devices: two "good" Pis and the "bad" Pi. Make one of the "good" Pis the server, the other the client, and run the "-d" duplex test. Then use the "bad" Pi as the client in the duplex test.

If the "bad" Pi gives noticeably worse throughput in either direction then there may be a real network performance problem, but if the iperf numbers are approximately the same then the problem lies in one of two area:

1. the TCP stack is, for some reason, performing worse in a way that only affects multiple hops or longer round-trips, or
2. the speedtest bandwidth measurement app is performing badly.

[unix4life]

The speedtest-cli with the distribution contains a bug.
Downloaded the recent version and the problem is gone:-)

root@unix4life:~# /usr/bin/speedtest-cli --version
2.0.2
root@unix4life:~# /usr/bin/speedtest-cli
Retrieving speedtest.net configuration...
Testing from Ziggo (83.82.72.134)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Qweb | Full-Service Hosting (Alblasserdam) [31.23 km]: 29.001 ms
Testing download speed................................................................................
Download: 92.34 Mbit/s
Testing upload speed......................................................................................................
Upload: 3.57 Mbit/s

root@unix4life:~# ./speedtest-cli_212 --version
speedtest-cli 2.1.2
Python 2.7.16 (default, Apr 6 2019, 01:42:57) [GCC 8.2.0]
root@unix4life:~# ./speedtest-cli_212
Retrieving speedtest.net configuration...
Testing from Ziggo (83.82.72.134)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Creative Applications (Rotterdam) [39.50 km]: 28.303 ms
Testing download speed................................................................................
Download: 90.07 Mbit/s
Testing upload speed................................................................................................
Upload: 25.58 Mbit/s

Thanks for your help!!!

[benlongmire]

I just found this too. Installing via apt-get gave me slow upload. Downloading from GitHub gave me my expected upload.

https://github.com/sivel/speedtest-cli#just-download-like-the-way-it-used-to-be

[JamesH65]

Closing this issue as questions answered/issue resolved.

[oviliz]

@unix4life , I'm getting similar iperf bandwidth results. How do you justify these as have nothing to do with the speedtest-cli bug?
Trying to figure out what's causing the upload speed to an SSD over ethernet on an RPi4 at around 10MB/s, same LAN, (tried two RPi4, connected to different gigabit switches).