Dietpi: General | Switch from "net-tools" to "iproute2" commands

Created on 30 Mar 2018 · 15Comments · Source: MichaIng/DietPi

Both packages provide commands with the same purpose: https://philosophos.github.io/articles/20170411~net-tools-VS-iproute2/
iproute2 is dependency for some core packages, e.g. ifupdown, resolvconf and mariadb-server-10.1.
net-tools has very rare and uncommon dependents.
iproute2 is actively developed, while net-tools is not any more, being considered as replaced:
- https://git.kernel.org/pub/scm/network/iproute2/iproute2.git
- https://sourceforge.net/projects/net-tools/
But the commands of net-tools are still widely used and found within guides/forums: https://packages.debian.org/de/stretch/amd64/net-tools/filelist

So to go with time I suggest we move the few net-tools commands we use up to the equivalent iproute2 commands. This also allows us to kick net-tools from DietPi core dependencies. If we really want that is another question, as many users might expect net-tools commands being available? Personally I prefer the output format of net-tools, looks a clearer in some cases... But if it's just about getting the info I prefer having just one package/set of commands in use instead of two with exactly same value.

Enhancement

Source

MichaIng

Most helpful comment

@Fourdee
Hmm wanted to add the above, but this doesn't work well on next data update.

I did some research: https://unix.stackexchange.com/questions/3446/shomehow-the-rx-tx-counters-on-the-interface-resets

Could it have been related to this (x32 OS does not support >4GB data)? Although it seems in this case that only the command outputs are effected and according to the solution script, the file scrapes not.
But maybe it's worth to test this, if we can manage to push the transfer data close to 4 GB?
Otherwise we need some NETWORK_USAGE_FIX_RECIEVED/SENT variable that is always added to NETWORK_USAGE_TOTAL_CURRENT_RECIEVED/SENT and is always adjusted if the above condition matches:

if (( $NETWORK_USAGE_TOTAL_CURRENT_RECIEVED < $total_previous_recieved )); then
    NETWORK_USAGE_FIX_RECIEVED+=$total_previous_recieved
fi

or something like this, to allow several resets/cap reaches as well. But hmm, I think, as also this is not necessarily precise (as long as it's not really the 4 GB cap), as we don't know home much was transferred between last update and cap, I would just leave it as it is and just keep the issue in mind, in case of future bug reports.

MichaIng on 28 Jun 2018

❤1 👍1

All 15 comments

Replacements, some even simpler via /sys/class/net 😄:

netstat

🈯️ https://github.com/Fourdee/DietPi/blob/testing/dietpi/dietpi-cloudshell#L705-L707

mtu_size=$(cat /sys/class/net/"$NETWORK_DETAILS_ADAPTER"/mtu)
network_usage_current_recieved=$(cat /sys/class/net/"$NETWORK_DETAILS_ADAPTER"/statistics/rx_bytes) Actually as the bytes can be scraped here, mtu value is not necessary any more, and it's even more precise as not just the maximum packet value is used?
- The current netstat scrape actually gets transferred error values?

root@DietPi:~# netstat -N -i | awk '{print $4}'

RX-ERR
0
0

network_usage_current_sent=$(cat /sys/class/net/"$NETWORK_DETAILS_ADAPTER"/statistics/tx_bytes)

route

🈯️ https://github.com/Fourdee/DietPi/blob/testing/dietpi/boot#L214

elif ! ip r | grep -q ' via '; then ? At least via seems to be a secure indicator for available gateway, like G flag from route.

ifconfig

🈯️ https://github.com/Fourdee/DietPi/blob/testing/dietpi/func/obtain_network_details#L52

ip l > "$FP_TEMP" PR: https://github.com/Fourdee/DietPi/pull/1790
🈯️ https://github.com/Fourdee/DietPi/blob/testing/dietpi/func/obtain_network_details#L102
IP_ADDRESS="$(ip a show $ACTIVE_DEVICE | grep -m1 'inet ' | awk '{ print $2 }' | sed -E 's/(:|\/).*$//')" PR: https://github.com/Fourdee/DietPi/pull/1790
🈯️ https://github.com/Fourdee/DietPi/blob/testing/dietpi/dietpi-bugreport#L171
"ip l"
🈯️ https://github.com/Fourdee/DietPi/blob/testing/dietpi/dietpi-cloudshell#L661
ip a "$NETWORK_DETAILS_ADAPTER" > "$FP_TEMP"
NETWORK_DETAILS_IP_INT="$(cat "$FP_TEMP" | grep -m1 'inet ' | awk '{ print $2 }' | sed -E 's/(:|\/).*$//')"
- The current command does not scrape the IP address correctly?

root@DietPi:~# ifconfig eth0 | grep -m1 'inet '| cut -d: -f2 | awk '{ print $1}'
inet

🈺 https://github.com/Fourdee/DietPi/blob/8cf60d9e708eeaa55b6f13635ef8dbb94793b68b/dietpi/dietpi-config#L3189 + 🈺 https://github.com/Fourdee/DietPi/blob/8cf60d9e708eeaa55b6f13635ef8dbb94793b68b/dietpi/dietpi-config#L3217

ETH_IP="$(ip a show eth$ETH_INDEX | grep -m1 'inet ' | awk '{ print $2 }' | sed -E 's/(:|\/).*$//')"
Sadly for getting the netmask there is no replacement in iproute2. The netmask is included via CIDR notation and would need translation. We could also use CIDR? Don't know whats the best solution here without keeping net-tools or using netmask or some 3rd party converter...

So replacements works fine besides I didn't find a good method to get netmask in IP range notation.
We could make conversion by ourself:

cidr2mask() {
  local i mask=''
  local full_octets=$(($1/8))
  local partial_octet=$(($1%8))

  for ((i=0;i<4;i++)); do
    if [ $i -lt $full_octets ]; then
      mask+=255
    elif [ $i -eq $full_octets ]; then
      mask+=$((256 - 2**(8-$partial_octet)))
    else
      mask+=0
    fi  
    test $i -lt 3 && mask+=.
  done

  echo $mask
}

Ref: https://www.linuxquestions.org/questions/programming-9/bash-cidr-calculator-646701/#post3433298

MichaIng on 31 Mar 2018

❤1 👍1

Another advantage if ip is, that it is doesn't require root privileges.

MichaIng on 22 May 2018

PR ready to test: https://github.com/Fourdee/DietPi/pull/1839

MichaIng on 11 Jun 2018

Notes, dietpi-config > Networking

Device "0" does not exist.
Device "0" does not exist.
/DietPi/dietpi/dietpi-config: line 3153: / 8 : syntax error: operand expected (error token is "/ 8 ")

Fourdee on 12 Jun 2018

👍1

@Fourdee
Whoopsie fixed: https://github.com/Fourdee/DietPi/commit/538112f62935b1b39e32f2f99b997d3278bbed3b

Btw.: The net mask conversion is maybe not needed. I put it inside to keep the info as it was, but CIDR notation might be sufficient: https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing

€: Another alternative for network statistics: ip -s l show eth0

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:b6:37:59 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    3914343    6400     0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    592907     3640     0       0       0       0

The -s[tatistics] option also works as ip -s a s eth0 but the above simply has less output lines. I didn't find a way to show the statistics only.
-o[neline] allows a one line output, which could to skip grep here and there.

MichaIng on 12 Jun 2018

👍1

Last thing to verify:

NB: #/sys/class/net/ values are being reset by system/kernel when they reach X size. Some sort of "cap".

Retest? Even on long time running dietpi.com server, values match with output of currently used netstat -i
Alternatives: cat /proc/net/dev, ip -s l show eth0
Otherwise workaround/fail_safe by sum new value to old value, if (( $NETWORK_USAGE_TOTAL_CURRENT_RECIEVED < $total_previous_recieved ))?

Test on dietpi.com running for 51 days:

root@DietPi:~# cat /sys/class/net/eth0/statistics/rx_packets
1355648031
root@DietPi:~# cat /sys/class/net/eth0/statistics/tx_packets
836666076
root@DietPi:~# cat /sys/class/net/eth0/statistics/rx_bytes
118365960242
root@DietPi:~# cat /sys/class/net/eth0/statistics/tx_bytes
2517213821894
root@DietPi:~# ip -s l show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether ...
    RX: bytes  packets  errors  dropped overrun mcast
    118366253415 1355655651 0       1       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    2517216422488 836668063 0       0       0       0
root@DietPi:~# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500 1355657892      0      1 0      836668319      0      0      0 BMRU
lo       65536 19051218      0      0 0      19051218      0      0      0 LRU
root@DietPi:~# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo: 4451406549 19051835    0    0    0     0          0         0 4451406549 19051835    0    0    0     0       0          0
  eth0: 118368400310 1355684911    0    1    0     0          0         0 2517239423193 836678878    0    0    0     0       0          0

🈯️ So far all values match 👍.

Test on own RPi2 running for 10 days:

2018-06-20 13:38:43 root@micha:/var/log# cat /sys/class/net/eth0/statistics/rx_packets
592984
2018-06-20 13:38:46 root@micha:/var/log# cat /sys/class/net/eth0/statistics/tx_packets
400995
2018-06-20 13:38:52 root@micha:/var/log# cat /sys/class/net/eth0/statistics/rx_bytes
198867848
2018-06-20 13:38:56 root@micha:/var/log# cat /sys/class/net/eth0/statistics/tx_bytes
119316631
2018-06-20 13:39:03 root@micha:/var/log# ip -s l show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether ...
    RX: bytes  packets  errors  dropped overrun mcast
    198872856  593035   0       294     0       0
    TX: bytes  packets  errors  dropped carrier collsns
    119317657  401046   0       0       0       0
2018-06-20 13:39:11 root@micha:/var/log# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500   593050      0    294 0        401051      0      0      0 BMRU
lo       65536    21336      0      0 0         21336      0      0      0 LRU
2018-06-20 13:39:20 root@micha:/var/log# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
    lo: 1695204   21336    0    0    0     0          0         0  1695204   21336    0    0    0     0       0          0
  eth0: 198874576  593055    0  294    0     0          0         0 119319613  401056    0    0    0     0       0          0

🈯️ Passed test as well, although the values are veeery low, compared to dietpi.com.

MichaIng on 20 Jun 2018

@MichaIng

NB: #/sys/class/net/ values are being reset by system/kernel when they reach X size. Some sort of "cap".

Yep, this happened on OPi device if I remember, 3.x kernel and a few years ago lol.

Fourdee on 20 Jun 2018

@Fourdee
Hmm okay, just to be on the secure side, we could check and in case sum old and new value up:

if (( $NETWORK_USAGE_TOTAL_CURRENT_RECIEVED < $total_previous_recieved )); then
    NETWORK_USAGE_TOTAL_CURRENT_RECIEVED=$(( $NETWORK_USAGE_TOTAL_CURRENT_RECIEVED + $total_previous_recieved ))
fi

MichaIng on 20 Jun 2018

👍1

@Fourdee
Hmm wanted to add the above, but this doesn't work well on next data update.

I did some research: https://unix.stackexchange.com/questions/3446/shomehow-the-rx-tx-counters-on-the-interface-resets

Could it have been related to this (x32 OS does not support >4GB data)? Although it seems in this case that only the command outputs are effected and according to the solution script, the file scrapes not.
But maybe it's worth to test this, if we can manage to push the transfer data close to 4 GB?
Otherwise we need some NETWORK_USAGE_FIX_RECIEVED/SENT variable that is always added to NETWORK_USAGE_TOTAL_CURRENT_RECIEVED/SENT and is always adjusted if the above condition matches:

if (( $NETWORK_USAGE_TOTAL_CURRENT_RECIEVED < $total_previous_recieved )); then
    NETWORK_USAGE_FIX_RECIEVED+=$total_previous_recieved
fi

MichaIng on 28 Jun 2018

❤1 👍1

@MichaIng

Could it have been related to this (x32 OS does not support >4GB data)? Although it seems in this case that only the command outputs are effected and according to the solution script, the file scrapes not.

Possible 👍

unsigned long int (32bit) = 4,294,967,295 max
unsigned long long int (64bit) = +18,446,744,073,709,551,615 max

Although on a 32bit machine (with C++ standards), 2x long int's are used to make up the 64bit long long int
https://stackoverflow.com/questions/330374/long-long-implementation-in-32-bit-machine

@MichaIng You are right 👍 🥇

ARMv7:

root@DietPi:~# cat /sys/class/net/eth0/statistics/rx_bytes
4248714927
root@DietPi:~# cat /sys/class/net/eth0/statistics/rx_bytes
20173913

ARMv8:

root@DietPi:~# cat /sys/class/net/eth0/statistics/rx_bytes
5868826024

I would just leave it as it is and just keep the issue in mind, in case of future bug reports.

Yep agree, arch limitation 👍

Fourdee on 29 Jun 2018

👍1

@Fourdee
Hmm, okay, but if we now identified the issue, which should be present on RPi's as well, then we should work around for all (( G_HW_ARCH < 3 ))?

Is the 4248714927 just frozen? This would be even worse then just reverted to 0.
And do cat /proc/net/dev + ip -s l show eth0 have the same issue?

MichaIng on 30 Jun 2018

@MichaIng

Yep, all get reset, when they reach the 32bit int limit (4.4~GB)

Client connecting to 192.168.0.113, TCP port 5001
TCP window size:  101 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.24 port 44468 connected with 192.168.0.113 port 5001
[  5] local 192.168.0.24 port 5001 connected with 192.168.0.113 port 53192
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  1.84 GBytes   527 Mbits/sec
[  5]  0.0-30.0 sec  5.83 GBytes  1.67 Gbits/sec

root@DietPi:~# ip -s l show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 88:d7:f6:c2:f0:42 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    2437201455 10394597 0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    1309784650 7185665  0       0       0       0

root@DietPi:~# cat /proc/net/dev
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
  tun0:       0       0    0    0    0     0          0         0      272       5    0    0    0     0       0          0
  sit0:       0       0    0    0    0     0          0         0        0       0    0    0    0     0       0          0
    lo:    1096      15    0    0    0     0          0         0     1096      15    0    0    0     0       0          0
  eth0: 2437203741 10394626    0    0    0     0          0         0 1309786646 7185675    0    0    0     0       0          0

then we should work around for all (( G_HW_ARCH < 3 ))?

Its tricky, as we have no way of knowing when the value is reset, we need to know this is we are to create our own total variable. And, we'd have to background monitor the whole thing for any changes every X seconds.

I suppose we could do?

if (( $current < $previous )); then

math_to_calculate_new_total=

fi

Fourdee on 30 Jun 2018

And, be nice to have network sent/receive stats in dietpi-config?

Fourdee on 30 Jun 2018

@Fourdee
Hmm, logically it should be reset exactly at 4294967296 bytes. We could check for x32 OS and new < old and then add 4294967296 to new value something like that:

local reset_count=0
[[ -f /tmp/.dietpi-net_reset_count ]] && reset_count=$(</tmp/.dietpi-net_reset_count)
if [[ x32 OS && $new < $old ]]; then
  ((reset_count++))
  echo $reset_count > /tmp/.dietpi-net_reset_count
fi
new=$(( $scrape + $reset_count * 4294967296))

But depending on how long the last check was ago, resets could have been missed.

I think as long as there is no other (non net-tools) tool/command to get the real non-reset net stats, there is no failure safe method that we can implement.

MichaIng on 2 Jul 2018

👍1

@MichaIng

I think as long as there is no other (non net-tools) tool/command to get the real non-reset net stats, there is no failure safe method that we can implement.

Yep agree, and requires a background process 👍
Its not worth the implementation currently I believe.

Lets mark this as completed, with the 32bit int limitation being a known arch issue.

Great work on this by the way 👍

Fourdee on 2 Jul 2018

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

help me Nextcloud

ludji49 · 91Comments

Image | Raspberry Pi OS (64-bit)

FusionPlmH · 173Comments

DietPi + AmiBerry | Optimized installation for Uae4arm

Fourdee · 113Comments

DietPi-Software | ownCloud/Nextcloud | Improve installation and configuration

MichaIng · 64Comments

Image | NanoPC T4

Fourdee · 83Comments