Packages: mwan3: dead on 4.9.58 kernel

[teqlee]

I'm install snapshot r5175 on ERL3, mwan3 not work.
My config work well on 17.01.4.

Interface status:
 interface wan is error and tracking is active
 interface wan2 is error and tracking is active

Check IP rules:

Missing 1 of the 2 required interface IP rules
Rules found:
1001:   from all iif pppoe-wan lookup main

```
root@ERL3:~# mwan3 restart
ip: invalid argument '0xfd00/0xff00' to 'fwmark'
ip: invalid argument '0xfe00/0xff00' to 'fwmark'
ip: invalid argument '0xfd00/0xff00' to 'fwmark'
ip: invalid argument '0xfe00/0xff00' to 'fwmark'
ip: invalid argument '0x100/0xff00' to 'fwmark'
ip: invalid argument '0xfd00/0xff00' to 'fwmark'
ip: invalid argument '0xfe00/0xff00' to 'fwmark'
ip: invalid argument '0xfd00/0xff00' to 'fwmark'
ip: invalid argument '0xfe00/0xff00' to 'fwmark'
ip: invalid argument '0x200/0xff00' to 'fwmark'

Troubleshooting:

OpenWrt - LEDE Reboot SNAPSHOT r5175-1dcd8e7
LuCI - git-17.298.65971-f24ae58

mwan3 - 2.6.6-1
mwan3-luci - git-17.298.65971-f24ae58-1

Output of "cat /etc/config/mwan3" :

config policy 'secure_ports'
list use_member 'wan_m1_w3'
list use_member 'wan2_m1_w2'
option last_resort 'unreachable'

config policy 'balanced'
list use_member 'wan_m1_w3'
list use_member 'wan2_m1_w3'
option last_resort 'unreachable'

config interface 'wan'
option down '3'
option initial_state 'online'
option family 'ipv4'
option track_method 'ping'
option size '56'
option flush_conntrack 'never'
option reliability '1'
option count '2'
option timeout '3'
option interval '60'
option failure_interval '10'
option recovery_interval '10'
option up '3'
option enabled '1'
list track_ip 'x.x.x.x'

config interface 'wan2'
option reliability '1'
option down '3'
option initial_state 'online'
option family 'ipv4'
option track_method 'ping'
option count '2'
option size '56'
option timeout '3'
option interval '60'
option failure_interval '10'
option recovery_interval '10'
option up '3'
option flush_conntrack 'never'
option enabled '1'
list track_ip 'x.x.x.x'

config member 'wan_m1_w3'
option interface 'wan'
option metric '1'
option weight '3'

config member 'wan_m2_w3'
option interface 'wan'
option metric '2'
option weight '3'

config member 'wan2_m1_w2'
option interface 'wan2'
option metric '1'
option weight '2'

config rule 'https'
option sticky '1'
option proto 'tcp'
option use_policy 'secure_ports'
option dest_port '443,22,993,995,465,587'

config rule 'default_rule'
option dest_ip '0.0.0.0/0'
option use_policy 'balanced'

config globals 'globals'
option local_source 'lan'
option mmx_mask '0xff00'

config member 'wan2_m1_w3'
option interface 'wan2'
option metric '1'
option weight '3'

Output of "cat /etc/config/network" :

config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config globals 'globals'
option ula_prefix 'fd99:75eb:8fa7::/48'

config interface 'lan'
option type 'bridge'
option ifname 'eth0'
option proto 'static'
option netmask '255.255.255.0'
option delegate '0'
option ipaddr '192.168.168.1'

config interface 'wan'
option ifname 'eth1'
option _orig_ifname 'eth1'
option _orig_bridge 'false'
option proto 'pppoe'
option delegate '0'
USERNAME HIDDEN
PASSWORD HIDDEN
option ipv6 '0'
option metric '5'
option peerdns '1'
option pppd_options 'debug'

config interface 'wan2'
option proto 'pppoe'
option ifname 'eth2.35'
option delegate '0'
USERNAME HIDDEN
PASSWORD HIDDEN
option ipv6 '0'
option metric '10'
option peerdns '0'
option pppd_options 'debug'

Output of "cat /etc/config/wireless" :

No data found

Output of "ifconfig" :

br-lan Link encap:Ethernet HWaddr DC:9F:DB:xx:xx:xx
inet addr:192.168.168.1 Bcast:192.168.168.255 Mask:255.255.255.0
inet6 addr: fe80::de9f:dbff:fe28:691b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17556 errors:0 dropped:0 overruns:0 frame:0
TX packets:39951 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1733854 (1.6 MiB) TX bytes:49269339 (46.9 MiB)

eth0 Link encap:Ethernet HWaddr DC:9F:DB:xx:xx:xx
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17571 errors:0 dropped:0 overruns:0 frame:0
TX packets:40017 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2052008 (1.9 MiB) TX bytes:49273655 (46.9 MiB)

eth1 Link encap:Ethernet HWaddr DC:9F:DB:xx:xx:xx
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:41160 errors:0 dropped:0 overruns:0 frame:0
TX packets:18978 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:50064877 (47.7 MiB) TX bytes:1911957 (1.8 MiB)

eth2 Link encap:Ethernet HWaddr DC:9F:DB:xx:xx:xx
inet6 addr: fe80::de9f:dbff:fe28:691d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5923 errors:0 dropped:0 overruns:0 frame:0
TX packets:4038 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:391434 (382.2 KiB) TX bytes:140225 (136.9 KiB)

eth2.35 Link encap:Ethernet HWaddr DC:9F:DB:xx:xx:xx
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4127 errors:0 dropped:0 overruns:0 frame:0
TX packets:4031 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:178512 (174.3 KiB) TX bytes:123283 (120.3 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:2773 errors:0 dropped:0 overruns:0 frame:0
TX packets:2773 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:268477 (262.1 KiB) TX bytes:268477 (262.1 KiB)

pppoe-wan Link encap:Point-to-Point Protocol
inet addr:x.x.x.x P-t-P:125.235.251.181 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1492 Metric:1
RX packets:37162 errors:0 dropped:0 overruns:0 frame:0
TX packets:14980 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:48842733 (46.5 MiB) TX bytes:1462414 (1.3 MiB)

pppoe-wan2 Link encap:Point-to-Point Protocol
inet addr:y.y.y.y P-t-P:125.235.251.181 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1492 Metric:1
RX packets:129 errors:0 dropped:0 overruns:0 frame:0
TX packets:33 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:9504 (9.2 KiB) TX bytes:2574 (2.5 KiB)

Output of "route -n" :

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 125.235.251.181 0.0.0.0 UG 5 0 0 pppoe-wan
0.0.0.0 125.235.251.181 0.0.0.0 UG 10 0 0 pppoe-wan2
125.235.251.181 0.0.0.0 255.255.255.255 UH 0 0 0 pppoe-wan2
125.235.251.181 0.0.0.0 255.255.255.255 UH 0 0 0 pppoe-wan
192.168.168.0 0.0.0.0 255.255.255.0 U 0 0 0 br-lan

Output of "ip rule show" :

0: from all lookup local
1001: from all iif pppoe-wan lookup main
1002: from all iif pppoe-wan2 lookup main
32766: from all lookup main
32767: from all lookup default

Output of "ip route list table 1-250" :

No data found

Firewall default output policy (must be ACCEPT) :

ACCEPT

Output of "iptables -L -t mangle -v -n" :

Chain PREROUTING (policy ACCEPT 16265 packets, 5908K bytes)
pkts bytes target prot opt in out source destination
16278 5909K mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT 4810 packets, 488K bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 11444 packets, 5407K bytes)
pkts bytes target prot opt in out source destination
250 15000 TCPMSS tcp -- * pppoe-wan 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 /* !fw3: wan (mtu_fix) / TCPMSS clamp to PMTU
0 0 TCPMSS tcp -- * pppoe-wan2 0.0.0.0/0 0.0.0.0/0 tcp flags:0x06/0x02 / !fw3: wan (mtu_fix) */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 4842 packets, 795K bytes)
pkts bytes target prot opt in out source destination
4853 797K mwan3_hook all -- * * 0.0.0.0/0 0.0.0.0/0

Chain POSTROUTING (policy ACCEPT 16284 packets, 6202K bytes)
pkts bytes target prot opt in out source destination

Chain mwan3_connected (2 references)
pkts bytes target prot opt in out source destination
8196 4805K MARK all -- * * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected dst MARK or 0xff00

Chain mwan3_hook (2 references)
pkts bytes target prot opt in out source destination
21131 6706K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore mask 0xff00
2075 161K mwan3_ifaces_in all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
2024 157K mwan3_connected all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
758 55656 mwan3_ifaces_out all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
232 13928 mwan3_rules all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
21131 6706K CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save mask 0xff00
12784 5581K mwan3_connected all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0xff00/0xff00

Chain mwan3_iface_in_wan (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- pppoe-wan * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected src mark match 0x0/0xff00 /* default / MARK or 0xff00
22 1337 MARK all -- pppoe-wan * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 / wan */ MARK xset 0x100/0xff00

Chain mwan3_iface_in_wan2 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- pppoe-wan2 * 0.0.0.0/0 0.0.0.0/0 match-set mwan3_connected src mark match 0x0/0xff00 /* default / MARK or 0xff00
29 2997 MARK all -- pppoe-wan2 * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 / wan2 */ MARK xset 0x200/0xff00

Chain mwan3_iface_out_wan (1 references)
pkts bytes target prot opt in out source destination
510 40428 MARK all -- * * 171.233.235.126 0.0.0.0/0 mark match 0x0/0xff00 /* wan */ MARK xset 0x100/0xff00

Chain mwan3_iface_out_wan2 (1 references)
pkts bytes target prot opt in out source destination
16 1300 MARK all -- * * 27.74.70.98 0.0.0.0/0 mark match 0x0/0xff00 /* wan2 */ MARK xset 0x200/0xff00

Chain mwan3_ifaces_in (1 references)
pkts bytes target prot opt in out source destination
2075 161K mwan3_iface_in_wan all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
2050 159K mwan3_iface_in_wan2 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00

Chain mwan3_ifaces_out (1 references)
pkts bytes target prot opt in out source destination
758 55656 mwan3_iface_out_wan all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
247 15176 mwan3_iface_out_wan2 all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00

Chain mwan3_policy_balanced (1 references)
pkts bytes target prot opt in out source destination
21 1276 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 statistic mode random probability 0.50000000000 /* wan2 3 6 / MARK xset 0x200/0xff00
25 1500 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 / wan 3 3 */ MARK xset 0x100/0xff00

Chain mwan3_policy_secure_ports (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 statistic mode random probability 0.39999999991 /* wan2 2 5 / MARK xset 0x200/0xff00
1 60 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 / wan 3 3 */ MARK xset 0x100/0xff00

Chain mwan3_rule_https (1 references)
pkts bytes target prot opt in out source destination
185 11100 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 MARK xset 0x200/0xff00
185 11100 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x200/0xff00 ! match-set mwan3_sticky_https src,src MARK and 0xffff00ff
185 11100 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 MARK xset 0x100/0xff00
1 60 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x100/0xff00 ! match-set mwan3_sticky_https src,src MARK and 0xffff00ff
1 60 mwan3_policy_secure_ports all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00
185 11100 SET all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0xfc00/0xfc00 del-set mwan3_sticky_https src,src
185 11100 SET all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0xfc00/0xfc00 add-set mwan3_sticky_https src,src

Chain mwan3_rules (1 references)
pkts bytes target prot opt in out source destination
185 11100 mwan3_rule_https tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport sports 0:65535 multiport dports 443,22,993,995,465,587 mark match 0x0/0xff00 /* https /
46 2776 mwan3_policy_balanced all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x0/0xff00 / default_rule */

[teqlee]

Work again on snapshot r5217. Tks.

[noname77]

I am on r5297 and seeing the same behaviour / error:

ip: invalid argument '0xfd00/0xff00' to 'fwmark'

were you able to determine what was causing this other than just that r5217 fixed this?

Thanks

[azuwis]

@noname77 I've found the causing of the problem, https://git.lede-project.org/?p=source.git;a=commit;h=1cec4d4ef0145b48094ec0df40c18b8fbb315cf0

# opkg whatprovides -A ip
What provides ip
    busybox
    ip-full
    ip-tiny

ip applet of busybox does not work with mwan3, installing ip-tiny or ip-full should fix the problem.

[noname77]

@azuwis thanks for the suggestion, but unfortunately i already have these installed.

# opkg whatprovides -A ip
What provides ip
    busybox
    ip-full
    ip-tiny

and still seeing the ip complaining (output below is repeated for every interface configured in mwan3)

# mwan3 restart
ip: invalid argument '0xfd00/0xff00' to 'fwmark'
ip: invalid argument '0xfe00/0xff00' to 'fwmark'
ip: invalid argument '0xfd00/0xff00' to 'fwmark'
ip: invalid argument '0xfe00/0xff00' to 'fwmark'
ip: invalid argument '0x100/0xff00' to 'fwmark'
uci: Entry not found
uci: Entry not found

I am using davidc502 builds from https://davidc502sis.dynamic-dns.net/ on Linksys WRT1900ACS with LEDE r5422, will try to seek some help there too.

kernel and version info:

# cat /proc/version 
Linux version 4.9.65 (davidc502@pc2) (gcc version 5.5.0 (LEDE GCC 5.5.0 r5422-9fe59abef8)

Thanks

[azuwis]

@noname77 Install ip-tiny or ip-full, which means:

# opkg update
# opkg install ip-tiny

opkg whatprovides -A ip will just list which packages provide the ip virtual package.

[noname77]

@azuwis thanks a lot, that fixed it.

misunderstood what opkg whatprovides does

[teqlee]

@azuwis
Thanks for pointing exactly. The problem is due to lack of ip-tiny/ip-full package.
I'll reopen this issue.

[Ricty]

I have the same problem after updating LEDE (kernel 4.9.58) on my route.
“interface wan is error and tracking is active”
And the traffic on each wan only depends on the metric, regardless of setting in mwan3.
Mwan3 can work again after ip-tiny installed. :)

[feckert]

Do we have to change the DEPENDS from ip to ip-full | ip-tiny?
On my system i always install ip-full so i have not seen this error before.
https://github.com/openwrt/packages/blob/master/net/mwan3/Makefile#L22

[teqlee]

@feckert
Yes, please.

[camelcamro]

still have the same issue on current trunk (23.1.2018)
:(
is it not possible to install all needed packages on trunk version ?

[feckert]

@camelcamro from my point off view mwan3 will install all needed packages as dependency.
Only ip-full must be installed per hand until following pull request is merged
See https://github.com/openwrt/packages/pull/5092