V2ray-core: 透明代理不能正常工作

Created on 25 May 2020 · 16Comments · Source: v2ray/v2ray-core

使用环境：
树莓派最新Lite系统，旁路由，静态IP（192.168.1.64），已经开启IPv4@IPv6转发，上网设备已设置默认网关和DNS为树莓派。

问题描述：
参考v2ray白话文中关于Tproxy配置，透明代理不能工作。上网设备为两台手机，一台能上CN网，一台完全无法联网，周末调试了两天，无法解决问题。v2ray error错误日志在最后

v2ray配置：
{
"log":{
"loglevel": "warning",
"access": "/var/log/v2ray/access.log",
"error": "/var/log/v2ray/error.log"
},
"dns": {
"servers": [
"1.1.1.1",
"8.8.8.8",
"114.114.114.114",
{
"address": "223.5.5.5",
"port": 53,
"domains": [
"geosite:cn",
"ntp.org",
"我的域名"
]
}
]
},
"inbounds": [
{
"tag":"transparent",
"port": 11080,
"protocol": "dokodemo-door",
"settings": {
"network": "tcp,udp",
"followRedirect": true
},
"sniffing": {
"enabled": true,
"destOverride": [
"http",
"tls"
]
},
"streamSettings": {
"sockopt": {
"tproxy": "tproxy" // 透明代理使用 TPROXY 方式
}
}
},
{
"port": 1080,
"protocol": "socks",
"sniffing": {
"enabled": true,
"destOverride": [
"http",
"tls"
]
},
"settings": {
"auth": "noauth"
}
}
],
"outbounds": [
{
"tag": "proxy",
"protocol": "VMess",
"settings": {
"vnext": [
{
"address": "我的域名",
"port": 443,
"users": [
{
"id": "",
"alterId": 64
}
]
}
]
},
"streamSettings": {
"network": "ws",
"security": "tls",
"wsSettings": {
"path": "/"
},
"sockopt": {
"mark": 255
}
},
"mux": {
"enabled": false
}
},
{
"tag": "block",
"protocol": "blackhole",
"settings": {
"response": {
"type": "http"
}
}
},
{
"tag": "direct",
"protocol": "freedom",
"settings": {
"domainStrategy": "UseIP"
},
"streamSettings": {
"sockopt": {
"mark": 255
}
}
},
{
"tag": "dns-out",
"protocol": "dns",
"streamSettings": {
"sockopt": {
"mark": 255
}
}
}
],
"routing": {
"domainStrategy": "IPOnDemand",
"rules": [
{ // 劫持 53 端口 UDP 流量，使用 V2Ray 的 DNS
"type": "field",
"outboundTag": "dns-out",
"inboundTag": [
"transparent"
],
"port": 53,
"network": "udp"
},
{ // 直连 123 端口 UDP 流量（NTP 协议）
"type": "field",
"outboundTag": "direct",
"inboundTag": [
"transparent"
],
"port": 123,
"network": "udp"
},
{
"type": "field",
"outboundTag": "direct",
"ip": [
"223.5.5.5",
"114.114.114.114"
]
},
{
"type": "field",
"outboundTag": "proxy", // 改为你自己代理的出站 tag
"ip": [
"8.8.8.8",
"1.1.1.1"
]
},

  {  // 广告拦截
    "type": "field",
    "outboundTag": "block",
    "domain": [
      "geosite:category-ads-all"
    ]
  },
  { // BT 流量直连
    "type": "field",
    "protocol":["bittorrent"], 
    "outboundTag": "direct"
  },
  {
    "type": "field",
    "outboundTag": "direct",
    "domain": [
      "geosite:cn"
    ]
  },
  {
    "type": "field",
    "outboundTag": "direct",
    "ip": [
      "geoip:cn",
      "geoip:private"
    ]
  }
]

}
}

iptables配置（包括 IP rule和IP router）：
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
V2RAY all -- anywhere anywhere

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
V2RAY_MASK all -- anywhere anywhere

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Chain V2RAY (1 references)
target prot opt source destination
RETURN all -- anywhere localhost
RETURN all -- anywhere base-address.mcast.net/4
RETURN all -- anywhere 255.255.255.255
RETURN tcp -- anywhere 192.168.0.0/16
RETURN udp -- anywhere 192.168.0.0/16 udp dpt:!domain
TPROXY udp -- anywhere anywhere TPROXY redirect 0.0.0.0:11080 mark 0x1/0xffffffff
TPROXY tcp -- anywhere anywhere TPROXY redirect 0.0.0.0:11080 mark 0x1/0xffffffff

Chain V2RAY_MASK (1 references)
target prot opt source destination
RETURN all -- anywhere base-address.mcast.net/4
RETURN all -- anywhere 255.255.255.255
RETURN tcp -- anywhere 192.168.0.0/16
RETURN udp -- anywhere 192.168.0.0/16 udp dpt:!domain
RETURN all -- anywhere anywhere mark match 0xff
MARK udp -- anywhere anywhere MARK set 0x1
MARK tcp -- anywhere anywhere MARK set 0x1
root@raspberrypi:~# /usr/bin/v2ray/v2ray -test -config /etc/v2ray/config.json
V2Ray 4.23.1 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.13 linux/arm)
A unified platform for anti-censorship.
2020/05/24 22:23:23 [Info] v2ray.com/core/common/platform/ctlcmd:
v2ctl> Read config: /etc/v2ray/config.json
Configuration OK.
root@raspberrypi:~# ip rule
0: from all lookup local
32765: from all fwmark 0x1 lookup 100
32766: from all lookup main
32767: from all lookup default
root@raspberrypi:~# ip route
default via 192.168.1.1 dev wlan0 src 192.168.1.64 metric 303
192.168.1.0/24 dev wlan0 proto dhcp scope link src 192.168.1.64 metric 303

v2ray error日志：
2020/05/24 22:46:09 [Warning] v2ray.com/core: V2Ray 4.23.1 started
2020/05/24 22:46:16 [Warning] v2ray.com/core/app/proxyman/outbound: failed to process outbound traffic > v2ray.com/core/proxy/vmess/outbound: failed to find an available destination > v2ray.com/core/common/retry: [v2ray.com/core/transport/internet/websocket: failed to dial WebSocket > v2ray.com/core/transport/internet/websocket: failed to dial to (wss://我的域名/ray): > dial tcp: operation was canceled] > v2ray.com/core/common/retry: all retry attempts failed
2020/05/24 22:46:19 [Warning] v2ray.com/core/app/proxyman/outbound: failed to process outbound traffic > v2ray.com/core/proxy/vmess/outbound: failed to find an available destination > v2ray.com/core/common/retry: [v2ray.com/core/transport/internet/websocket: failed to dial WebSocket > v2ray.com/core/transport/internet/websocket: failed to dial to (wss://我的域名/ray): > dial tcp: operation was canceled] > v2ray.com/core/common/retry: all retry attempts failed
2020/05/24 22:46:19 [Warning] v2ray.com/core/app/proxyman/outbound: failed to process outbound traffic > v2ray.com/core/proxy/vmess/outbound: failed to find an available destination > v2ray.com/core/common/retry: [v2ray.com/core/transport/internet/websocket: failed to dial WebSocket > v2ray.com/core/transport/internet/websocket: failed to dial to (wss://我的域名/ray): > dial tcp: operation was canceled] > v2ray.com/core/common/retry: all retry attempts failed
设备接入旁路有网关后，短时间v2ray生成大量错误日志，都是如上重复的日志。个人理解是outbound找不到出口，在旁路由内部形成了回环，但是无法定位问题，所以前来求助。

Source

jerry-ch-issue

Most helpful comment

打开 info 日志看看，是不是有SO_MARK 设置失败的问题. 如果这个标记没设置成功，就会出现环回。

如果系统iptables支持cgroup2，可以考虑改用-m cgroup --path来避免环回。如下面这条规则是在systemd环境下的配置，他的含义是来自 v2ray group进程的消息直接出去，不重新路由.

    iptables -t mangle -A OUTPUT -m cgroup --path system.slice/v2ray.service -j RETURN
    iptables -t mangle -A OUTPUT -j V2RAY_MASK

lisongmin on 25 May 2020

👍5

All 16 comments

补充一下，在PC的装的v2ray客户端能正常工作，v2ray服务器和客户端均为最新版

jerry-ch-issue on 25 May 2020

打开 info 日志看看，是不是有SO_MARK 设置失败的问题. 如果这个标记没设置成功，就会出现环回。

    iptables -t mangle -A OUTPUT -m cgroup --path system.slice/v2ray.service -j RETURN
    iptables -t mangle -A OUTPUT -j V2RAY_MASK

lisongmin on 25 May 2020

👍5

打开 info 日志看看，是不是有SO_MARK 设置失败的问题. 如果这个标记没设置成功，就会出现环回。

如果系统iptables支持cgroup2，可以考虑改用-m cgroup --path来避免环回。如下面这条规则是在systemd环境下的配置，他的含义是来自 v2ray group进程的消息直接出去，不重新路由.
    iptables -t mangle -A OUTPUT -m cgroup --path system.slice/v2ray.service -j RETURN
    iptables -t mangle -A OUTPUT -j V2RAY_MASK

看了一下内核日志，复制出来用mark搜索，后面都接的是=0x1，从来没有0xff，是不是表明没有打上直连标记？

jerry-ch-issue on 25 May 2020

这种情况更像是时间没有同步，或其他原因导致服务端拒绝？

lisongmin on 25 May 2020

这种情况更像是时间没有同步，或其他原因导致服务端拒绝？

首先非常感谢。
昨天回去又调试了一个晚上，因为是外行，所以技术原理什么的都不懂，只能摸索。

1、你上面cgroup2，昨天试了一下，我的树莓派官方最新系统（buster）应该是不支持的。

2、服务器拒绝问题，除了时间没有同步问题，我PC的客户端使用正常，是不是表明其他问题可以排除了？

3、服务器的日志我看了，搜索了一下很像时间不同步的问题（之前我做过时间同步，服务器用的纽约的时区，本地用的是北京，手动调整的，时间相差在30S以内，但是还是不行），等晚上回去再试一下吧
现在新的疑问是，透明代理环境下，一旦服务器reject连接，在开启iptables的日志情况下，会在本地短时间形成大量的日志文件吗（一度达到了近2G，如果不手动停止的话，估计还会继续下去）？包括v2ray的error日志也是不断的生成，同样文件也会积累到很大，导致我不能用winSCP打开(目前用wifi连接到局域网)。

jerry-ch-issue on 26 May 2020

buster 应该是支持的, 我用的也是这个版本，只是不是树莓派的

root@A1:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

内核默认支持cgroup的过滤

root@A1:~# cp /proc/config.gz /tmp/
root@A1:~# gunzip /tmp/config.gz
root@A1:~# grep -i xt_ /tmp/config | grep -i cgroup
CONFIG_NETFILTER_XT_MATCH_CGROUP=y

iptables版本

root@A1:~# iptables --version
iptables v1.8.2 (nf_tables)

v2ray通过systemd启动,启动后, 可以看到cgroup下有这个服务的信息

root@A1:~# ls /sys/fs/cgroup/systemd/system.slice/v2ray.service/
cgroup.clone_children  cgroup.procs  notify_on_release  tasks

另外，防火墙设置需要在v2ray启动后设置，否则cgoup不存在可能会导致规则不生效。
我的做法是增加下面文件来完成：

root@A1:~# cat /etc/systemd/system/v2ray.service.d/02_tproxy.conf
[Service]
ExecStartPost=-+/bin/bash /usr/bin/v2ray_firewall start
ExecStopPost=-+/bin/bash /usr/bin/v2ray_firewall stop
LimitNPROC=500
LimitNOFILE=1000000

v2ray_firewall是简单的创建防火墙规则的shell脚本，需要自己实现。

日志增长速度太快，这种还是可能存在环路，建议
先不要配置防火墙，看socks是否可以正常和服务端连接.

时间同步最好是开启ntp服务，让系统自动同步,这是一劳永逸的方法。参见systemd-timesyncd.service

lisongmin on 26 May 2020

我没开access log 所以不确定日志的大小问题，这部分没有经验。

lisongmin on 26 May 2020

root@raspberrypi:~# iptables -t mangle -A OUTPUT -m cgroup --path system.slice/v2ray.service -j RETURN
iptables v1.8.2 (nf_tables): Couldn't load match `cgroup':No such file or directory
是不是不支持的意思？

jerry-ch-issue on 26 May 2020

root@raspberrypi:~# iptables -t mangle -A OUTPUT -m cgroup --path system.slice/v2ray.service -j RETURN
iptables v1.8.2 (nf_tables): Couldn't load match `cgroup':No such file or directory
是不是不支持的意思？

可能是v2ray还没启动，对应的cgroup路径不存在

lisongmin on 26 May 2020

可能是v2ray还没启动，对应的cgroup路径不存在

我的iptables 版本和你的一样，

截图第一个指令，我是可以重现的，但是和iptables连用就出现之前的问题

jerry-ch-issue on 27 May 2020

之前给出的命令有问题, 应该是ls /sys/fs/cgroup/unified/system.slice/v2ray.service 因为cgourp v2的根路径是/sys/fs/cgroup/unified.
估计应该也是有的。

这种情况，可以从两个方面进行进一步的分析

通过strace命令跟踪iptable的执行过程，看报错点上下有什么特殊的线索

strace iptables -t mangle -A OUTPUT -m cgroup --path system.slice/v2ray.service -j RETURN

查看dmesg的输出，看有什么报错

另外，除了通过cgoup过滤这个方法，还可以让v2ray在专门的用户下运行，通过用户名来过滤，参考这里https://github.com/Dreamacro/clash/issues/158#issuecomment-480444965

lisongmin on 27 May 2020

root@raspberrypi:~# ls /sys/fs/cgroup/unified/system.slice/v2ray.service
cgroup.controllers cgroup.max.descendants cgroup.subtree_control cpu.stat
cgroup.events cgroup.procs cgroup.threads
cgroup.max.depth cgroup.stat cgroup.type
root@raspberrypi:~# strace iptables -t mangle -A OUTPUT -m cgroup --path system.slice/v2ray.service -j RETURN
execve("/usr/sbin/iptables", ["iptables", "-t", "mangle", "-A", "OUTPUT", "-m", "cgroup", "--path", "system.slice/v2ray.service", "-j", "RETURN"], 0xbe8ef778 /* 18 vars /) = 0
brk(NULL) = 0x162000
uname({sysname="Linux", nodename="raspberrypi", ...}) = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f12000
access("/etc/ld.so.preload", R_OK) = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=54, ...}) = 0
mmap2(NULL, 54, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0xb6f11000
close(3) = 0
readlink("/proc/self/exe", "/usr/sbin/xtables-nft-multi", 4096) = 27
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(01000254300004000"..., 512) = 512
_llseek(3, 16748, [16748], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 960) = 960
_llseek(3, 16424, [16424], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t2n3f122424"..., 45) = 45
_llseek(3, 16748, [16748], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 960) = 960
_llseek(3, 16424, [16424], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t2n3f122424"..., 45) = 45
_llseek(3, 16748, [16748], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 960) = 960
_llseek(3, 16424, [16424], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t2n3f122424"..., 45) = 45
_llseek(3, 16748, [16748], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 960) = 960
_llseek(3, 16424, [16424], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t2n3f122424"..., 45) = 45
_llseek(3, 16748, [16748], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 960) = 960
_llseek(3, 16424, [16424], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t2n3f122424"..., 45) = 45
_llseek(3, 16748, [16748], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 960) = 960
_llseek(3, 16424, [16424], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t2n3f122424"..., 45) = 45
fstat64(3, {st_mode=S_IFREG|0644, st_size=17708, ...}) = 0
mmap2(NULL, 81964, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6ecf000
mprotect(0xb6ed3000, 61440, PROT_NONE) = 0
mmap2(0xb6ee2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0xb6ee2000
close(3) = 0
munmap(0xb6f11000, 54) = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=27697, ...}) = 0
mmap2(NULL, 27697, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb6f0b000
close(3) = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libmnl.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(010000042500004000"..., 512) = 512
_llseek(3, 16736, [16736], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 16388, [16388], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 16736, [16736], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 16388, [16388], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 16736, [16736], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 16388, [16388], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 16736, [16736], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 16388, [16388], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 16736, [16736], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 16388, [16388], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
fstat64(3, {st_mode=S_IFREG|0644, st_size=17816, ...}) = 0
mmap2(NULL, 81928, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6eba000
mprotect(0xb6ebe000, 61440, PROT_NONE) = 0
mmap2(0xb6ecd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0xb6ecd000
close(3) = 0
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libnftnl.so.11", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(01000314Z00004000"..., 512) = 512
_llseek(3, 133060, [133060], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 132712, [132712], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 133060, [133060], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 132712, [132712], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 133060, [133060], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 132712, [132712], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 133060, [133060], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 132712, [132712], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 133060, [133060], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 132712, [132712], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
fstat64(3, {st_mode=S_IFREG|0644, st_size=134100, ...}) = 0
mmap2(NULL, 198252, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6e89000
mprotect(0xb6ea8000, 61440, PROT_NONE) = 0
mmap2(0xb6eb7000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e000) = 0xb6eb7000
close(3) = 0
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libnetfilter_conntrack.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(01000330<00004000"..., 512) = 512
_llseek(3, 91016, [91016], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 800) = 800
_llseek(3, 90724, [90724], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222424125"..., 47) = 47
_llseek(3, 91016, [91016], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 800) = 800
_llseek(3, 90724, [90724], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222424125"..., 47) = 47
_llseek(3, 91016, [91016], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 800) = 800
_llseek(3, 90724, [90724], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222424125"..., 47) = 47
_llseek(3, 91016, [91016], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 800) = 800
_llseek(3, 90724, [90724], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222424125"..., 47) = 47
_llseek(3, 91016, [91016], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 800) = 800
_llseek(3, 90724, [90724], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222424125"..., 47) = 47
_llseek(3, 91016, [91016], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 800) = 800
_llseek(3, 90724, [90724], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222424125"..., 47) = 47
fstat64(3, {st_mode=S_IFREG|0644, st_size=91816, ...}) = 0
mmap2(NULL, 156308, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6e62000
mprotect(0xb6e78000, 61440, PROT_NONE) = 0
mmap2(0xb6e87000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0xb6e87000
close(3) = 0
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libnfnetlink.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(0100082200004000"..., 512) = 512
_llseek(3, 20936, [20936], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 680) = 680
_llseek(3, 20672, [20672], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222424125"..., 49) = 49
_llseek(3, 20936, [20936], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 680) = 680
_llseek(3, 20672, [20672], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222424125"..., 49) = 49
_llseek(3, 20936, [20936], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 680) = 680
_llseek(3, 20672, [20672], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222424125"..., 49) = 49
_llseek(3, 20936, [20936], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 680) = 680
_llseek(3, 20672, [20672], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222424125"..., 49) = 49
_llseek(3, 20936, [20936], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 680) = 680
_llseek(3, 20672, [20672], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222424125"..., 49) = 49
fstat64(3, {st_mode=S_IFREG|0644, st_size=21616, ...}) = 0
mmap2(NULL, 53440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6e54000
mprotect(0xb6e59000, 28672, PROT_NONE) = 0
mmap2(0xb6e60000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0xb6e60000
close(3) = 0
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libxtables.so.12", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(01000370$00004000"..., 512) = 512
_llseek(3, 45408, [45408], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 45064, [45064], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 45408, [45408], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 45064, [45064], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 45408, [45408], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 45064, [45064], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 45408, [45408], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 45064, [45064], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 45408, [45408], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 45064, [45064], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(3, 45408, [45408], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1040) = 1040
_llseek(3, 45064, [45064], SEEK_SET) = 0
read(3, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
fstat64(3, {st_mode=S_IFREG|0644, st_size=46448, ...}) = 0
mmap2(NULL, 121636, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6e36000
mprotect(0xb6e41000, 61440, PROT_NONE) = 0
mmap2(0xb6e50000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0xb6e50000
mmap2(0xb6e52000, 6948, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6e52000
close(3) = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libm.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(01000240r00004000"..., 512) = 512
_llseek(3, 463312, [463312], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 462960, [462960], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222423124"..., 49) = 49
_llseek(3, 463312, [463312], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 462960, [462960], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222423124"..., 49) = 49
_llseek(3, 463312, [463312], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 462960, [462960], SEEK_SET) = 0
read(3, "A0000aeabi01&0000056066101t1n222423124"..., 49) = 49
fstat64(3, {st_mode=S_IFREG|0644, st_size=464392, ...}) = 0
mmap2(NULL, 528504, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6db4000
mprotect(0xb6e25000, 61440, PROT_NONE) = 0
mmap2(0xb6e34000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x70000) = 0xb6e34000
close(3) = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libc.so.6", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(01000300x10004000"..., 512) = 512
_llseek(3, 1293204, [1293204], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 2800) = 2800
_llseek(3, 1289788, [1289788], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
_llseek(3, 1293204, [1293204], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 2800) = 2800
_llseek(3, 1289788, [1289788], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
_llseek(3, 1293204, [1293204], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 2800) = 2800
_llseek(3, 1289788, [1289788], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
_llseek(3, 1293204, [1293204], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 2800) = 2800
_llseek(3, 1289788, [1289788], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
_llseek(3, 1293204, [1293204], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 2800) = 2800
_llseek(3, 1289788, [1289788], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
_llseek(3, 1293204, [1293204], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 2800) = 2800
_llseek(3, 1289788, [1289788], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
fstat64(3, {st_mode=S_IFREG|0755, st_size=1296004, ...}) = 0
mmap2(NULL, 1364764, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6c66000
mprotect(0xb6d9e000, 65536, PROT_NONE) = 0
mmap2(0xb6dae000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x138000) = 0xb6dae000
mmap2(0xb6db1000, 8988, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb6db1000
close(3) = 0
openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libdl.so.2", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
read(3, "177ELF11100000000030(01000 n00004000"..., 512) = 512
_llseek(3, 8688, [8688], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 8340, [8340], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
_llseek(3, 8688, [8688], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 8340, [8340], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
_llseek(3, 8688, [8688], SEEK_SET) = 0
read(3, "00000000000000000000000000000000"..., 1080) = 1080
_llseek(3, 8340, [8340], SEEK_SET) = 0
read(3, "A.000aeabi01$0000056066101t1n222423124"..., 47) = 47
fstat64(3, {st_mode=S_IFREG|0644, st_size=9768, ...}) = 0
mmap2(NULL, 73924, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb6c53000
mprotect(0xb6c55000, 61440, PROT_NONE) = 0
mmap2(0xb6c64000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0xb6c64000
close(3) = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6f09000
set_tls(0xb6f09a90) = 0
mprotect(0xb6dae000, 8192, PROT_READ) = 0
mprotect(0xb6c64000, 4096, PROT_READ) = 0
mprotect(0xb6e34000, 4096, PROT_READ) = 0
mprotect(0xb6e50000, 4096, PROT_READ) = 0
mprotect(0xb6e60000, 4096, PROT_READ) = 0
mprotect(0xb6ecd000, 4096, PROT_READ) = 0
mprotect(0xb6e87000, 4096, PROT_READ) = 0
mprotect(0xb6eb7000, 8192, PROT_READ) = 0
mprotect(0xb6ee2000, 4096, PROT_READ) = 0
mprotect(0x47000, 4096, PROT_READ) = 0
mprotect(0xb6f14000, 4096, PROT_READ) = 0
munmap(0xb6f0b000, 27697) = 0
brk(NULL) = 0x162000
brk(0x183000) = 0x183000
socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 3
bind(3, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, nl_pid=996, nl_groups=00000000}, [12]) = 0
stat64("/usr/lib/arm-linux-gnueabihf/xtables/libipt_cgroup.so", 0xbe9110f8) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/arm-linux-gnueabihf/xtables/libxt_cgroup.so", {st_mode=S_IFREG|0644, st_size=5732, ...}) = 0
openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/xtables/libxt_cgroup.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 4
read(4, "177ELF11100000000030(01000,500004000"..., 512) = 512
_llseek(4, 4732, [4732], SEEK_SET) = 0
read(4, "00000000000000000000000000000000"..., 1000) = 1000
_llseek(4, 4400, [4400], SEEK_SET) = 0
read(4, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(4, 4732, [4732], SEEK_SET) = 0
read(4, "00000000000000000000000000000000"..., 1000) = 1000
_llseek(4, 4400, [4400], SEEK_SET) = 0
read(4, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(4, 4732, [4732], SEEK_SET) = 0
read(4, "00000000000000000000000000000000"..., 1000) = 1000
_llseek(4, 4400, [4400], SEEK_SET) = 0
read(4, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(4, 4732, [4732], SEEK_SET) = 0
read(4, "00000000000000000000000000000000"..., 1000) = 1000
_llseek(4, 4400, [4400], SEEK_SET) = 0
read(4, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
_llseek(4, 4732, [4732], SEEK_SET) = 0
read(4, "00000000000000000000000000000000"..., 1000) = 1000
_llseek(4, 4400, [4400], SEEK_SET) = 0
read(4, "A,000aeabi01"0000056066101t1n222424125"..., 45) = 45
fstat64(4, {st_mode=S_IFREG|0644, st_size=5732, ...}) = 0
mmap2(NULL, 69940, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0xb6c41000
mprotect(0xb6c42000, 61440, PROT_NONE) = 0
mmap2(0xb6c51000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0) = 0xb6c51000
close(4) = 0
mprotect(0xb6c51000, 4096, PROT_READ) = 0
gettimeofday({tv_sec=1590585661, tv_usec=646382}, NULL) = 0
socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 4
bind(4, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(4, {sa_family=AF_NETLINK, nl_pid=-62603628, nl_groups=00000000}, [12]) = 0
sendto(4, {{len=48, type=0xb00 / NLMSG_??? /, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1590585661, pid=0}, "x02x00x00x00x0bx00x01x00x63x67x72x6fx75x70x00x00 x08x00x02x00x00x00x00x00x08x00x03x00x00x00x00x00"}, 48, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 48
recvmsg(4, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=68, type=NLMSG_ERROR, flags=0, seq=1590585661, pid=4 232363668}, {error=-ENOENT, msg={{len=48, type=0xb00 / NLMSG_??? /, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1590585661, pid=0}, "x02x00x00x00x0bx00x01x00x63x67x 72x6fx75x70x00x00x08x00x02x00x00x00x00x00x08x00x03x00x00x00x00x00"}}}, iov_len=16536}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 68
close(4) = 0
gettimeofday({tv_sec=1590585661, tv_usec=662172}, NULL) = 0
socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 4
bind(4, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(4, {sa_family=AF_NETLINK, nl_pid=-372981224, nl_groups=00000000}, [12]) = 0
sendto(4, {{len=48, type=0xb00 / NLMSG_??? /, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1590585661, pid=0}, "x02x00x00x00x0bx00x01x00x63x67x72x6fx75x70x00x5e x08x00x02x00x00x00x00x01x08x00x03x00x00x00x00x00"}, 48, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 48
recvmsg(4, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=68, type=NLMSG_ERROR, flags=0, seq=1590585661, pid=3 921986072}, {error=-ENOENT, msg={{len=48, type=0xb00 / NLMSG_??? /, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1590585661, pid=0}, "x02x00x00x00x0bx00x01x00x63x67x 72x6fx75x70x00x5ex08x00x02x00x00x00x00x01x08x00x03x00x00x00x00x00"}}}, iov_len=16536}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 68
close(4) = 0
gettimeofday({tv_sec=1590585661, tv_usec=678020}, NULL) = 0
socket(AF_NETLINK, SOCK_RAW, NETLINK_NETFILTER) = 4
bind(4, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 0
getsockname(4, {sa_family=AF_NETLINK, nl_pid=-408350909, nl_groups=00000000}, [12]) = 0
sendto(4, {{len=48, type=0xb00 / NLMSG_??? /, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1590585661, pid=0}, "x02x00x00x00x0bx00x01x00x63x67x72x6fx75x70x00x5e x08x00x02x00x00x00x00x02x08x00x03x00x00x00x00x00"}, 48, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 48
recvmsg(4, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=68, type=NLMSG_ERROR, flags=0, seq=1590585661, pid=3 886616387}, {error=-ENOENT, msg={{len=48, type=0xb00 / NLMSG_??? */, flags=NLM_F_REQUEST|NLM_F_ACK, seq=1590585661, pid=0}, "x02x00x00x00x0bx00x01x00x63x67x 72x6fx75x70x00x5ex08x00x02x00x00x00x00x02x08x00x03x00x00x00x00x00"}}}, iov_len=16536}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 68
close(4) = 0
write(2, "iptables v1.8.2 (nf_tables): ", 29iptables v1.8.2 (nf_tables): ) = 29
write(2, "Couldn't load match cgroup':No "..., 55Couldn't load matchcgroup':No such file or directory
) = 55
write(2, "n", 1
) = 1
write(2, "Try iptables -h' or 'iptables -"..., 61Tryiptables -h' or 'iptables --help' for more information.

jerry-ch-issue on 27 May 2020

从报错点看，应该是内核返回的报错，核对一下xt_cgroup配置是y还是m，如果是m，需要手动加载一下。

如果是y，那可能是内核的问题，或systemd的问题？查看系统是否更新到了最新版本. 这方面我也没有什么思路了。

我这边用的是5.4.32内核，241版本systemd没有问题。

root@A1:~# uname -a
Linux A1 5.4.32-rockchip64 #20.02.11 SMP PREEMPT Tue Apr 14 17:30:19 CEST 2020 aarch64 GNU/Linux
root@A1:~# systemctl --version
systemd 241 (241)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

lisongmin on 27 May 2020

从报错点看，应该是内核返回的报错，核对一下xt_cgroup配置是y还是m，如果是m，需要手动加载一下。

如果是y，那可能是内核的问题，或systemd的问题？查看系统是否更新到了最新版本. 这方面我也没有什么思路了。

我这边用的是5.4.32内核，241版本systemd没有问题。
root@A1:~# uname -a
Linux A1 5.4.32-rockchip64 #20.02.11 SMP PREEMPT Tue Apr 14 17:30:19 CEST 2020 aarch64 GNU/Linux
root@A1:~# systemctl --version
systemd 241 (241)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

jerry-ch-issue on 27 May 2020

应该是解决问题了，我的v2ray用的是在线脚本安装的，最新的V4.23.1的官方脚本的 v2ray.service 配置有问题，不能做so_mark标记，按照前楼的方法，已经解决问题。现在是墙过不去，应该是我的v2ray配置有问题，阿里的DNS服务器不能解析我的域名，回去之后改一下试试看是不是完全ok了，当然这是另外一个问题了

jerry-ch-issue on 28 May 2020

应该是解决问题了，我的v2ray用的是在线脚本安装的，最新的V4.23.1的官方脚本的 v2ray.service 配置有问题，不能做so_mark标记，按照前楼的方法，已经解决问题。现在是墙过不去，应该是我的v2ray配置有问题，阿里的DNS服务器不能解析我的域名，回去之后改一下试试看是不是完全ok了，当然这是另外一个问题了

之前疏忽没测试透明代理相关导致的问题，抱歉