Core: "reply-to" feature (enabled by default) breaks RFC1122 section 3.3.1.1 (local/remote decision)

[ctminime]

Connections from WAN network IP to WAN interface IP don't work when an allow rule is created and the reply-to option is not disabled.

In my scenario, I was testing OPNsense in a lab environment. After a lot of beating my head against the wall and forum searching and then back and forth in the forum, I discovered that this behavior actually breaks RFC1122 section 3.3.1.1. It has been mentioned that this is potentially an upstream bug, but there has been no evidence given that it is an upstream bug. This topic was discussed on the forum here:
https://forum.opnsense.org/index.php?topic=15900.0

In my my test case (as shown in Wireshark screenshots) other computer has IP 192.168.169.200 and OPNsense WAN interface has IP 192.168.169.191.

To Reproduce
Steps to reproduce the behavior:

1. Fresh install of OPNsense
2. WAN network has other IPs that will access the WAN interface (ie. SSH)
3. In my test setup WAN interface receives a private IP via DHCP
4. If WAN has private IP, allow Private IPs and Bogons
5. Gateway is stateful firewall (ie. OPNsense)
6. Create rule allowing something (ie, SSH,) from WAN network to WAN interface.
7. For my test... Enable SSH access.
8. Run tcpdump on WAN interface collecting for port 22 and write to file.
9. From another computer on WAN network SSH to WAN interface. This will time out.
10. In packet capture, you should see a SYN from other computer to WAN interface
11. The SYN ACK that the WAN interface responds with, sends to the gateway MAC address instead of the MAC address of the other computer.

NOTE: disabling reply-to on the firewall rule allows a computer on the WAN network to connect. However, since this behavior breaks RFC, it is my opinion that this needs to be addressed despite a checkbox work-around.

Expected behavior
Even with reply-to enabled (which is default), the test scenario should be functional instead of timing out.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
The version below was the only one tested.

Environment
Software version used and hardware type if relevant.
e.g.:

OPNsense 20.1.1-amd64
FreeBSD 11.2-RELEASE-p16-HBSD
OpenSSL 1.1.1d 10 Sep 2019

[mimugmail]

The problem is not that reply-to breaks RFC, the problem is that reply-to is default on.

But turning it off (checkbox enabled) will break more setups than it will help people fighting with this problem. If counted right you were the third guy asking about it and the support time this will produce when changing a default may not worth it, or better invested somewhere else (like API enabled firewall rules way more ppl ask for).

Just my personal opinion.

[pv2b]

Thanks for writing up a bug report for this. I'm the original poster of the forum thread, and I completely agree the way OPNsense does this is bogus. The fact that it's been bogus for a very very long time (way pre-fork from pfSense) doesn't change that. :-)

Just my two cents:

This is not an upstream bug. FreeBSD is doing exactly what the pf rule says. The rule says to send the reply traffic to the gateway, and that's exactly what's happening. You might argue that reply-to is a misfeature, and that there should be some handy flag to exclude replies to the same subnet, but it's certainly not a bug.

If this is a behaviour that should change, that change should happen in what pf rules are generated by OPNsense, not in how pf itself acts. For example, by generating one rule with reply-to for traffic coming from outside the WAN subnet, and one without reply-to set for traffic coming from inside the WAN subnet.

Fortunately, if you want this to happen, it's already possible to do this yourself in the OPNsense UI. It'll require duplicating all rules on the WAN, so it's super ugly, but it's possible. Which is useful if you actually want to do multi-WAN and use reply-to. IMO OPNsense should just do this behind the scenes. It won't break any reasonable setup.

[fichtner]

Misfeature, maybe, maybe not as the use case has been established many years ago.

To make this discussion somewhat useful, I would like to make an actual suggestion:

If we only have at most one gateway-capable interface (that includes user-created gateways) we disable reply-to as we don’t have a multi-wan setup anyway.

I have no idea if this is feasible, but it’s certainly more useful than arguing do and don’t.

On 27. Feb 2020, at 13:04, pv2b [e-mail] wrote:


Just my two cents:

This is not an upstream bug. FreeBSD is doing exactly that the pf rule says. The rule says to send the reply traffic to the gateway, and that's exactly what's happening. You might argue that reply-to is a misfeature, but it's certainly not a bug.

If this is a behaviour that should change, that change should happen in what pf rules are generated by OPNsense, not in how pf itself acts.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

[mimugmail]

If it doesn't touch existing installs and only new ones, why not. Maybe need some small documentation updates.

I think the most clean way would be to negate the WAN net described here, but I have no idea if this is possible with pf:

https://forum.opnsense.org/index.php?topic=15900.msg73203#msg73203

[pv2b]

I think it's good to talk about solutions, @fitchner. My suggestion is to change OPNsense rule generator to apply reply-to for packets only coming from different subnets. Something like this.

Before change:

pass in quick on igb1 reply-to { igb1 192.0.2.1 } inet proto tcp from {any} to {{self}} port {443} keep state

After change:

pass in quick on igb1 inet proto tcp from {{igb1:network}} to {{self}} port {443} keep state
pass in quick on igb1 reply-to { igb1 192.0.2.1 } inet proto tcp from {any} to {{self}} port {443} keep state

This would not break any reasonable setups, while still doing what the feature is intended to do in the first place.

If you're still afraid of breaking existing setups, make it an optional checkbox only checked by default in new installs, not on existing ones.

Combine this with the idea of only enabling this feature once you actually have multi-wan and you won't double the number of firewall rules in the pf ruleset for most users.

[AdSchellevis]

Can we please stop with this? We're not planning to expand the problem by generating more rules auto-magically than what it already does (error-prone and has other visibility issues) and we're not changing the defaults now, maybe in time, you never know, but certainly not now.

How it works is documented behaviour on our end, as stated multiple times, we would have done it differently if we would have designed it from the start, but looking at our support desk, the number of questions flowing in regarding this is very low and would certainly be way more when changing the current behaviour.

[ctminime]

I was able to confirm this behavior does indeed exist in FreeBSD 11.3. So, I opened a bug report there as well.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244514

I will say that I totally understand that there are only a handful of use cases that will be affected by this. So, when determining where to invest DEV time, this probably falls to the bottom of the list. And now that I know about this issue and its work around, I am OK with with that. Especially since I am also a person who would love to see API enabled firewall rules.

During this whole thing I have had 2 major issues with this:

1. Just because this is the way it has worked for a decade+, doesn't mean it is right or should stay that way. Especially since that RFC was written in 1989.
2. I understand that Multi-WAN is an advanced configuration and since "reply-to" is meant for those scenarios, that is where the option is (in the advanced section in a firewall rule) and why the documentation for reply-to is in the Policy Based Routing section of the documentation. However, I was NOT doing any advanced setup NOR does routing come into play for a connection between 2 IPs on the subnet (or shouldn't). And I was extremely frustrated trying to figure out what the hell was going on. Especially since networking is what I do for a living.

So @AdSchellevis, when I have the time, I will do a PR to update the help dialog for the reply-to option. I will also see if I can come up with something for the OPNsense documentation as well. So that way, I am making "useful additions" instead of just complaining.

[ctminime]

The bug report at freebsd got closed almost immediately saying "works as intended" I have re-opened it requesting that it be left open for at least a week for commenters.

@pv2b @fichtner @AdSchellevis @mimugmail , it would be great if each of you could comment either for or against.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=244514

[AdSchellevis]

I'm most definitely not going to comment on a this report, for reasons explained earlier.

Let's go back one step, if you (all) check "Disable force gateway" in Firewall --> Settings --> Advanced, is the behaviour as you would expect it to be? (Which has been my assumption)

[AdSchellevis]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.