Core: NAT before IPsec: limitations and workarounds

[fraenki]

As described in #440 there are two limitations in our new implementation:

The first one is related to "My identifier":

root@opnsense:~ # /sbin/setkey -f /tmp/setkeyAsL4tW
libipsec: invalid IP address while parsing "host.example.com"
line 1: hostname nor servname provided, or not known at [ out ipsec esp/tunnel/host.example.com-1.2.3.4/require]
parse failed, line 1.

This means that it's required to use an IP address for "My identifier". Anything else, for example "Distinguished name", will not work. (Well, maybe it might work to get the IP address from the selected interface for those cases... but I'm not sure.)

---

Besides that I've found another case where "NAT before IPsec" is not yet working: when OPNsense is behind NAT. In this case there's a mismatch (as can be seen in the setkey output):

# working; created by strongSwan
  esp/tunnel/<localRouterIP>-<remoteIP>/require

# mismatch; created for "NAT before IPsec"
  esp/tunnel/10.0.0.254-<remoteIP>/unique:1

Apparently, the Router IP that is used for this tunnel is not configured on the OPNsense firewall. But instead there's a port forward on the Router.

---

As opposed to my original submission, I think it might be possible to workaround both limitations. The reason why it's not working is the same in both cases: the local entry is erroneous in the setkey output. We should add a new option to the GUI to allow the user to overwrite this value. I think in both cases the user should be able to enter the right value, it's hard to "guess" it because there are many corner cases. Additionally a bit of background information as to when this is required should be added to the help text.

[fraenki]

I plan to work on this (later this year), please assign the issue to me.

[fichtner]

(can always move milestone up if you want)

[ruffy91]

We are affected by both this issue and #2173
Our constellation is: two Site-2-Site Tunnels, on the second one the remote side is behind NAT for phase 1 (ESP-UDP) and we need to use BiNAT because we have colliding subnets.
@fraenki are you still planning to work on this soon?

We manually added the correct spdadd entries via setkey which works.

How can I add spdadd entries so they survive a reboot/upgrade?

[fraenki]

@ruffy91 I still plan to work on this, but I had to postpone it to July/August (it will not be ready for OPNsense 18.7).

[banym]

Hi, I had similar requirement for some BINAT requirements.

I run into the problem that autogenerated spd entries are not correctly generated when the Remote ID in phase 1 is not the external remote IP. For reasons the other side is not able to change their used ID to the external IP and so I am stuck to a workaround.

Manual set SPD entries fixed it for me temporarily but I would like to manage this entries persistent over api or gui that this configurations are backed up and documented.

Regards,
Dominik

[bluesource]

@fraenki was there a change with OpnSense 18.7.10 ?
spdadd workaround isn't working anymore :(

regards,
Michael

[AdSchellevis]

@bluesource https://github.com/opnsense/core/commit/8f45e0d8623a6da395e12be5b51b3ab74e490f77 should fix your issue.

[bluesource]

just for info: its working again after a call with deciso (business support).

The Issue was: Remote FW is only accepting a Cert as peer identifier and therefore we set "KeyID" with value "%any, although remote FW has a fix IP. Then the Manual SPD entries are not set.

Fix from deciso:
https://github.com/opnsense/core/commit/8f45e0d8623a6da395e12be5b51b3ab74e490f77

and btw: great support from Deciso!

[jroehler23]

Hi, is this still open to fix in 19.1.x? I had to change my WAN configuration in several remote sites from PPPoE to a static IP behind a NAT router. The normal IPsec tunnels are all working normal, but the additional networks of the remote sides, which must be NATed before IPsec are not entering the tunnel any more.

Regards,
Jochen

[AdSchellevis]

we don't have other fixed queued than https://github.com/opnsense/core/commit/8f45e0d8623a6da395e12be5b51b3ab74e490f77 at the moment.

[fichtner]

8f45e0d8623a was shipped as fe38ca7acc3fc in 19.1.3

[jroehler23]

OK. Can you help me to find out where or why the packets are not finding the way into the tunnel? I really need this feature back working!

When I remove the manual SPD for the additional network in phase 2, the packets were send out over the standard gateway via WAN. So IPsec configuration is recognizing the additional network. But I can't see any SNAT transformation in the firewall log of the packets. The configuration for this is unchanged and made in the 1to1 NAT of the firewall section.

[jroehler23]

To make it short: I've found my mistake, NAT before IPsec is working when OPNsense is behind a NAT router.

Before I had to change the WAN configuration, I used 'my IP address' and 'peer IP address' in the phase 1 as identifiers at both ends. After I moved one router behind a NAT router, I had to use 'IP address' on one side for one identifier. I chose 'my identifier' in the OPNsense behind the NAT router and put the public WAN IP address of the NAT router in. Everything worked fine, except the additional networks behind this OPNsense.
So the router behind the NAT must have 'my IP address' set as 'my identifier' otherwise the NATed packets will not find the way into the IPsec tunnel! The change in the identifiers must be done at the other end of the tunnel: the 'peer identifier' must be set to the private WAN IP of the OPNsense behind the NAT router. Now this is obvious to me...

Maybe it is possible to put this hint into the IPsec BINAT wiki.

[mimugmail]

https://forum.opnsense.org/index.php?topic=12849.msg59564;topicseen#msg59564

[olivierIllogika]

Hi, we ran into this issue with the router behind a NAT. The phase 1 identifier is set to 'IP address' (with public IP in value field). The manual SPD entry was incorrectly using this IP address as the local tunnel endpoint instead of the opnsense IP from which the tunnel is initiated.

My fix was to modify /usr/local/etc/inc/plugins.inc.d/ipsec.inc with the following:

    // add manual added spd entries
    if (!empty($config['ipsec']['phase1']) && !empty($config['ipsec']['phase2'])) {
        foreach ($config['ipsec']['phase1'] as $ph1ent) {
            foreach ($config['ipsec']['phase2'] as $ph2ent) {
                if (!isset($ph2ent['disabled']) && $ph1ent['ikeid'] == $ph2ent['ikeid'] && !empty($ph2ent['spd'])) {
-                    $myid_data = ipsec_find_id($ph1ent, "local");
+                    $myid_data = ipsec_get_phase1_src($ph1ent);
                    $peerid_spec = ipsec_find_id($ph1ent, "peer");
                    if (!is_ipaddr($peerid_spec)) {
                        if (is_ipaddr($ph1ent['remote-gateway'])) {
                            $peerid_spec = $ph1ent['remote-gateway'];
                        } else {
                            log_error(sprintf(
                                "spdadd: unable to match remote network on %s or %s [skipped]",
                                $peerid_spec,
                                $ph1ent['remote-gateway']
                            ));
                            continue;
                        }
                    }
                    foreach (explode(',', $ph2ent['spd']) as $local_net) {

This is probably incorrect for a general solution, but it fixed my issue.

[AdSchellevis]

@olivierIllogika good to hear this solved your issue, I'm always struggling a bit with the setkey (https://www.freebsd.org/cgi/man.cgi?query=setkey) documentation.

At a first glance it looks like we missed the correct policy source definition here, which resulted in a working setup when using the external address.

Looking at the code, it looks like it will only work at the moment if ipsec_find_id($ph1ent, "local") and ipsec_get_phase1_src($ph1ent) outcomes are equal, now we need to validate if that is indeed the case.

I've prepared a fix https://github.com/opnsense/core/commit/30a55363aeceff26b32cc1b6fa1d19cacb15490f, which incorporates your change. now we need to find some people to validate this on their functional setups.

@fraenki @mimugmail do you have the time and spirit to test this patch? Thanks in advance!

[olivierIllogika]

@AdSchellevis fair warning: I'm not at all familiar with the code or setkey. The setkey documentation could indeed be a little more specific, but as I understand it, the "tunnel/%s-%s/require" string needs to match the tunnel's endpoint:

[...] If mode is tunnel you must specify the end-point addresses of the SA as src and dst with `-' between these addresses, which is used to specify the SA to use. [...]

I'm not sure about the 'SA' part, but this (linux) documentation mentions the tunnel explicitly: https://www.mad-hacking.net/documentation/linux/networking/ipsec/spd-setkey.xml

If this assumption is correct, it might make sense to (1) remove the $peerid_spec is_ipaddr() check and just get the phase 1 destination from the start; and (2) rename the variables from $myid_data/$peerid_spec to $tunnel_src/$tunnel_dst

[AdSchellevis]

@olivierIllogika I think you are right, we could do a bit more here to clarify things. let me refactor this a bit further and add some logging in case the end result varies from what it does on current stable.

[AdSchellevis]

@olivierIllogika this should be it https://github.com/opnsense/core/commit/1e8ebbfc0d54e85e48df06c0927dfa5343ca3a51 , it needs https://github.com/opnsense/core/commit/30a55363aeceff26b32cc1b6fa1d19cacb15490f first as a starting point.

My setup still works with the patch applied, hopefully this removes the limitations, it will simplify the code as well (after removal of the extra temporary logging).

[AdSchellevis]

@fraenki I think we can close this issue, most limitations should be solved with https://github.com/opnsense/core/commit/755849d2f47a1df6f2a084c4588d5f39dee43ef0, what do you think?

[Jesper-fr]

Hi,
is this NAT before IPSEC tunnel still an issue. I have just upgraded my Opnsense to ver. 19.7.2, but it still doens't work. Is Strongswan patch still required to get it to work ? Strongswan ver. 5.8.0

[AdSchellevis]

@Jesper-fr works like a charm on my end, we don't need patches to Strongswan. Just don't forget to add manual spd entries (some documentation can be found here https://docs.opnsense.org/manual/how-tos/ipsec-s2s-binat.html)

[Jesper-fr]

My problem is that I only have access to one end of the tunnel. The opposite end is at a customer site.

[AdSchellevis]

IPsec can be tricky to debug, certainly if you can't access both sides.

[Jesper-fr]

I can't see that the NAT address is being sent through the tunnel. Encode counters are not increasing when I do a ping from a server in my end of the tunnel towards the customer server.

[mimugmail]

Screenshots and logs please, best via Forum threas

[Jesper-fr]

I litle more explanation
It's an existing tunnel, My end 10.222.0.0/16 remote end 10.1.204.0/24
I have a server in my end coming from 10.220.2.72 which I translate to 10.222.8.4. If I ping from the local server and do a packet capture on WAN interface , then I can see the translated addr. towards the server I try to reach (10.1.204.108).

firewall interfaces:
VLAN 10 (10.220.0.9/22
VLAN 32 (10.222.8.4/22)
WAN

My local server is comming via VLAN 10 and is being translated to interface addr. on VLAN 32.

I will provide screenshots tomorrow.

[Jesper-fr]

please find attached a packet capture . Ping done from a server 10.220.2.13 and it's being translated to 10.222.8.4, but the packet towards 10.1.204.108 is not being tunneled.

I have tried to do 2 configs. Both configs are outbound nat

1) WAN as outgoing interface (packet capture WAN)
2) IPSEC as outgoing interface (packet capture IPSEC)

PACKET CAPTURE WAN
VLAN10
em0 09:29:01.766196 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32194, offset 0, flags [none], proto ICMP (1), length 60)
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43537, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43538, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43539, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43540, length 40
VLAN10
em0 09:29:06.475942 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32195, offset 0, flags [none], proto ICMP (1), length 60)
VLAN10
em0 09:29:11.465461 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32196, offset 0, flags [none], proto ICMP (1), length 60)
VLAN10
em0 09:29:16.475901 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32197, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1 09:29:01.766572 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32194, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1
10.222.8.4 > 10.1.204.108: ICMP echo request, id 8203, seq 43537, length 40
10.222.8.4 > 10.1.204.108: ICMP echo request, id 8203, seq 43538, length 40
10.222.8.4 > 10.1.204.108: ICMP echo request, id 8203, seq 43539, length 40
10.222.8.4 > 10.1.204.108: ICMP echo request, id 8203, seq 43540, length 40
WAN
em1 09:29:06.476327 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32195, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1 09:29:11.465774 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32196, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1 09:29:16.476087 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32197, offset 0, flags [none], proto ICMP (1), length 60)

---

PACKET CAPTURE IPSEC
VLAN10
em0 09:36:51.695292 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32198, offset 0, flags [none], proto ICMP (1), length 60)
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43547, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43548, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43549, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43550, length 40
VLAN10
em0 09:36:56.466359 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32199, offset 0, flags [none], proto ICMP (1), length 60)
VLAN10
em0 09:37:01.483428 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32200, offset 0, flags [none], proto ICMP (1), length 60)
VLAN10
em0 09:37:06.466768 00:26:0a:27:d6:00 > 00:50:56:a8:de:54, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 32201, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1 09:36:51.695470 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32198, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43547, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43548, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43549, length 40
10.220.2.13 > 10.1.204.108: ICMP echo request, id 1, seq 43550, length 40
WAN
em1 09:36:56.466440 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32199, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1 09:37:01.483573 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32200, offset 0, flags [none], proto ICMP (1), length 60)
WAN
em1 09:37:06.466915 00:50:56:a8:47:5c > 00:26:0a:27:d6:00, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 32201, offset 0, flags [none], proto ICMP (1), length 60)

in below image, test is 10.220.2.13 - Salling group is 10.1.204.108 - VLAN32_NAT_INTERFACE is 10.222.8.4

[AdSchellevis]

@Jesper-fr can you please use our forum for discussions (as Michael requested)? .

[Jesper-fr]

Sorry....i'm new to GitHub. Were do I find forum ?

[AdSchellevis]

no problem, things happen, our forum can be found here : https://forum.opnsense.org/

[Jesper-fr]

Sorry, but I have tried to post my issue on that forum, but I didn’t get
any response what so ever

[AdSchellevis]

please just add enough detail there as @mimugmail requested. (free) community support is provided by people helping out in their own time. If you have an actual reproducible bug or a feature request, you can always open a new ticket of your own.

Spamming threads in existing issues won't help solving the original ticket. (this case should probably already be closed, since the specific work-arounds where fixed a month ago. @fraenki close now?)

[AdSchellevis]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.