Core: 6rd support upgrade original patch to freebsd 11

[cobradevil]

Goodmorning,

I still would like to have proper 6rd support but lack the skills to do this myself.
https://forum.opnsense.org/index.php?topic=4566.msg17514#msg17514

And the original patch:
https://github.com/pfsense/FreeBSD-src/commit/62498dd06a33a82a579d1cc113c2fa04d995ac91

https://redmine.pfsense.org/issues/7272

I can give access to my setup if needed.
My provider is telfort (KPN) in the Netherlands which has support for 6rd.

Another option could be to support it through a gif interface like this post:
https://blog.feld.me/posts/2015/02/ipv6-via-6rd-on-freebsd/
The only problem then would be not to have access to hosts on the same 6rd implementation.

With best regards,
William van de Velde

[cobradevil]

Hello opnsense team.

It looks like the issue has been resolved within pfsense.
Could there be an effort to get this code back into opnsense?

Best regards,
William

[fichtner]

Hi @cobradevil,

pfSense did not release a single patch for their upcoming 2.4 version since July 24 despite their claims that they are open source and actively working on it:

https://github.com/pfsense/FreeBSD-src/tree/RELENG_2_4

Until this changes or hits FreeBSD, there is nothing we will do, because the patch they claim they have is not available publicly.

pfSense did not respond to my inquiries to why they do not open source their changes anymore.

Thus, I am closing this until further notice.

Cheers,
Franco

[cobradevil]

Hi @fichtner
Ok did not notice their commit change policy .. weird company.
Thanks for the info.

Best regards,
William

[fichtner]

If something changes, we are glad to revive this. But I have no willingness to go through the drama that happened in 2014 one more time in a slightly different wrapping. If they want to mess with their community and ultimately the FreeBSD community at large they are certainly free to try. Last time it created OPNsense, something they did not expect. ;)

[paride]

Any news on this? Has pfSense finally released their 6rd patches?

[fichtner]

@paride no, see https://forum.pfsense.org/index.php?topic=138822.0

contributors raising the issue are expelled from the community rather than the simple solution of bringing the source code back or saying that it's not open source anymore. ;)

[cobradevil]

Hello Frank,
It looks like pfsense made the patches available.
Is this correct or am i missing something?

https://github.com/pfsense/FreeBSD-src/commit/cb59ac304d30d5009537d7de0429792fb33d3db0

Best regards,
William

[fichtner]

It would seem so, let me reopen and check what we have :)

[cobradevil]

If you have any news on this like testing the functionality then give me a message.

Best regards,
William

[paride]

I'm also available to test stuff, just ping me. Given the number of providers using 6rd, I think it's very important for OPNsense to support it.

[toyowheelin]

I see this has been removed from the 18.7 milestone. Is there any news on when this will be implemented? I am looking to migrate from pfsense to opnsense but this feature would be needed to get ipv6 connectivity.

[fichtner]

Sorry, GitHub added „projects“ but the global projects are not visible to end users. We initially switched from milestone to project „18.7“, but didn’t know it looked like there was no designation at all now.

[toyowheelin]

Ah, no worries. Glad it didn't fall off the radar for the project as a whole.

[darkain]

If you need any help testing, this is a highly desired feature for me. I'm in the states with CenturyLink Gigabit Fiber service, a dedicated /29 block of IPv4 addresses from them (currently 1 unused that I can assign to testing OPNsense+6RD out for a while), and they use 6RD for their IPv6 implementation. I have dedicated lab hardware I can shove on this connection and open up SSH or other access to any devs that need to poke around (since I know not everyone has access to a live 6RD network)

[fichtner]

Alright, this is the patch not yet added to the 18.7 release candidate https://github.com/opnsense/src/commit/2bab086d4

You can test this update on an all up-to-date 18.1.x like so:

# opnsense-update -bkr 18.7-stftest -n "snapshots\/dummy"
# /usr/local/etc/rc.reboot

I would expect there is more to do in the GUI but if this doesn't crash we have a way forward. Please note that I have no viable test setup whatsoever so we need your help in bringing it back.

Cheers,
Franco

[cobradevil]

Hello Franco,

I can confirm that the wan_stf interface is back but it does not work yet.

Not sure what is wrong but this is the first attempt so I wanted to let you know it seems to work.

Best regards,
William

[fichtner]

Hi William,

Splendid! Can you poke the system log for ifconfig errors?

And maybe an ifconfig output for wan_stf from a working system and our system to compare.

Best case this is only a minor hiccup in the interface management code.

Thanks,
Franco

[cobradevil]

Ok, after looking at the working setup from 16.7 I saw my error in the 6rd prefix.
I have now a working setup with the patch you gave me without adjustments.

Anyone else too confirm this working without interfering the rest of the system?

Best regards,
William

[fichtner]

This sounds shiny. 1-2 more confirms would be awesome indeed. :)

[toyowheelin]

Sounds like I need to migrate my firewall when I get home :)

[fichtner]

I can provide test images if you need them.... easy to check with live mode boot without clobbering your install

[toyowheelin]

Not a bad idea, if thats something easy to roll together and I can just boot it off a usb stick I will give it a go.

[fichtner]

yeah, of course... amd64 in serial or vga ?

[toyowheelin]

amd64 vga

[paride]

Thanks @fichtner! If I update to 18.7-stftest, will it seamlessly update to 18.7 once it's out? I'll be away from home for about 10 days, but then I'll be glad to give it a try.

[fichtner]

It will, but make sure to lock the kernel and base under system: settings: firmware: packages. It will unlock for the major upgrade and I'll make sure to land this in the final images (maybe not RC so careful).

[fichtner]

amd64/vga for 18.1.10 with stf changes: https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-18.1.10-stftest-OpenSSL-vga-amd64.img.bz2

[toyowheelin]

Thank you, I will give it a try tonight when I get home.

[toyowheelin]

So, I had all intentions to getting this tested tonight, however windows is causing me issues getting the image onto my usb drive. It may not be until this weekend before I can report back.

Ok, I finally got the image to boot. So this will be a tomorrow/weekend project.

[RehaagJ]

Not working here, no wan_stf interface showing up. Found this in the log:
opnsense: /interfaces.php: The interface IPv4 '100.66.173.169' address on interface 'em1_vlan12' is not public, not configuring 6RD tunnel

I think that public IPv4 network addresses should not be a prerequisite, as providers who are using DS Lite are typically using private networks.

[fichtner]

@toyowheelin take your time 👍

@Uica I agree. Parts of the code already do this so I've committed c3a4e45a86 which will be in 18.1.11.

[toyowheelin]

That's the first time I have ever seen anybody using that address space. I suspect that opnsense uses a bogon filter on the public interface like pfsense. If that's true I suspect turning off the bogon filter on the wan interface will allow stf to work.

Edit: never mind, looks like I should have refreshed the page before commenting.

[fichtner]

is_private_ip() checks for 100.64.0.0/10 so that happens. But there was one spot in 6RD code that already had it commented out so.... :)

[RyuunoAelia]

I can't get it to work with swisscom 6rd. I don't know if it is because swisscom changed something since last time I made it work or if it is a problem with opnsense...

I get the wan_stf interface, everything seems configured correctly but I when I run packet capture on internet facing interface no packets goes to the 6rd border gateway and I get "Network is Down" errors. Nothing wrong in dmesg.

# uname -a
FreeBSD XXXXXXXXX 11.1-RELEASE-p11 FreeBSD 11.1-RELEASE-p11  2bab086d417(stf)  amd64

wan_stf: flags=4041<UP,RUNNING,LINK2> metric 0 mtu 1280
    inet6 2a02:120X:XXXX:XXXX:: prefixlen 60 
    nd6 options=1<PERFORMNUD>
    v4net 0.0.0.0/32 -> tv4br 193.5.29.1
    groups: stf 

## The prefix is 2a02::1200/28 and border gateway is 6rd.swisscom.com the complete ipv4 is embedded into the ipv6 address

PING6(56=40+8+8 bytes) 2a02:120X:XXXX:XXXX:: --> 2a00:1450:400a:801::200e
ping6: sendmsg: Network is down
ping6: wrote google.com 16 chars, ret=-1

I just don't know which diagnostics to run more than that.

[RyuunoAelia]

Forgot the gateway info:

# netstat -r -n
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            Y.Y.Y.1         UGS         re0
X.X.X.0/23      link#1             U           re0
X.X.X.X       link#1             UHS         lo0
[...]
193.5.29.1         Y.Y.Y.1         UGHS        re0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
default                           2a02:120X:XXXX:XXXX::c105:1d01 UGS    wan_stf
::1                               link#5                        UH          lo0
2a02:120X:XXXX:XXXX::             link#11                       UHS         lo0
2a02:120X:XXXX:XXXX::/60          link#11                       U       wan_stf
[...]

[fichtner]

stf patch was merged into 18.7-RC1, thanks for the feedback so far!

@RyuunoAelia not sure about swisscom yet, it would be best to have a larger discussion here as soon as RC1 is out in a week hopefully https://forum.opnsense.org/index.php?board=30.0

[RehaagJ]

I tested again with 18.1.11, and can confirm that it works now also with private networks. This is great news - I can finally get rid of pfSense! (Well, I can get rid of that once the Denverton drivers are backported, different topic, but that seems to be under way).

Thanks!

[fichtner]

@Uica thanks for confirming! Denverton ? For network? The stf test kernel also has the network backports from 11.2... can you check? If not we still need to do something, I'm not entirely sure.

[RehaagJ]

@fichtner: Yes, for network. Good to know that the backports are supposed to be there! I will test this, but it'll take some time, since I only have that Denverton hardware in my production system, which is currently running pfSense. I'll need to plan the migration carefully, so I can stay on that system if it works and wouldn't have to reinstall / restore the old system.

[fichtner]

well we do have a live system usb install... if you need one based on 18.1.11 let me know.

[fichtner]

Patches seem to work so I'm closing this. If you have specific issues with configuration please open a new ticket so we can look at it.

Thanks to all for your patience ❤️

[RehaagJ]

Good point, didn't think of a live system. Yes, would be great if you could provide one based on 18.1.11, I can then do the test much sooner.

[fichtner]

@Uica amd64 ... vga or serial?

[RehaagJ]

amd64 vga

[fichtner]

@Uica there you go https://pkg.opnsense.org/FreeBSD:11:amd64/snapshots/OPNsense-18.1.11-stf-OpenSSL-vga-amd64.img.bz2

[RehaagJ]

Thanks! I will test that tonight.

[RehaagJ]

@fichtner: I have tested the live system. The good news is that the backported Denverton network driver worked; 6rd with those interfaces looked good.
The bad news is that the hard disk was not recognized; several error messages about AHCI timeouts. The Denverton AHCI driver should probably be backported, too. Want me to open a new issue about that?

[fichtner]

Is pfSense already on 11.2 ? I don't feel so great about backporting AHCI. Is this not solvable with BIOS tweaking the controller ?

[RehaagJ]

pfSense releases are not yet on 11.2, development snapshots for 2.4.4 are.
I’ve tried both, the development snapshot and the latest release with the network kernel module (and only that one) loaded manually. Both worked and recognized the Denverton AHCI driver.
I would prefer not to tweak the BIOS settings, as it works well with pfSense, but if that’s the only way to migrate anytime soon, I’m willing to try.

[fichtner]

I'm just curious at this time to hear how much it differs from FreeBSD, because 11.2 is barely out. Let me check the sources again...

[fichtner]

okay, so this would be our backport, but not yet tested https://github.com/opnsense/src/commit/4d136a0d91

can we move this discussion to https://github.com/opnsense/core/issues/2473 ?

[RyuunoAelia]

@fichtner I quintuple-checked my configuration with as many sources as I could but still didn't find any problem with it. Is it possible to get an stf module with debugging enabled (since this is a compile-time option)? I tested in my installed system by enabling your update source so I can do it again no problem.

[RyuunoAelia]

btw I made some comments on the patch for things I find weird I'll let you have a look if these are accurate.

[fichtner]

@RyuunoAelia thanks for the source code review! I'll comment when I have more time because I would assume that it works in general but we can always go back and improve it on our own. I don't have a setup to test so we have to be careful about this though.

For now, would you mind opening a ticket for your specific case and insert all info about your connection / ISP again? And does this in fact work in pfSense for you (which version) or defunct there as well? I don't want to assume it, but there is the possibility that this code is not the code that is used to build the binary versions giving earlier trouble with the visibility of the code and their owner's attitude towards open source.

I'll look into the debug thing, but it will take till Monday to give you such a kernel.

[RyuunoAelia]

For the "ticket" could you be more specific where I should open it?

As for switching to pfsense I would rather let that as a last resort since swisscom is really a pain with custom routers since their infrastructure is made to control the routers with TR-069 and gives the router a token that needs to be passed in Option dhcp-class-identifier and pfsense did not support the format of that option correctly (full of commas everywhere). This is why I switched to opnsense in the first place since your dhcp configuration generation code was much smarter than the pfsense one.

If I do not set this DHCP option correctly, I will have a few hours downtime on my internet connection (EACH time I change routers) resolved by a phone call to the technical support of swisscom getting to the support level 2/3 and having them whitelist my custom router...

No problem waiting a bit for the debug. Last time I had 6rd working for swisscom was during their test phase some years ago.

[fichtner]

Ticket here in core is fine and I understand your motivation. We can make this work eventually. Worst case there are Swisscom contacts that I can ask if they know something. 😊

[darkain]

For whatever reason, I couldn't get this to work on 18.1.11 with the patch, but upgrading to 18.7-RC1 today, everything is working beautifully.

ISP: CenturyLink Gigabit FttH (United States)

As a note for others trying: make sure your IPv6 prefix is correct. I was trying to configure this thing based on some notes in some online forums, and they had the ISP's prefix wrong.

A note for the UI. In one of my tests, I put in the IPv6 prefix without specifying the bit mask length (because they're the same field, and I forgot about that). Maybe make this into two separate fields with a drop-down for the IPv6 bitmask, just like the IPv4 bitmask?

Also, another interesting issue. Setting up 6RD on the WAN interface creates the 6RD virtual interface. In the UI, you can "assign" and then configure the interface from there. This causes all sorts of issues, such as the IPv6 address being lost. Perhaps it would be best to now allow manual assignment of the virtual interface in the UI?

[RyuunoAelia]

Might be linked to a bunch of patches that went through due to #2521
I am not sure if they were included in 18.7.rc1 or not.

[fichtner]

@darkain yay, thanks for the update! We're collecting all the bits and pieces and hopefully land all of them in said beautifully working 18.7 release. FWIW, everything we discussed up until now will be in 18.7-RC2 for further testing. :)

A note for the UI. In one of my tests, I put in the IPv6 prefix without specifying the bit mask length (because they're the same field, and I forgot about that). Maybe make this into two separate fields with a drop-down for the IPv6 bitmask, just like the IPv4 bitmask?

I'm adding proper validation while not changing the field's input. It's too late to split the prefix from the input, but fixes your issue nevertheless.

Also, another interesting issue. Setting up 6RD on the WAN interface creates the 6RD virtual interface. In the UI, you can "assign" and then configure the interface from there. This causes all sorts of issues, such as the IPv6 address being lost. Perhaps it would be best to now allow manual assignment of the virtual interface in the UI?

Yes, it makes total sense to avoid selecting them as they already belong to an interface.

Both fixed in cc2902e4d -- you can try via:

# opnsense-patch cc2902e4d