Core: Feature Request: Allow CARP to Connect/Disconnect a PPPoE Connection

[SirWobbyTheFirst]

I have not seen any similar functionality in OPNsense, at least as of 18.7.10 so I am requesting a feature here, would it be possible to have CARP begin dialling a PPPoE connection when it detects a node has moved from Backup to Master and likewise would it be possible for CARP to then disconnect said PPPoE connection when it detects the Master has come back online?

I've seen many posts on the Internet that state this needs a single modem capable of doing the PPPoE connection itself or to use a script to poll the Master and connect/disconnect as needed but that seems like it would not work if the Master has just been taken down for maintenance and is still actively listening on it's LAN interface.

[maurice-w]

This feature was added to "the (m)other sense" about a year ago: https://github.com/pfsense/pfsense/pull/3830

[AdSchellevis]

@maurice-w @SirWobbyTheFirst
You can probably do this quite easily with the events in place, an interface_configure configure should probably be enough to reconnect in this case.

You can find an example of an event script here https://github.com/opnsense/plugins/blob/master/net/frr/src/etc/rc.syshook.d/carp/50-frr (installs into /usr/local/etc/rc.syshook.d/carp/50-frr)

The action to reconfigure an interface on OPNsense can be started using:

configctl interface reconfigure opt1 

(replace opt1 with the actual interface you like to reconfigure)

In OPNsense these services are pluggable, so its quite easy to extend with logic which suits specific needs.

I don't think we should try to reconnect pppoe interfaces blindly, this might have side affects for people with separate connections.

[mimugmail]

Also saw two guys asking today in IRC, also regarding DHCP:

(02:33:59) phibs: Can I get any type of HA with just a single external dynamic IP from my ISP?

(11:10:25) hemi770: wondering if it's possible to do CARP with a dynamic public IP?

The call to reconfigure should be easy, but isn't there an action required to "unconfigure" the interface on backup unit?

[AdSchellevis]

It might be, although I'm not sure. I can't test this on my end either. If the protocol disconnects when a new connection is created, there shouldn't be a need to unconfigure the backup. The feature in pfSense doesn't seem to be doing something like that, but I can be mistaken.... it's a lot of code, for something that should be rather easy.

[fichtner]

a single script in rc.syshook.d/carp could do this. the only magic seems to be in reconfiguring PPP specifically and skipping the BACKUP CARP: https://github.com/pfsense/pfsense/pull/3830/files#diff-1332c372788c9e1a8c6c9bae9ebb55a5R1848

[mimugmail]

This kind of setup only works when you have a modem with multiple switch ports in front. So for PPPOE you'd need to disable the interface or disconnect the pppoe session on carp event. Regarding DHCP it'll be a bit tricky since most cable providers only allow ONE mac address to be connected to it's modem.

I should be able to offer test equipment here if needed / interested

[AdSchellevis]

well, let first wait until someone who needs/wants the issue tries the above..... I don't think we should add a feature, but we might be able to guide someone to a solution with what we already have.

[mimugmail]

While testing #2279 I also tested behaviour with this. Plugged in both systems in a modem running in bridge mode. Both systems have assigned interface and same pppoe credentials. With first come first serve one of the dials in, other stucks. As soon as I manually disable on the unit dialed in the other one comes up. As multiple action might reconfigure interfaces we'd need a script also disabling pppoe interface in config.xml

[AdSchellevis]

@mimugmail I've looked into the code a bit, can you create the following (executable) file on your box and test how it behaves?

/usr/local/etc/rc.syshook.d/carp/20-ppp

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

list ($vhid, $iface) = explode('@', $subsystem);

$a_ppps = &config_read_array('ppps', 'ppp');
foreach ($a_ppps as $ppp) {
    if ($ppp['ports'] == $iface) {
        foreach($config['interfaces'] as $ifkey => $interface) {
            if ($ppp['if'] == $interface['if']) {
                log_error("{$iface} is connected to ppp interface {$ifkey} set new status {$type}");
        if ($type == 'BACKUP') {
                   interface_bring_down($ifkey);
        } else {
                   interface_ppps_configure(false, $ifkey);
        }
            }
        }
    }
}

I couldn't test the interface_ppps_configure part, since I don't have pppoe over here, if you can provide me with a test setup which I can easily switch from master <-> backup, I'll gladly take a closer look if this code isn't enough.

[prog42]

@AdSchellevis Thank you very much for the script you posted. I tested it with PPPoE on VDSL on OPNsense 19.7.6.
It works perfectly after changing the command for bringing up the interface to:

interface_ppps_configure($ifkey);

It would be nice to have this as a GUI configurable option.

[AdSchellevis]

@prog42 thanks for testing, I'm not sure if we're going to include it by default, it could also be an option to add it as a plugin, I'll keep the issue open for now and wait for some more feedback from others as well.

[mimugmail]

@prog42 does this also works for you when entering maintainence mode? I just tested the script, but it doesn't seems to work with mnt mode or disabling carp on master device

[prog42]

@mimugmail I tested again and it works also in maintenance mode in my setup. I had to wait one or two minutes for the pppoe link to connect.

You can check the messages generated by the script. When becoming MASTER I get:
opnsense: /usr/local/etc/rc.syshook.d/carp/20-ppp: vtnet2 is connected to ppp interface opt3 set new status MASTER

When becoming BACKUP:
opnsense: /usr/local/etc/rc.syshook.d/carp/20-ppp: vtnet2 is connected to ppp interface opt3 set new status BACKUP

[mimugmail]

Did you Set a carp pppoe address or just the usual dialin in both units?

[mimugmail]

I just retried in my lab, but I only see openvpn messages related to rc.syshook.d and carp.

[mimugmail]

@prog42 can you please quick check on your side, did you anything to virtual IP's or anything else than just placing the script and chmod? Does not work for me.

[MarcSN311]

I have the same problem as @mimugmail . The hook does not seem to run. I added a log_error(); at the start of the script, just to make sure, but nothing in the logs...

[AdSchellevis]

Since I'm not using this feature myself nor have customers depending on it, I'm going to drop the ticket back into the pool if there's nothing concrete any time soon (which will auto close/time-out shortly after)

Possible ways forward for this event hook are the following:

• Add as example in the docs, so people can build their own variations
• Add as plugin, needs a maintainer, preferably not me :)
• Integrate in core, which needs at least a couple of people testing and confirming its relevance or explaining where it needs improvement.

[prog42]

@mimugmail

Did you Set a carp pppoe address or just the usual dialin in both units?

The interface the vdsl-modem is connected to has a local ip-address on both nodes and a local virtual address with CARP. I need these addresses anyway to access the web-ui of the vdsl-modem.

I just double checked the setup and even when enabling CARP Maintenance Mode the switchover works.

[mimugmail]

How can you set a local IP address when type is pppoe?

[prog42]

@mimugmail The physical ethernet interface the modem is connected to has a local IP address. Not the virtual pppoe device.

[mimugmail]

Sorry, I dont get it, the interface connected to the modem must be set to pppoe in order to enter ppp credentials, there is no field for an IP address

[prog42]

This is the configuration of the physical ethernet interface vtnet2 with the static ip address. Unfortunately the internal name vtnet2 is not visible on this screenshot, as I renamed the interface to br254_uplink_dsl:

In the PPPoE configuration this interface vtnet2 is used as the link interface:

vtnet2 / br254_uplink_dsl also has a CARP virtual IP:

I hope this helps.

[mimugmail]

Ok, thanks for the screenshots. I was able to reproduce successfully.
What didn't work was stable manually switching over. I need to enter mnt mode, and then go to interfaces and disconnect pppoe. After this I must not forget to connect on backup unit again so the mpd5 process is started again. If I forget this and want to switch back both units are offline. (I only manage from remote).

Since I can't take over because of the lack of PHP knowledge I'll also not cover this topic.

Maybe some day one customer is willing to sponsor this feature, but for now I'll add a dsl router doing the dialin and behind CARP with static IPs and exposed host.

[AdSchellevis]

This issue has been automatically timed-out (after 180 days of inactivity).

For more information about the policies for this repository,
please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md for further details.

If someone wants to step up and work on this issue,
just let us know, so we can reopen the issue and assign an owner to it.

[qdrop17]

Hi everyone.
I'm trying to get this working: Triggering a command upon CARP state change. Yet I don't find any further information on how to place scripts inside this directory (/usr/local/etc/rc.syshook.d/carp/). Are there any examples for shell scripts? How to change a value inside the configuration and apply it? Any help is highly appreciated.

[jPhy]

You can probably do this quite easily with the events in place, an interface_configure configure should probably be enough to reconnect in this case.

You can find an example of an event script here https://github.com/opnsense/plugins/blob/master/net/frr/src/etc/rc.syshook.d/carp/50-frr (installs into /usr/local/etc/rc.syshook.d/carp/50-frr)

The action to reconfigure an interface on OPNsense can be started using:

configctl interface reconfigure opt1 

(replace opt1 with the actual interface you like to reconfigure)

Would it be possible to add detailed instructions how to do this? Triggering a pppoe connect/disconnect on CARP state change backup->master/master->backup is precisely what I would like to do.

[AdSchellevis]

@jPhy https://github.com/opnsense/core/issues/3185#issuecomment-537631280

[s3m73x]

Hello, I am also missing DHCP for CARP-Interfaces because we have a ISP that hands out his static-IPs only if you do DHCP-requests. But not PPPoE but simple Ethernet TCP/IP.

It could be done with an ugly hack where we would spoof the DHCP-requests with the source-MAC of the CARP-NIC and then check which firewall is up and then automate the configuration of the CARP-Interface there. But honestly that is nothing I would want to experiment with in a running production environment. :D