Frr: Centos7 systemd startup failure

[filipecifali]

Me and @cristiancardoso just updated and ran into some problems today.

Using the Latest release 3.0

(https://github.com/FRRouting/frr/releases/download/frr-3.0/frr-3.0-2017101501.el7.centos.x86_64.rpm)

we got an error running after 1~3 minutes online

journalctl -u frr.service -xe returns:

Oct 19 18:03:42 router-sp-01.kinghost.net polkitd[548]: Unregistered Authentication Agent for unix-process:15442:233274034 (system bus name :1.1434, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.utf8) (disconnected from bus)
Oct 19 18:04:43 router-sp-01.kinghost.net systemd[1]: frr.service watchdog timeout (limit 1min)!
Oct 19 18:04:43 router-sp-01.kinghost.net systemd[1]: Unit frr.service entered failed state.
Oct 19 18:04:43 router-sp-01.kinghost.net systemd[1]: Triggering OnFailure= dependencies of frr.service.
Oct 19 18:04:43 router-sp-01.kinghost.net systemd[1]: Failed to enqueue OnFailure= job: No such file or directory
Oct 19 18:04:43 router-sp-01.kinghost.net systemd[1]: frr.service failed.
Oct 19 18:04:48 router-sp-01.kinghost.net systemd[1]: frr.service holdoff time over, scheduling restart.
Oct 19 18:04:48 router-sp-01.kinghost.net systemd[1]: Starting FRRouting (FRR)...

The service doesn't not work after the update, we had to make a rollback to 2.0.

We did not temper the RPM by any mean and your CI is saying it's building properly.

Is there a proper way to debug the service?

[mwinter-osr]

Thanks for reporting this. I'll look into it.

Can you tell me if this was an upgrade (installing the rpm over a previous installed 2.0 version) or a clean install?

[mwinter-osr]

There seems to be 2 issues here:

1) FRR fails somehow to start or crashes immediatly
I'm unable to reproduce this issue right now. Best guess is something in the config (ie from 2.0 if it's an upgrade) which causes it to break.
Can you please look in /var/log/messages for more details and can you share your configuration files?

2) watchfrr fails to restart frr in a failed case.
I found an issue here and will fix this. (and add a new test to CI to test the restart with watchfrr)

[filipecifali]

Sorry for the delay, yes it was an upgrade from 2.0 to 3.0 w/o changing the config.

Reading the docs: https://github.com/FRRouting/frr/wiki/FRR-2.0-%E2%86%92-FRR-3.0-Changelog

There's no indication on what to do that I found, maybe I'm missing something, should we recreate the config on this major release?

Thanks in advance, I'll test it as soon as it's available

[mwinter-osr]

No need to recreate config. (that would be a bug).
If possible, then please post config (or send it to me as email or on slack if you don't want to make it public). This should give me enough information to do further tests.

The issue with the failed restart is found and I have a fix. Working on adding a test for this as well before posting it. But this won't fix your issue as frr should never have crashed (or maybe it failed to start).

Happy to debug together with you over a skype/hangout/... - just ping me on slack or by email if you need help (but offline for the next approx 8 hrs).

[filipecifali]

I can't seem to join the free slack channel, can you provide me an e-mail that I can send you our config? Thanks.

Edit: now I was able to join LOL
my gtalk contact is the same for hangout if needed

[donaldsharp]

@filipecifali Please send me an email at sharpd at cumulusnetworks com and I'll get you in

[mwinter-osr]

RPM Fixes are in as Pull Requests for all the branches. This fixes the restart, but does NOT fix the reason for the crash or failure to start it at all.
Main Fixes:

• watchfrr config on systemd referenced init.d file which didn't exist (in /etc/frr/daemon)
• Upgrade from old package to new package failed to restart frr

Pull Request #1353 for master, Pull Request #1352 for 3.0 and Pull Request #1351 for 2.0

[mwinter-osr]

@filipecifali Any update on config or any chance to do some debug session over some screen-sharing service (Skype/Hangout/...) ?
Would love to find out what the cause of the crash or failure to start is... Ping me on Slack.

[mwinter-osr]

[Got the config privately from Filipe - Thanks]
Retesting with the config showed no issue so far.

Please provide the following output from the system with frr 2.0 installed:

• rpm -q -a frr (should show exact version of the RPM package installed - or if you built your own RPM, then please provide RPM to me)
• contents of /etc/frr/daemons (should show which daemons are running and how)
• If you can restest, then please check for messages in /var/log/messages (or wherever you
  have syslog configured to write to)
• Add the following config BEFORE the upgrade and provide the log afterwards:

    debug bgp neighbor-events
    log file /tmp/frr.log

• I suggest for the retest you are using the latest RPM which has the other fixes. Rebuilt packages are here: https://ci1.netdef.org/browse/FRR-FRR4/latestSuccessful/artifact

Alternatively, I'm can try to be online when you try this. (Ping me on Slack with a date/time if you prefer this)

[janaki-r-bhagwath]

Hi Any update on this?

I have the same error when I try versions Stable/3.0 as well as Stable/2.0 - Fedora 26
Attached is the error for reference. I compiled frr with -enable-systemd. Even then after certain period of inactivity I get following error
Nov 11 18:39:23 EPC-HV03.empatco.com systemd[1]: frr.service: Failed to enqueue OnFailure= job: No such file or directory
Nov 11 18:39:23 EPC-HV03.empatco.com systemd[1]: frr.service: Failed with result 'watchdog'.
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: frr.service: Service hold-off time over, scheduling restart.
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: Stopped FRRouting (FRR).
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: frr.service: Start request repeated too quickly.
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: Failed to start FRRouting (FRR).
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: frr.service: Unit entered failed state.
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: frr.service: Triggering OnFailure= dependencies.
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: frr.service: Failed to enqueue OnFailure= job: No such file or directory
Nov 11 18:39:29 EPC-HV03.empatco.com systemd[1]: frr.service: Failed with result 'watchdog'.

Possible connection to this error too - https://github.com/FRRouting/frr/issues/446

@rwestphal @mwinter-osr

[mwinter-osr]

Thanks for the failure log, but this (unfortunately) only shows the failed restart. The restart issue is an error in /etc/frr/daemons file with the wrong directory in it for watchfrr.
This is fixed in 3.0.2.

However, the real question is why any of the daemons crashed and that part is unresolved. Any chance you can try 3.0.2 and if it still happens provide the information I've asked in the previous comment?

Thanks,
Martin

[janaki-r-bhagwath]

Ok will try that out. Also can you please confirm that it is not because I am trying to use the frr on Fedora 26 instead of Fedora 24? Saw some dependency related post w.r.t quagga [0] (Ubuntu 12.04 version on Ubuntu 16.04) - [0] so just wanted to check if using a different Fedora version might help?

[0] - https://getsatisfaction.cumulusnetworks.com/cumulus/topics/warning-closing-connection-to-bgpd-because-of-an-i-o-error

[janaki-r-bhagwath]

New to Linux and the Routing world. For now have picked up the log file without the debug option enabled. Will figure out how to set the debug option. Also using version 3.0.2 and downloaded frr from GIT stable 3.0 branch hence don't have an RPM version. Verified the configure.ac file to confirm it's 3.0.2
loganddaemonfile.zip

Added the config for debug. Also attaching the new log file and the sh ip bgp neighbors o/p
messages_bg_neighbors.zip

@mwinter-osr

[mwinter-osr]

Thanks for the update.
However, it seems like you missed to include the /tmp/frr.log (if configured the way I suggested above).

I can see the watchdog timeout and the constant restart, but don't see the reason for the restart in the logs you've provided. Could you please provide me this logfile? Thanks.

I have not tried Fedora 26, but I'll see if I can do some quick check over the next day.

[mwinter-osr]

I've built the packages on Fedora 26 and installed it, with no issues so far (but very limited testing).
I really need some error messages, tracebacks or other logs to narrow down this issue. Or let me know if you are willing to do some screen sharing session (Hangout, skype, etc) or are able to give remote access to it.

[mwinter-osr]

Closing this issue in 5 days if nobody is able to provide more information on the failed startup or crash. I can't replicate the issue here.

[tomammon]

I believe I am hitting this same issue, I get the same error messages on my fresh centos 7 build with a fresh install of FRR from https://github.com/FRRouting/frr/releases/download/frr-3.0.2/frr-3.0.2-2017110701.el7.centos.x86_64.rpm. I am willing to do screen sharing to give you visibility on this, please contact me and we can get something scheduled.

[mwinter-osr]

Thanks to @tomammon for a long shared screen debugging issue.

The bug is not a FRR crash, but systemd restarting FRR

Here is the issue:
1) The systemd config of FRR uses a watchdog (as seen with WatchdogSec=60s in /usr/lib/systemd/system/frr.service
2) watchfrr is the process handling and sending the watchdog signal.

If a user only enables some routing daemons, but no watchfrr, then systemd will timeout on the watchdog and restart FRR

Possible ways to fix:

• Make watchfrr mandatory (ie don't even allow enable/disable in daemon file anymore)
• Disable systemd watchdog somehow if watchfrr is not enabled
• ???

[mwinter-osr]

Temp Workaround for anyone running into this issue:

1) Enable watchfrr in /etc/frr/daemon (preferred)
or
2) comment out WatchdogSec=60s in /usr/lib/systemd/system/frr.service

[philipishungry]

Simply for your information ,I am running into the same situation ( fresh centos7-1708-minimal install / fresh frr-3.0.2-2017110701.el7.centos.x86_64.rpm install).

[mwinter-osr]

@philipishungry Should be same issue with watchfrr not enabled by default in the daemon config.
Can you verify the workaround?
I'll try to make sure to have watchfrr enabled by default in future RPMs (and add comment with this info in daemon config)

[philipishungry]

@mwinter-osr , yes, both work-arounds function correctly. I have chosen to enable watchfrr.

[cardosocristian]

Hello, I installed version 3.0.3 in my centos 7 and the bug did not occur anymore.
Thank you

[philipishungry]

I have had a look at the code to see what is going on with systemd/watchdog and I have found that the function call, "systemd_send_started()", to start up the systemd side of things is initiated from "watchfrr/watchfrr.c".
So that means, watchfrr needs to be enabled to allow the systemd watchdog, therefore we would have 2 watchdogs running.
Further, if we do run with the systemd startup, the sd_watchdog_enabled() is sent from the watchfrr process, however, sd_watchdog_enabled() returns 0, ie, watchdog not enabled, complaining that the main PID which is currently not known, even if PIDFile=/var/run/frr/watchfrr.pid is set. I am assuming that the pidfile for watchfrr is only created after the sd_watchdog_enabled() function is called.

Would it not be better for zebra daemon to transmit the sd_watchdog_enabled check / sd_notify keepalives but then, that would not track, say , if ospfd or another daemon crashes. Then should each daemon require a systemd unit file, watchdog etc.?

here is output of some tinkering i did to discover what I have.

Jan 25 14:58:02 frr-01 frr[1627]: Starting FRRouting monitor daemon:
Jan 25 14:58:02 frr-01 watchfrr[1655]: watchfrr 3.0.3 watching [zebra], mode [phased global restart for any failure]
Jan 25 14:58:03 frr-01 watchfrr[1655]: zebra state -> up : connect succeeded
Jan 25 14:58:03 frr-01 systemd[1]: frr.service: Got notification message from PID 1655, but reception only permitted for main PID which is currently not known
Jan 25 14:58:03 frr-01 watchfrr[1655]: Watchfrr: Notifying Systemd we are up and running
Jan 25 14:58:03 frr-01 watchfrr[1655]: PID sent from systemd_send_started: 0
Jan 25 14:58:03 frr-01 watchfrr[1655]: WATCHDOG_PID: 1387798403
Jan 25 14:58:03 frr-01 watchfrr[1655]: systemd watchdog not enabled - not sending watchdog keepalives!
Jan 25 14:58:03 frr-01 frr[1627]:  watchfrr[  OK  ]
Jan 25 14:58:03 frr-01 systemd[1]: Started FRRouting (FRR).

[donaldsharp]

@philipishungry why are you stating that you have 2 watchdogs running?

We consider watchfrr the responsible party for all the FRR daemons up and running, which is why we put this systemd call here.

[philipishungry]

when you start watchfrr and then call "systemd_send_started()" from watchfrr, there is a call to sd_watchdog_enabled() which then starts a thread to send keepalives to systemd.
so that would be 2 watchdogs, watchfrr & systemd keepalives? is that not correct or I may be barking up the wrong tree ;-)

[donaldsharp]

@philipishungry I think your understanding is a bit off.

1) watchfrr sends pings to each running daemon to ensure that they are listening and responding. If they do not respond in the timeout window, watchfrr considers the daemon dead so it will restart the malfunctioning daemon following it's own set of rules.
2) watchfrr sends a message to systemd that it FRR has successfully started up on bootup.
3) watchfrr sends a I'm alive message every timeout interval/3 to systemd that it is still running.

So effectively watchfrr watches individual daemons and ensures that they are running, if watchfrr breaks and stops notifying systemd than it restarts the FRR service from the ground up.

In the long term we expect watchfrr to gain new functionality( like auto-start of daemons if vtysh sees daemons cli being entered )

[Amar-Chaudhari]

I am still facing this issue on CentOS 7 with installation from source.
However, the temporary solution mentioned above does seem to solve this.

[mwinter-osr]

@Amar-Chaudhari: I assume you are running FRR 3.0.3?
If you needed the workaround, which one did you use? Did you had to enable the watchfrr?
(It should have been enabled after upgrade to 3.0.3)

Was this a clean install of FRR 3.0.3 or an upgrade from an earlier version (which previous version?)

[Amar-Chaudhari]

@mwinter-osr
Yes, I am using FRR 3.0.3.
I had to comment out WatchdogSec=60s in /usr/lib/systemd/system/frr.service
It was a clean install build from source as per the documentation here -https://github.com/FRRouting/frr/wiki/Building-FRR-on-CentOS-7

[donaldsharp]

Documentation is missing a --enable-systemd=yes. I would rebuild with this.
Additionally I'll fix the doc.

[Amar-Chaudhari]

@donaldsharp
Trying to rebuild with the systemd flag but getting the following error.

./configure \
    --bindir=/usr/bin \
    --sbindir=/usr/lib/frr \
    --sysconfdir=/etc/frr \
    --libdir=/usr/lib/frr \
    --libexecdir=/usr/lib/frr \
    --localstatedir=/var/run/frr \
    --with-moduledir=/usr/lib/frr/modules \
    --enable-pimd \
    --enable-snmp=agentx \
    --enable-multipath=64 \
    --enable-ospfclient=yes \
    --enable-ospfapi=yes \
    --enable-user=frr \
    --enable-group=frr \
    --enable-vty-group=frrvt \
    --enable-rtadv \
    --disable-exampledir \
    --enable-watchfrr \
    --disable-ldpd \
    --enable-fpm \
    --enable-nhrpd \
    --enable-eigrpd \
    --enable-babeld \
    --enable-systemd=yes \
    --with-pkg-git-version \
    --with-pkg-extra-version=-MyOwnFRRVersion

Error:
configure: error: enable systemd has been specified but systemd development env not found on your system
OS
CentOS Linux release 7.4.1708 (Core)

[donaldsharp]

install systemd-devel

[q2dg]

I can't start FRR 7.0 in Fedora 29 yet.

What I've found is that in install instructions (http://docs.frrouting.org/projects/dev-guide/en/latest/building-frr-for-fedora.html#) there are two files ("redhat/frr.service" and "redhat/frr.init") which are missing: they must be "tools/frr.service" and "tools/frr.in" instead. But then, when I want to start systemd service, I get a "which stop-start-daemon not found" message error: looking at "frrinit.sh" (script called by ExecStart= line in frr.service file) I've seen it looks for a "ssd" binary which is searched in /usr/lib/frr folder but it's missing, so I've thinking it causes the error; I've tried to copy this from "tools" folder but the error remains the same.

I gave up.

[qlyoung]

The instructions need to be updated as we have updated our init script setup.

[ton31337]

this is fixed. closing.