Fail2ban: bsd-sshd.conf not filtering on FreeBSD 12.0-RELEASE (RFC 5424 log-format)

[g-a-c]

Environment:

• Fail2Ban version (including any possible distribution suffixes): Fail2Ban v0.10.4
• OS, including release name/version: FreeBSD 12.0-RELEASE
• [x] Fail2Ban installed via OS/distribution mechanisms
• [x] You have not applied any additional foreign patches to the codebase
• [x] Some customizations were done to the configuration (provide details below is so)

The issue:

Fail2Ban doesn't seem to be banning SSH connections, it looks to me like the log prefix format has changed between 11.2-RELEASE-p6 and 12.0-RELEASE such that the Fail2Ban default bsd-sshd.conf file doesn't match anymore

Steps to reproduce

Install FreeBSD 11.2-RELEASE
Set up a jail using the included bsd-sshd.conf filter
Observe that users are banned
Upgrade to FreeBSD 12.0-RELEASE
Observe that users are no longer being banned

Expected behavior

According to my jail.local, SSH users should be banned after 2 failures

Observed behavior

One IP has been able to try 39 times and still not been banned

Any additional information

Configuration, dump and another helpful excerpts

fail2ban-regex /var/log/auth.log /usr/local/etc/fail2ban/filter.d/bsd-sshd.conf shows no matches after the time I did the upgrade.

This is the old format of errors (for example "PAM Authentication"):
Dec 18 17:38:24 server sshd[74255]: error: PAM: authentication error for root from 121.22.80.117

This is the new format of the same message:
Dec 20 04:32:17 server 1 2018-12-20T04:32:17.177809+00:00 server.domain.com sshd 96961 - - error: PAM: Authentication error for root from 95.211.230.165

Any customizations done to /etc/fail2ban/ configuration

jail.local:

[ssh]
enabled = true
filter  = bsd-sshd
action  = pf[port=22]
logpath = /var/log/auth.log
maxretry = 2

Relevant parts of /var/log/fail2ban.log file:

_preferably obtained while running fail2ban with loglevel = 4_

Relevant lines from monitored log files in question:

[sebres]

Hmm... Really dual date-time strings (and server names)?
Looks weird (and unintentionally).
@fail2ban/maintainers, @fail2ban/bsd Is there someone with the knowledge what is going on here?

In this case the default common __prefix_line would not match (and possibly not only in sshd).

Don't know what exactly filter bsd-sshd does (maintained by bsd?), but you can overwrite the prefix-line with your own for default sshd-filter.
Either directly in jail:

[ssh]
enabled = true
filter  = sshd[__prefix_line="^\s?\S+ \d+ \S+ \S+ sshd \d+ \S+ \S+\s+"]
...

Or in local-version of the filter-file (e. g. filter.d/sshd.local or filter.d/bsd-sshd.local):

[Definition]
__prefix_line = ^\s?\S+ \d+ \S+ \S+ sshd \d+ \S+ \S+\s+

I don't know which values exactly all this entries may have, possibly this regex-part could be written more precise. At the moment it is simply covered like here (ahead-anchored):

-       server  1  2018-12-20T04:32:17.177809+00:00 server.domain.com sshd 96961  -   - 
+^\s?   \S+    \d+ \S+                              \S+               sshd \d+    \S+ \S+ \s+

PoC:

$ msg='...'
$ fail2ban-regex "$msg" sshd
... Lines: 1 lines, 0 ignored, 0 matched, 1 missed ...
$ fail2ban-regex "$msg" 'sshd[__prefix_line="^\s?\S+ \d+ \S+ \S+ sshd \d+ \S+ \S+\s+"]'
... Lines: 1 lines, 0 ignored, 1 matched, 0 missed ...

[churchers]

I've noticed the same problem, but strangely only on one of two FreeBSD 12 servers. On top of this, my maillog is still in the old format. I really don't know what's going on but there are various references around to the inclusion of rfc5424 format in 12, although this shouldn't be on by default.

In my case I am trying to match pure-ftpd log messages. Note that because some logs appear to still show the old format, I don't quite know how you would work it if you need jails for multiple logs that are using different formats... (I expect it will mean specific regex's for difference services rather than being able to use the global prefix regex)

On top of this, as already mentioned the lines have two timestamps, and also the hostname twice. (Once just the first part, and the full hostname in the rfc5424 part).

Also of note, while writing a fail2ban regex for this, it seems that FreeBSD doesn't include a PRI value, but I can't see this being optional in the spec.

Anyway, below is the filter.d/common.local I am currently using for these messages and this is working for my pure-ftpd logs -

This is probably a bit over the top but I thought I may as well match the actual spec rather than just bodge it.

[DEFAULT]

# structured data, either NIL (-) or data pairs inside square brackets
#
__rfc5424_sd_id = \S+
__rfc5424_sd_param = [^\=]+=\"[^\"]*\"
__rfc5424_sd_element = \[%(__rfc5424_sd_id)s(?:\s%(__rfc5424_sd_param)s)+\]
__rfc5424_data = (?:\-|(?:%(__rfc5424_sd_element)s)+)

# 5424 timestamp
#
__rfc5424_ts_date = [\d\-]+
__rfc5424_ts_time = [\d\:\.]+
__rfc5424_ts_offset = (?:Z|[\+\-][\d\:]+)
__rfc5424_timestamp = (?:\-|%(__rfc5424_ts_date)sT%(__rfc5424_ts_time)s%(__rfc5424_ts_offset)s)

# 5424 parts
#
__rfc5424_nil_or_ascii = (?:\-|\S+)
__rfc5424_pri = \<\d+\>
__rfc5424_version = \d+
__rfc5424_app_pid_msgid = %(__rfc5424_nil_or_ascii)s\s%(__rfc5424_nil_or_ascii)s\s%(__rfc5424_nil_or_ascii)s

# 5424 header
# not sure PRI is optional but my logs don't seem to have it
# note the spec lists no space between PRI and VERSION
#
__rfc5424_header = (?:%(__rfc5424_pri)s)?%(__rfc5424_version)s\s%(__rfc5424_timestamp)s\s%(__hostname)s\s%(__rfc5424_app_pid_msgid)s

# full prefix
# bsd log seems to have old format date then partial hostname, then rfc5424 message (including timestamp and hostname again)
#
__prefix_line = %(__date_ambit)s?\s*%(__hostname)s\s(?:%(__rfc5424_header)s)\s(?:%(__rfc5424_data)s)\s+

# fail2ban-regex -vvvv "Dec 19 15:22:25 ftp 1 2018-12-19T15:22:25.456002+00:00 full.host.name pure-ftpd 69605 - - ([email protected]) [WARNING] Authentication failed for user [matt]" ./filter.d/pure-ftpd.conf
... snip ...
Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.01 sec]

(Edited 21/12 to fix data regex and allow to match multiple SD elements - [...][...]

[sebres]

I've noticed the same problem, but strangely only on one of two FreeBSD 12 servers.

So it may be then just some configuration issues (of syslog etc)?

[churchers]

I really don't get it. The server I'm using is a completely stock 12 install (although I did perform an 11 install as I had the installer lying around, then immediately upgrade).

No syslog config has been changed and the defaults/rc.conf file has no mention of the new -O option which is supposed to allow changing to the new format. I'm not aware of any other way of affecting syslog config.

• Add fprintlog_rfc5424() to emit RFC 5424 formatted log entries.
• Add a "-O" command line option to enable RFC 5424 formatting. It would
  ...

AFAIK it shouldn't be using a new format without being set manually. Clearly based on this issue it is affecting base services like ssh, and I am seeing the same format for pkg messages. However my maillog is the old format...

Dec 17 15:28:23 ftp 1 2018-12-17T15:28:23.788516+00:00 host.name pkg 75253 - - perl5-5.26.2 installed

Dec 20 05:30:03 ftp sm-mta[71919]: alias database /etc/mail/aliases.db out of date

I may ask the question on the mailing list as I don't see why I should be seeing any new format logs without opting to use it.

[chtheis]

I, the maintainer of the FreeBSD port, just went on holidays. Thus I'm
not of much help to find a solution. But I'll follow the discussion,
there are already some suggestions how to deal with it. If you find a
solution feel free to submit a bug report for it at FreeBSD but please
make sure it handles also versions prior to 12. I can then accept it as
maintainer. Otherwise we'll have to wait until my return before I can
submit the bug report.

Fortunately FreeBSD 12 just came out and those updating now will come
over this thread. And hopefully find a workaround.

Christoph

[sebres]

It looks indeed that FreeBSD 12.0 automatically enabled RFC 5424 for system logging (per default).

We may provide some other common-conf to support this.
Unless it is not yet implemented (or unwanted at all), disable of rfc5424 logging may even solve the issue.

By the way, if I try to test it on my debian machine:

logger -t 'test:auth' -i -p auth.info "pam_unix(test:auth): test log"
logger --rfc5424 -t 'test:auth' -i -p auth.info "pam_unix(test:auth): test log"

auth.log gets then 2 new entries:

Dec 20 18:05:28 server test:auth[19582]: pam_unix(test:auth): test log
Dec 20 18:05:29 server 1 2018-12-20T18:05:29.535099+01:00 server test:auth 19583 - [timeQuality tzKnown="1" isSynced="0"] pam_unix(test:auth): test log

And even also with dual info for time and server (which looks still weird to me). Possibly there is still another config-entry that could disable old prefix-line (so will remove part Dec 20 18:05:29 server)...

Also my first shot of RFC5424 "support" was a bit too simple, so implies now capturing of more complex info, here one - got replaced with [timeQuality tzKnown="1" isSynced="0"], thus the regex in my example above should be replaced as something:

-^\s?\S+ \d+ \S+ \S+ sshd \d+ \S+ \S+\s+
+^\s?\S+ \d+ \S+ \S+ sshd \d+ \S+ (?:\S+|\[[^\]]*\])\s+

Just, because the case where this contains the bracket-char in-between (e. g. [somevalue, othervalue="[...]"]) is imaginable, this could cause in the future much more complex __prefix_line or even total impossibility to use prefregex-feature (because the anchoring from both sides could be necessary).

[churchers]

Apparently the following is valid 5424 structured data, but note that a space is not allowed between the two sections (and a closing square bracket is not valid in the data), so matching "]s" may be enough to make sure the regex matches the whole element.

[exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][examplePriority@32473 class="high"]

[churchers]

Just to add, my regex in the first comment does a decent job of matching the 5424 entries on FreeBSD (although it would need a slight change to handle multiple structured data sections - [fixed]). It could of course be heavily simplified by replacing most of the matches with a simple S regex.

(Sorry to add this as an additional comment but many GitHub features don't work on my iPad, including the drop down menu with the edit option...)

[sebres]

and a closing square bracket is not valid in the data

Why you are so sure? :)

$ logger --rfc5424 --sd-id xxx@123 --sd-param test=\"[brackets]\" -t 'test:auth' -i -p auth.info "pam_unix(test:auth): test 
$ grep xxx@123 /var/log/auth.log
... [timeQuality tzKnown="1" isSynced="0"][xxx@123 test="[brackets]"] ...

They are allowed until enclosed in quotes. Without quotes, it produces an error logger: invalid structured data parameter: 'test=[brackets]'.
Thus as already wrote there is no simple way to match it properly.

[sebres]

So the simplest regex that matches is then something like:

^\s?\S+ \d+ \S+ \S+ %(_daemon)s \d+ \S+ (?:\S+|(?:\[(?:[^\]"]*|"[^"]*")*\])+)\s+

(without taking a closer look at RFC / logger source code for implementation details)

[churchers]

Sorry the spec mentions that a bracket is invalid in the ID, but you're right, it can be present in the value. It does mention that this should be escaped, but then I assume that is purely for the application writing the log data and the information will appear un-escaped in the log.

[miklosq]

Thanks for the above hints, I made my own research as well but I actually do not think the issue is with the ___prefix_line_ on FreeBSD 12. For me, the following in the bsd-sshd.conf filter config fix worked:

Old:
failregex = ^%(__prefix_line)s(?:error: PAM: )?[A|a]uthentication (?:failure|error) for .* from <HOST>\s*$
New:
failregex = ^%(__prefix_line)s.+(?:error: PAM: )?[A|a]uthentication (?:failure|error) for .* from <HOST>\s*$

Notice the _.+_ added after the (__prefix_line)s. With the above fix, I managed to get matches for the failregex. I left the ___prefix_line_ from stock config unchanged in the common.conf .

For the following SSH log pattern I currently have (/tmp/badsshattempts):

Dec 24 14:18:26 your.host 1 2018-12-24T14:18:26.590346+01:00 your.host sshd 45097 - - error: PAM: Authentication error for root from 112.85.42.144

$ fail2ban-regex /tmp/badsshattempts /usr/local/etc/fail2ban/filter.d/bsd-sshd.conf
Running tests
=============

Use   failregex filter file : bsd-sshd, basedir: /usr/local/etc/fail2ban
Use      datepattern : Default Detectors
Use         log file : /tmp/badsshattempts4
Use         encoding : US-ASCII


Results
=======

Failregex: 429 total
|-  #) [# of hits] regular expression
|   1) [429] ^(?:\[\])?\s*(?:<[^.]+\.[^.]+>\s+)?(?:\S+\s+)?(?:kernel: \[ *\d+\.\d+\]\s+)?(?:@vserver_\S+\s+)?(?:(?:(?:\[\d+\])?:\s+[\[\(]?sshd(?:\(\S+\))?[\]\)]?:?|[\[\(]?sshd(?:\(\S$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [429] {^LN-BEG}(?:DAY )?MON Day %k:Minute:Second(?:\.Microseconds)?(?: ExYear)?
`-

Lines: 429 lines, 0 ignored, 429 matched, 0 missed
[processed in 0.10 sec]

Just for clarification, in my analysis the content of ___prefix_line_ does not matter. I have got log patterns with lines of two different formats (lines with non-rfc5424 and lines with rfc5424) and they both are matched indeed after the fix in the filter shown above. For e.g. see the relevant filter line in _bsd-sshd.conf_ below:

failregex = ^%(__prefix_line)s.+Did not receive identification string from <HOST> port \d*$

Notice the .+ added. The above regex filter matches all of the following log patterns, regardless of RFC5424 log format:

Dec 24 21:30:15 your.host 1 2018-12-24T21:30:15.965780+01:00 your.host sshd 47098 - - Did not receive identification string from 173.255.205.93 port 48916
Dec 25 11:48:18 your.host 1 2018-12-25T11:48:18.198628+01:00 your.host sshd 50775 - - Did not receive identification string from 196.52.43.98 port 54847
Dec 25 20:16:40 your.host 1 2018-12-25T20:16:40.835696+01:00 your.host sshd 53272 - - Did not receive identification string from 79.49.217.58 port 60000
Dec 28 07:56:18 your.host sshd[68117]: Did not receive identification string from 196.52.43.110 port 55139
Dec 29 16:26:42 your.host sshd[76054]: Did not receive identification string from 71.6.146.186 port 44385

Just for the sake of completeness, my ___prefix_line_ currently from _common.conf_ is (hopefully the stock version, unchanged but please compare it with yours):

__prefix_line = %(__date_ambit)s?\s*(?:%(__bsd_syslog_verbose)s\s+)?(?:%(__hostname)s\s+)?(?:%(__kernel_prefix)s\s+)?(?:%(__vserver)s\s+)?(?:%(__daemon_combs_re)s\s+)?(?:%(__daemon_extra_re)s\s+)?

My syslogd is running with _-s_ option and verbose mode is not turned on.

I hope the above helps.

[churchers]

Notice the .+ added after the (__prefix_line)s. With the above fix, I managed to get matches for the failregex. I left the __prefix_line from stock config unchanged in the common.conf .

By the look of it, everything in the original __prefix_line has a ? quantifier to make it optional. As such I expect none of that is matching and your .+ is effectively matching the entire start of the log line.

To be honest I'm not 100% sure why the original prefix patterns make such an effort to match all the various components of the line. They're all non-capturing so it doesn't seem like anything can make use of the matches anyway..

[miklosq]

Hi

By the look of it, everything in the original __prefix_line has a ? quantifier to make it optional. As such I expect > none of that is matching and your .+ is effectively matching the entire start of the log line.

Except for the ___date_ambit_ qualifier, I suppose (question mark within a paranthesis or outside of it makes the difference). I am not a regex guru, so I am assuming it. You can, however, try it: just feed a log pattern replaced with spaces in the place of ___prefix_line_ and it will not match if it does not find a date string in the pattern with RFC5424 format.

Thanks for the hint though: I re-visited and tested with the old date format and it is matched. So the issue is with any log pattern that is containing the new RFC5424 date format, to which FreeBSD v12 syslogd daemon might have switched to, as the previous contributors have managed to figure out. I suppose the full solution would be to fix the ___prefix_line_ but that comes from the global common.conf. Keeping the scope within this bug report, I would suggest to apply the .+ (or with 'dotstar' which would also work) modifier in the bsd-sshd.conf filter.

You can apply the following patch to the file /usr/local/etc/fail2ban/filter.d/bsd-sshd.conf and fail2ban will start to work on logline patterns that contain the RFC5424 date format:

26,34c26,34
< failregex = ^%(__prefix_line)s(?:error: PAM: )?[A|a]uthentication (?:failure|error) for .* from <HOST>\s*$
<             ^%(__prefix_line)sDid not receive identification string from <HOST>$
<             ^%(__prefix_line)sFailed [-/\w]+ for .* from <HOST>(?: port \d*)?(?: ssh\d*)?$
<             ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM <HOST>\s*$
<             ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from <HOST>\s*$
<             ^%(__prefix_line)sUser \S+ from <HOST> not allowed because not listed in AllowUsers$
<             ^%(__prefix_line)sauthentication failure; logname=\S* uid=\S* euid=\S* tty=\S* ruser=\S* rhost=<HOST>(?:\s+user=.*)?\s*$
<             ^%(__prefix_line)srefused connect from \S+ \(<HOST>\)\s*$
<             ^%(__prefix_line)sreverse mapping checking getaddrinfo for .* \[<HOST>\] .* POSSIBLE BREAK-IN ATTEMPT!$
---
> failregex = ^%(__prefix_line)s.+(?:error: PAM: )?[A|a]uthentication (?:failure|error) for .* from <HOST>\s*$
>             ^%(__prefix_line)s.+Did not receive identification string from <HOST> port \d*$
>             ^%(__prefix_line)s.+Failed [-/\w]+ for .* from <HOST>(?: port \d*)?(?: ssh\d*)?$
>             ^%(__prefix_line)s.+ROOT LOGIN REFUSED.* FROM <HOST>\s*$
>             ^%(__prefix_line)s.+[iI](?:llegal|nvalid) user .* from <HOST> port \d*$
>             ^%(__prefix_line)s.+User \S+ from <HOST> not allowed because not listed in AllowUsers$
>             ^%(__prefix_line)s.+authentication failure; logname=\S* uid=\S* euid=\S* tty=\S* ruser=\S* rhost=<HOST>(?:\s+user=.*)?\s*$
>             ^%(__prefix_line)s.+refused connect from \S+ \(<HOST>\)\s*$
>             ^%(__prefix_line)s.+reverse mapping checking getaddrinfo for .* \[<HOST>\] .* POSSIBLE BREAK-IN ATTEMPT!$

[sebres]

Nope. Extending with greedy catch-all (.+) is definitelly not an option, for several reasons.
I'll provide later proper prefix-line replacement for RFC 5424.

[miklosq]

Nope. Extending with greedy catch-all (.+) is definitelly not an option, for several reasons.
I'll provide later proper prefix-line replacement for RFC 5424.

Sure, thanks. That would be the final solution! Mine is just temporary in order to make it work.

[sebres]

There are already few temporary above that work...
A "solution" with catch-all is not a solution at all, the same is possible using unanchored regex, so both following RE's are quasi the same:

- ^%(__prefix_line)s.+Failed [-/\w]+ for .* from <HOST>...$
+ Failed [-/\w]+ for .* from <HOST>...$

At least both are vulnerable exactly the same way.

[churchers]

Hmm, looks like all of this may of been a false alarm, at least for me.
After getting a response from the devs, it looks like an additional reboot was required after installing the 12 upgrade (not needed according to the freebsd-upgrade handbook pages)

After a reboot, the machine I was having the issue on has gone back to the old format.

Jan  4 08:48:53 ftp pure-ftpd[2211]: ([email protected]) [WARNING] Authentication failed for user [test]

The way the changes to syslog were made involved changing the syslog lib function to send 5424 format messages. The daemon was modified to accept these, then write in the required format (old format by default). If your system is using the new library function, but you have the old daemon running, the daemon will get 5424 format messages, and write this as-is to the log.

From https://svnweb.freebsd.org/base/head/UPDATING?r1=332100&r2=332099&pathrev=332100

In addition to supporting RFC 3164 formatted messages, the syslogd(8) service is now capable of parsing RFC 5424 formatted log messages.
.. snip ...
Similarly, the syslog(3) C library function has been altered to send RFC 5424 formatted messages to the local system logging daemon. On systems using syslogd(8), this change should have no negative impact, as long as syslogd(8) and the C library are updated at the same time

This would explain why I was seeing old format messages in maillog. After the kernel upgrade and reboot, I performed the second step of the upgrade to install the rest of the FreeBSD-12 components. At this point I now had installed new library and syslog binaries, but the old daemon was still running. Software that never restarted, like Sendmail, continued to use a previously loaded copy of libc that was logging in the old format. Any application that was restarted picked up the new code and started sending 5424 format messages to an old daemon.

[woodsb02]

* Edit: this comment was posted before I saw the comment above from churchers *

Hi Gavin, I was just discussing this issue with churchers on the freebsd-hackers mailing list, and we established it was an issue where 1 more reboot was required after the upgrade to FreeBSD 12.

The problem being that the old syslogd process was still running, and expecting log messages from /var/run/log[priv] to be in the old RFC3164 format, whilst new processes were running (such as ssh) and logging to syslog(3) which was writing to the log socket in the new RFC5424 format. A restart ensures all processes are using the new libc syslog(3) function and the new syslogd daemon is running.

Can you confirm if this fixes this issue for you?

[miklosq]

There are already few temporary above that work...

Your solution in your first comment did not match lines with the old-only log format:

Dec 28 07:56:18 host sshd[68117]: Did not receive identification string from 196.52.43.110 port 55139

It did only match lines with the new log format:

Dec 25 11:48:18 host 1 2018-12-25T10:16:10.123828+01:00 host sshd 52745 - - Did not receive identification string from 173.255.205.93 port 48916

After reboot, the logs might contain entries from old and new formats if they do not rotate.

A "solution" with catch-all is not a solution at all, the same is possible using unanchored regex, so both following RE's are quasi the same:

It is a solution, indeed: ignoring the failing part. From operational perspective, it is not acceptable to wait for weeks and not having any fix or not even a fix as an interim solution. The logs are keep increasing and fail2ban did not indicate any errors/warnings but silently running as it if was all OK, which is quite bad. In such cases, most folks simply drop such software and begin using alternative solutions.

BTW, the "catch-all" is not literally catch-all as you are trying to indicate. The line pattern has to start with the old date format and then a catch-all. It does catch everything after, but at least it includes both date formats.

[sebres]

Your solution in your first comment did not match lines with the old-only log format

Because it is simply:

1. to rewrite it for both variants
2. to configure it separately per jail

it is not acceptable to wait for weeks and not having any fix or not even a fix as an interim solution.

The same way it is not acceptable to use wrong (potentially vulnerable) solution.
Simply compare "don't catch failure" with "catch wrong IP, injecting from intruder in the log-line" or "catch legitimate IP that don't produce failure".
Let alone "interim solutions" tend to be permanent at some point on some hosts.

In such cases, most folks simply drop such software and begin using alternative solutions.

You can switch to solution you want, but please stop to confuse people with wrong suggestions as regards this solution...

BTW, the "catch-all" is not literally catch-all as you are trying to indicate.

It is. And catch-all is catch-all. Take a look to the tests like this and following:
https://github.com/fail2ban/fail2ban/blob/1c1d2cc435d6e8f1eb4a1b60c935a1385a82e295/fail2ban/tests/files/logs/sshd#L131-L132
All this tests are covering similar vulnerability/injecting scenarios.
If you don't understand why it is so, this is not my problem.

[sebres]

@g-a-c, @fail2ban/contributors someone already tried to solve this with @churchers suggestion https://github.com/fail2ban/fail2ban/issues/2309#issuecomment-451392662 ?
So does reboot (or restart of services) solved that?

Another question I still have, if someone completely switched to RFC 5424 logging, how the log-line looks then? Are both date-patterns and host-names still present?
Or rather the log-message begins with second (long) date pattern only?

[churchers]

As already covered, if you're seeing this on a FreeBSD 12 machine, it's most likely because of an incomplete upgrade. A correct system should either be using the old format, or rfc5424 (but only if you've told it to).

A fully updated 12 system, when configured to use rfc5424 creates logs as below. I just tested this on the 12 system I was originally seeing the issue with, by added -O rfc5424 to the syslogd flags:

<92>1 2019-01-06T18:47:03.770262+00:00 host.name pure-ftpd 9369 - - ([email protected]) [WARNING] Authentication failed for user [sdf]

This appears to be a complete rfc compliant 5424 log message, including the <PRI> figure, which was missing from my original logs.

The match patterns in my first comment do a decent job of matching the 5424 logs, including nested brackets in the data fields, although they were designed for the "half upgraded" log lines. Wouldn't take much to modify that to match the actual rfc5424 format used by FreeBSD though as it's basically just the lack of time & host at the start.

I don't really see the ability to match both formats being a problem. Ignoring the upgrade issue, you should either be seeing one format or the other. If you have some logs in the new format, then I've no doubt fail2ban has a simple way of overriding the prefix for those jails.

[sebres]

Thus I'll close this issue as invalid (mixed rfc5424/non-rfc5424 compliant logging).
We should provide a correct prefix-line RFC 5424 (as well as correct parser for some parameters) in separate PR.

Thank you all for analyse, research and feedback.

[g-a-c]

* Edit: this comment was posted before I saw the comment above from churchers *

Hi Gavin, I was just discussing this issue with churchers on the freebsd-hackers mailing list, and we established it was an issue where 1 more reboot was required after the upgrade to FreeBSD 12.

Can you confirm if this fixes this issue for you?

Sorry, I haven't been keeping up with this thread as well as I could have been (and I'm not on the mailing list myself). I just rebooted again, and /var/log/auth.log has indeed gone back to the original format (or at least a different format to the one I've been having, all the logs from before I did the upgrade have been cycled away). So looks like not doing the extra reboot was my issue. Thanks for the pointer!

[miklosq]

Just for completeness and correctness sake, read below.

Your solution in your first comment did not match lines with the old-only log format

Because it is simply:

1. to rewrite it for both variants
2. to configure it separately per jail

...and that's what most people do not want to do. Neither of 1. or 2. , because it requires extra work effort (i.e. configuration change possibly on a production server) which - in many cases - is not a viable option, or simply not approved (not to mention the fact that the config changes are monitored by security ops in a more stringent environment). Unless you have complete control over your server configuration, this is not feasible. Quick and easy solutions (with prepared patch files) are what people are looking for in the real world.

it is not acceptable to wait for weeks and not having any fix or not even a fix as an interim solution.

The same way it is not acceptable to use wrong (potentially vulnerable) solution.

As I mentioned several times already, I did not purport it be as a full and complete solution, but rather a temporary hack which can be used with both date formats incl. a patch. Temporarily it works and being the regex hacked is not according to your taste, apparently :-)

You can switch to solution you want, but please stop to confuse people with wrong suggestions as regards this solution...

I am not the one who confuses people, but you stating or implying things I never meant.

BTW, the "catch-all" is not literally catch-all as you are trying to indicate.

It is. And catch-all is catch-all.

It is not. Try an empty line without starting it with the old or new date format strings and it will not be matched. The log pattern has to start with a date pattern.

Take a look to the tests like this and following:

Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 port 20000 ssh1: ruser from 1.2.3.4

All this tests are covering similar vulnerability/injecting scenarios.

In this case, it can be injected b/c of not my "catch-all" fix but b/c of how it is. The line you mentioned can be injected with extra rubbish not b/c of my fix but b/c the _original_ failregex from stock fail2ban _bsd-sshd.conf_ contains this:

^%(__prefix_line)sFailed [-/\w]+ for .* from <HOST>(?: port \d*)?(?: ssh\d*)?$

Notice the ending above, question mark before the dollar sign. You can also inject extra rubbish to another one, see below:

^%(__prefix_line)s(?:error: PAM: )?[A|a]uthentication (?:failure|error) for .* from <HOST>\s*$

With results something like below will be matched (without the "catch-all" addition):

Jan 2 17:28:05 host sshd[11226]: error: PAM: Authentication error for root from 218.92.1.140 for admin from 1.2.3.4

Ooops..this is matched on FreeBSD v12 ...and this is the old date format (now restored) even! IS IT STILL VULNERABLE???

However, try to inject extra rubbish to the below and it won't be matched (only digits are honoured):

^%(__prefix_line)s.+Did not receive identification string from <HOST> port \d*$

Try anything after the string 'port 55139' and will not catch it, not even with my "catch-all" addition. In summary, I bet your analysis was wrong and fail2ban still requires a lot of tweaking of failregexes.

On a sidenote, I would not worry about the injections. Fail2ban is only reading the logs gettings its input, not writing them. The software which writes the log entries in the first place has to apply proper input validation checks to disallow malicious log entries.

I wish you better analysis and better bug handling next time.

So does reboot (or restart of services) solved that?

Yes, it did. Date format in the log is restored.

[sebres]

@miklosq

It is not. Try an empty line without starting it with the old or new date format strings and it will not be matched. The log pattern has to start with a date pattern.

Of course it is so, and I did not meant both are really equal. Are you seriously?
I meant only that both are vulnerable the same manner.
Thus again - both are the same in sense of potential vulnerability.

In this case, it can be injected b/c of not my "catch-all" fix but b/c of how it is.

Our "catch-all" covers only foreign user input and because regex is anchored both-sided, it works proper.
Your "catch-all" covers everything after prefix-line, so regular part of logging messages, additionally it breaks the anchoring left, and so allow to parse quasi any other message, therefore this is vulnerable.
Why I should explain you this simply things?!

The simplest example showing what you don't understand all the time:

$ msg='20190101T000000 PREFIX SRV Another message altogether with some foreign user-input (e. g. user input is at end): Failed auth for user from 192.0.2.1' 
$ fail2ban-regex "$msg" '^\s*PREFIX SRV.+Failed [-/\w]+ for .* from <HOST>$'
Lines: 1 lines, 0 ignored, 1 matched, 0 missed
$ fail2ban-regex "$msg" 'Failed [-/\w]+ for .* from <HOST>$'
Lines: 1 lines, 0 ignored, 1 matched, 0 missed

You see? The same happens there, so the prefix line is almost to neglect.
Still questions?

Ooops..this is matched on FreeBSD v12 ...and this is the old date format (now restored) even! IS IT STILL VULNERABLE???

Nope. Because this message contains the host (IP) always at end (note regex was both-side anchored).
Emphasis on this. Totally different message could contain something else. You've broken at least one core principle, which prevent intruder to do this.
Savvy?

On a sidenote, I would not worry about the injections. Fail2ban is only reading the logs gettings its input, not writing them.

• the message fail2ban parse (vulnerable in sense of some parser), can be simply produced by corresponding service (ssh).
• fail2ban works as root and executes some actions (also defined from user).

You want really explain me the basics, how hacking resp. pentest occur?... As I could see, you have no idea at all, so please don't.

[pioto]

I'm seeing this issue with RFC5424 intentionally enabled in my syslog setup. Namely, I have this in /etc/rc.conf:

syslogd_flags="-s -O rfc5424"

This is a fresh FreeBSD 12 system, not an upgrade, where simply making that change seems to have broken things. Am I correct in understanding that fail2ban should be able to parse messages like this?

<38>1 2019-07-08T17:40:16.954167-04:00 hostname.example.com sshd 57915 - - Failed unknown for invalid user redmond from 129.213.97.191 port 59197 ssh2

Or, would support for that be a new feature request/PR?

[sebres]

@pioto if your fail2ban version is 0.10, you can simply provide it as a datepattern (and prefix-line) in your jail.local for jail (or default section):

[sshd]
datepattern = ^<\d+>\d+\s+{DATE}
filter = sshd[__prefix_line="^\s*<__hostname> <__daemon_re> \d+ \S+ \S+\s+"]

PoC:

msg='<38>1 2019-07-08T17:40:16.954167-04:00 hostname.example.com sshd 57915 - - Failed unknown for invalid user redmond from 192.0.2.1 port 59197 ssh2'
fail2ban-regex -d '^<\d+>\d+\s+{DATE}' "$msg" 'sshd[__prefix_line="^\s*<__hostname> <__daemon_re> \d+ \S+ \S+\s+"]'

Or, would support for that be a new feature request/PR?

Normally it is a matter of @fail2ban/maintainers, but we can indeed help them to support RFC 5424, e. g. to extend fail2ban with something like #2387, so as new logtype for this RFC. So theoretically it could be enough to write:

filter = sshd[logtype=rfc5424]

Just unfortunately the datepattern of some filters may be affected because they are anchored at line begin at the moment.

[pioto]

Thanks, @sebres, I can confirm the suggested change to jail.local for the sshd jail is logging matches in fail2ban.log again.

[sebres]

no sooner said than done...

There is new PR #2467 introducing my idea with new formatting filter-option logtype=rfc5424...

@fail2ban/maintainers, @chtheis Dunno it would help, but possibly you could use this option (e. g. to rewrite it in some includes) in order to provide support for RFC 5424 format for your distro's for some services or per default (if all is affected).
Don't hesitate to contact me for examples or enhancements requests may be needed (e. g. in jail.conf or paths-common.conf, etc).