Pfelk: Opnsense 19.7.3 Missing fields in index pattern

[mango1122]

I've got ELK installed and pulling logs from Opnsense 19.7.3. The issue if that I'm only seeing 20 fields in the index pattern. I suppose it is due to the grok filter but I don't seem to find the issue.
Any help is appreciated. Thank you !

Attached are the logs from Logstash. logstash-plain.log

dp-ubuntu@ubuntu:/etc/logstash/conf.d$ ls
01-inputs.conf 10-syslog.conf 11-pf.conf 30-outputs.conf patterns

dp-ubuntu@ubuntu:/etc/logstash/conf.d$ cat 11-pf.conf

filter { if "pf" in [tags] { grok { add_tag => [ "firewall" ] match => [ "message", "<(?<evtid>.*)>(?<datetime>(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\s+(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:[0-5][0-9])) (?<prog>.*?): (?<msg>.*)" ] } mutate { gsub => ["datetime"," "," "] } date { match => [ "datetime", "MMM dd HH:mm:ss" ] timezone => "Asia/Singapore"

P.S - I've set the timezone correctly and also the IP of the firewall

[a3ilson]

Mango1122,

Your setup appears to be correct. I am troubleshooting the issue when time permits (work). The latest version of OPNsense (v19.7.3) does not parse out correctly. As for the issue, the GROK pattern is configured correctly - I've analyzed the packets (Moloch/Wireshark) and the only modification from pfSense and OPNsesne (prior to v19.7) is the addition of the host name in the "msg". Your sample screenshot depicts this where "OPNsense.local" and "filterlog" are both displayed within the "prog" tag. I have time allocated later this month to coorect the issue.

Thanks,
Andrew

[mango1122]

Thanks a ton. Appreciate you pulling in time to work on this.

[a3ilson]

Fixed! Please update your 11-pf.conf file; I made changes to address the added hostname within OPNsense 19.7+

I also updated the GROK filter to accommodate OPENVPN. When you update your 11-pf.conf file you'll need to update the pattern file too.