Beats: [Filebeat] Module for Palo Alto Logs

[andrewkroh]

As a user I want to be able to ingest logs from Palo Alto PAN-OS. Of the available log types I'm thinking to start with:

• Traffic Logs
• Threat Logs

We'll need to map the fields in these logs to ECS.

[MikePaquette]

Here are some WIP logical mappings (no config files) of PAN log fields to ECS fields

Traffic Logs - Google Sheet
Threat Logs - Google Sheet

Also, there are some sample log snippets in this Google Team Drive folder. However, most of these are in CEF or LEEF format. I have requested samples of pan_traffic and pan_threat in syslog format and PAN has agreed to find or create some additional samples for us after Thanksgiving.

These samples appear to use only RFC 1918 addresses and public addresses from entities like Google and Akamai, so we can use these as samples.

[elasticmachine]

Pinging @elastic/secops

[Rukas]

Hello,

We're interested in this as well. Has there been any movement on this?

[MikePaquette]

Sorry for the trouble with those Google Docs.
Please see https://github.com/elastic/ecs/issues/352 for these examples.

@Rukas @andrewkroh

[willemdh]

Just wondering why a Filebeat module is preferred over a Logstash module? @andrewkroh? It seems so much more work to do this with Filebeat? A kv filter for traffic, threat, userid, system and correlation and you are set.. Just another small note, the syslog looks different depending on the Panos version. Recently upgraded to 8.1 and we had some mapping issues, as some new fields were added, check https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/threat-log-fields.html and compare fields between versions.

[davidhowell-tx]

In reviewing the code for the panw filebeat module I noticed that certain fields are duplicated, such as client.ip, source.ip, client.address. Is there a reason why duplicating the data is preferred instead of using an alias?

[learnitall]

I'm not sure if this answers your question @davidhowell-tx, but I don't believe those fields are meant to be exact duplicates in the purest sense. From the ECS client field schema:

Some event sources are defined ambiguously. The even will sometimes list an IP, a domain or a unix socket. You should always store the raw address in the .address field. Then it should be duplicated to .ip or .domain, depending on which on it is.

It's my understanding that if you added aliases for client.ip and client.domain that pointed to client.address, all data placed into the client.address field could then technically be interpreted as both an ip and a domain. By only duplicating the data to the appropriate field when needed, the interpretation of the data becomes clearer, as I don't think you can set aliases per-event.

I think this same line of reasoning applies when talking about the difference between client/source and server/destination.

[learnitall]

Also I'd like to add that I think it might be a good idea to explore the ability to specify the Panos version in the Beats config, and then add support for a wide range of Panos versions. This way regardless of the version of Panos a user has they can still use the module with full compatibility. Plus we'd have a defined way to update the module in the future when a new version of Panos comes out that uses an altered schema.

[strawgate]

We would very much like to see support added for the remainder of palo alto logs

[andrewkroh]

Closed by #11999.

Module docs: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-panw.html

[andrewkroh]

@strawgate For the other logs types that you are have interest in being parsed by the module please open a new issue.