Ecs: [1.5] IP fields missing array definitions

[codebrain]

Array definitions are missing for:

• client.ip
• destination.ip
• server.ip
• source.ip

...the schema indicates that these fields "Can be one or multiple IPv4 or IPv6 addresses." but is missing the normalize->array definition.

Relates to: https://github.com/elastic/ecs-dotnet/issues/64

[webmat]

While host.ip is without a doubt meant to capture an array of IPs, these IP fields are not. They are really meant to capture a single IP on each end of a network exchange. So just like user.id #777, here it's the definition we should adjust.

Known cases where a network exchange have more than a pair of IP addresses are:

• Protocols like Bittorrent that have many potential addresses for the source of a transfer. This is different semantics, so not applicable here. This would have to be captured via custom fields.
• A single gateway or proxy's IP address is in the middle of the transfer and should be captured via e.g. source.nat.ip. Resulting in:
  
  
  
  • "real" source IP at source.ip
  
  
  • gateway or proxy IP at source.nat.ip
  
  
  • destination IP at destination.ip
  
  
• multiple levels of proxying (e.g. CDN + multiple load balancers): ECS doesn't have an answer for that now, and would need to be captured via custom fields.

So here we'll adjust the field descriptions instead.

Thanks for raising that, Stuart :-)

[webmat]

@dainperkins you weren't there when we discussed this, but do you agree? ^

also @MikePaquette @dcode @simitt feel free to correct me here.

[webmat]

Also @ruflin do you agree with this https://github.com/elastic/ecs/issues/786#issuecomment-604073952?

Do you recall a specific use case why that wording was there?

[dainperkins]

Multicast would be the only thing I can think of that might require arrays on source/destination
(actually theres anycast in ipv6 too, suddenly that surprise ipv6 section on the ccda 20 years ago is relevant :)

Bitorrent's actually a large # of 1:1 connections...

Do we need to support it today? What does a capture even look like? not really sure, have to take a look (I hate multicast)

[dainperkins]

sorry - fat fingers closed by mistake

[MikePaquette]

@webmat LGTM.
@dainperkins I think even multicast just uses a multicast destination.ip, not multiple ip's, right?

[webmat]

Ok I'll merge #804 for now. We have a few weeks to address if important reservations are raised about this before then.

But I think use cases that have semantics that require an array of IPs would require different fields better adapted for that.

[neu5ron]

zeek files.log uses an array for src/dst.
I know the following may not be src/dst or client/server, but keeping these fields as an array could help with other IP fields that are arrays like dns answers ip ECS field.
Once tackling SMTP fields will be another.
Also, for certain things that do name resolution for a log which could get mapped to src/dst - it may return an array.

[dainperkins]

good points @neu5ron, thinking we'd be ok leveraging arrays for dns/smtp (whats the use case for array of ips in zeek?)

I checked a pcap @MikePaquette, multicast just sends to the multicast address, and each client gets the data based on joining a multicast group iirc. (handy dandy examples here: https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=rtp-norm-transfer.pcap)

I still hate multicast

[neu5ron]

zeek files.log tracks all the src/dst in the transmission of that file. like a file over ftp, will add the IPs seen from the other log sources that it saw.
which reminds me, another potential I could see is for alerting/grouping/correlation rules - like an example of where source IP performs some action to multi destination IPs

multicast makes everyone 🤔

[dcode]

Multicast is still source/destination with a multicast IP. The files.log from zeek does use an array for tx_hosts and rx_hosts. This log does not model a network connection per se, as theres no port or transport information in that file. Also, as much as I was sad to discover, Zeek doesn't currently have any supported protocols that populate those arrays. The BitTorrent, Gnutella, and other protocols that Zeek used to support are legacy and no longer analyzed. :sadpanda:

After all that and to have consistent modeling, I think single entity soure/destination makes the most sense. Which imply that we expect to analyze network connections on an individual session basis.