Ecs: SIP package

[jiriatipteldotorg]

Folks,

may I propose including a SIP package (RFC3261). I suggest it would include the fields as bellow, modelled after the HTTP package. What's different:

• SIP requests have many more URLs in them. Therefore the suggestions Use the url field set to store the url of the request. only applies to the request-URI which represents the URI in the request as received (SIP proxy elements may also alter it during forwarding). For all the other URIs, I included only the unparsed form full.
• other fields that offer themselves for use in SIP events are event.duration for reporting SIP call length, user_agent.* for SIP User Agent (example: AVM FRITZ!Box Fon WLAN 7270 v3 74.06.05 (Apr 9 2014), host.name for receiving SIP server's DNS name, host.id for SIP server's unique identification, host.ip for receiving server's IP address, client.ip and client.port for client's IP:port, user.id for digest authentication identity, client.geo.* for client's geographic info.
• client.transport would be useful but doesn't exist (#418 )
• event.action seems the proper candidate for values describing the actual nature of an event like call-attempt(failed INVITE request), call-start (succesful INVITE request), call-stop (received or initated BYE), register-new for a newly added contact binding, register-del for removed contact binding, register-expired for one that has expired.
• in SIP, response elements often mirror their request counterparts. Where identical for requests and response, I didn't prefix the field name with request/response -- I hope that's the proper use of the ECS conventions.

There are couple of fields which would be helpful too but I don't see them as SIP specific -- see the list bellow, merely for sake of context, if there is interest I can create separate issues for them.

• session id. It is often useful to correlate some events belonging to each other -- for example a SIP call's id may be used to correlate events describing start and stop of a VoIP call, or a shopping session id may be used to correlate additions to and removals from a shopping basket.
• for use with SIP proxy elements it would be useful to describe in events how such a proxy element describes changes to forwarded requests: most importantly how it is changing the URI, where it is forwarding a request to, and if a response was generated locally or proxied. However this doesn't seem SIP-specific, I think the same data would be useful for example for HTTP proxy servers like nginx or ha-proxy ,
• some topological information. Often, especially in multi-tenant environments, flows are correlated with their actual owners -- that helps in many aggregations. I havent' exactly captured the difference between source and client packages but I would say that for sake of tenant analysis a more "logical" view would help. For example it could use UUID to describe traffic from multiple parts of an organization (liberland's embassy to the US, liberland's embassy to Australia, etc)

Suggestions / clarification requests welcome.

  - name: sip
    title: SIP
    group: 2
    description: >
      Fields related to SIP activity.   Use the `url` field to store SIP request URI.
    type: group
    fields: 

   # HTTP-like SIP fields

    - name: method
      level: extended
      type: keyword
      short: SIP  request/response method.
      description: >
        SIP request/response method.

        The field value must be normalized to uppercase for querying.
      example: INVITE, BYE, REGISTER, SUBSCRIBE, NOTIFY

    - name: request.body.content
      level: extended
      type: keyword
      description: >
        The full SIP request body, typically SDP.
      example: See RFC3264 for SIP/SDP examples.

    - name: response.status_code
      level: extended
      type: long
      description: >
        SIP response status code.
      example: 404

    - name: response.body.content
      level: extended
      type: keyword
      description: >
        The full SIP  response body.
      example: See RFC3264 for SIP/SDP examples.

   # SIP-specific fields

    - name: call_id
      level: extended
      type: keyword
      short: SIP Call ID
      description: >
        SIP call id is used to relate dialog-initiating and in-dialog SIP transactions with each other.
        The value is case-sensitive. 
      example: 11e6585d-6017-4f83-a95b-4861d47fe678

    - name: request.contact
      level: extended
      type: keyword
      short: Full unparsed Contact URL.
      description: >
        This is the full Contact URI most often found in SIP REGISTER and INVITE requests.
         Note that it is NOT a full Contact header value which may additionally contain header field
         parameters, such as expires=3600.
      example: sip:[email protected]:5080;transport=udp;gw=d15d5fa6-6868-4b45-84f9-851db9d54ca3

    - name: from
      level: extended
      type: keyword
      short: Full unparsed From  URL.
      description: >
        This is the full From URI mandatory for any SIP requests. 
         Note that it is NOT a full From header value which may additionally contain header field
         parameters, such as tag=tpc6D8ce0NccD.
      example: sips:[email protected]

    - name: request.to
      level: extended
      type: keyword
      short: Full unparsed To  URL.
      description: >
        This is the full To URI mandatory for any SIP requests. 
         Note that it is NOT a full To header value which may additionally contain header field
         parameters, such as tag=tpc6D8ce0NccD.
         Also note that the RFC3261 permits the To URI to be different in response.
      example: sips:[email protected]

    - name: response.to
      level: extended
      type: keyword
      short: Full unparsed To  URL.
      description: >
        This is the full To URI mandatory for any SIP response. 
         Note that it is NOT a full To header value which may additionally contain header field
         parameters, such as tag=tpc6D8ce0NccD.
         Also note that the RFC3261 permits the To URI to be different than in the initiating request.
      example: sips:[email protected]

    - name: response.reason_phrase
      level: extended
      type: text
      short: human-readable form of status code
      description: >
         human-readable form of SIP status code, see RFC3261, section 7.2

    # Metrics
    - name: request.bytes
      level: extended
      type: long
      format: bytes
      description: >
        Total size in bytes of the request (body and headers).
      example: 1437

    - name: request.body.bytes
      level: extended
      type: long
      format: bytes
      description: >
        Size in bytes of the request body.
      example: 887

    - name: response.bytes
      level: extended
      type: long
      format: bytes
      description: >
        Total size in bytes of the response (body and headers).
      example: 1437

    - name: response.body.bytes
      level: extended
      type: long
      format: bytes
      description: >
        Size in bytes of the response body.
      example: 887

[webmat]

Thanks a lot for this very detailed proposal.

ECS is still in its infancy at this time, and still missing very basic protocols such as DNS and TLS. We need to address the basics first :-) So it's unlikely we'll be focusing on SIP in the near term.

However ECS is a permissive schema. You're welcome to add custom fields to your events that would suit your needs. When doing so, it's also a good idea to nest existing ECS field sets in your custom fields. This will ensure your events have the same consistency and feel as the rest of ECS. This is good for your users :-)

What I describe above is also how we hope new additions to ECS will come about. New use cases get worked out in the real world, in a real scenario. The issues encountered are fixed, and then the proposal for addition to ECS is more robust and less likely to fluctuate afterwards.

Now a few suggestions about the implementation of the use case you're describing:

• The counterpart to "client" should be "server", not "host". "host" is meant to describe the host where the event is observed. It's meta-data about the activity, rather than details describing the activity, if you will.
• I'm curious to understand more about client.transport. How would it be used? Would its counterpart server.transport also be populated? What are the possibilities / permutations?
• We will be providing more guidance on the acceptable content of a few fields in the future. event.action is one of them. For now SIP-specific actions should ideally be stored in a custom field. Once we've published acceptable values for event.action, you should be able to map the SIP-specific action to it.

[ebeahan]

@dainperkins do you have any SIP/VoIP experience? We're looking for some SME feedback on these field mappings.

[dainperkins]

@ebeahan I do, will take a look.

[ebeahan]

Related Packetbeat PR: https://github.com/elastic/beats/pull/21221

[ebeahan]

I ended up digging in a bit as well, but my VoIP knowledge is a bit rusty :joy:

I agree that the fields detailed in the original proposal would be a good starting point for modeling SIP:

• sip.method
• sip.request.body.content
• sip.response.status_code
• sip.response.body.content
• sip.call_id
• sip.request.contact
• sip.request.from**
• sip.request.to**
• sip.response.from**
• sip.response.to**

** The original proposal includes a single sip.from field vs. having distinct sip.request.to and sip.response.to fields. Would we want from and to fields for both request and response?

With sip.response.reason_phrase, there are a handful of similar headers that seem worthy of consideration. However, I'm not familiar enough to weigh in on their usefulness or frequency in real-world use.

• sip.request.subject
• sip.response.status
• sip.response.warning

[dainperkins]

Hah - I've been actively trying to forget everything I ever had to learn about skinny, sip, and everything involved.

I started with the Zeek SIP fields (https://docs.zeek.org/en/current/scripts/base/protocols/sip/main.zeek.html) and then moved onto wireshark (https://www.wireshark.org/docs/dfref/s/sip.html) and then over here: (https://support.yeastar.com/hc/en-us/articles/360007606533-How-to-Analyze-SIP-Calls-in-Wireshark)

I'm inclined to dig thru the 3 pages of wireshark stuff, do it once and be done. Even if we don't use everything with e.g packetbeat, it seems like just getting it done would be the easiest for ECS (particularly as my SIP troubleshooting is fairly limited). Adding MOS, Jitter, dscp, ip precedence, etc. (working on it for network) would also be key for SIP analysis / troubleshooting / performance management

Side note - the only thing I miss from Cisco Unified Messaging is jabber global federation :)

[dainperkins]

Yikes - thats 274 sip fields from Wireshark, I'd like to take back my previous answer :). quick re-review suggests zeek & the above were pretty aligned...

• IIRC the request from/to and response from/to are used quite a bit in SIP gateways, as well as handoffs from e.g. call manager - so they would probably be useful to separate out.

• in looking at the troubleshooting and remembering qos settings for various wan problems, some basic rtp fields would be a pretty natural extension as well to go beyond call control and into actual call analysis...

[dainperkins]

I started digging thru some example captures and think we should take this back 1 step and start with which of the sip methods we want to implement - register, invite, & bye seem like good starts (and I think most, if not all, of the fields I'd be looking for are covered in the above - I'll try and map out the captures this weekend, and thinking about any RTP specifics outside of network fields that might be necessary.

I'm thinking that we might want to define auth / sessions iff the sessions are user based (as opposed to e.g. phones registering with a number or mac address). I;m thinking given the specific nature of the sip events it might be better to keep a reason field specific to sip (normal call clearing, and all the other sip/call specifics, unless we really want to dig into details in the event fields

[marc-gr]

Thanks @dainperkins ! will wait for your recommendation before doing any further changes in elastic/beats#21221 then.

[dainperkins]

@marc-gr
I went thru a handful of wireshark captures for REGISTER, AUTHORIZATION, INVITE... and came up with the following
https://docs.google.com/spreadsheets/d/1FjbLTqOfnXPbMXZZI4Ms6hFGGnkbjMaEUXle2KpbjAc/edit?usp=sharing

It seems like splitting up request response as separate nested fields seems unnecessary - the basics are shared, and with a header (e..g sip.type) its easier to normalize both (requests and responses) with no loss of reporting, etc.

As far as I can tell most of the ids and text is included in the packet (e.g. 401 unauthorized)

Theres probably some settings I haven't picked up on from e.g. ack, message, etc, have to go thru some more captures.

I bolded the things I can remember using to troubleshoot various sip issues, and tried to mark where I was unsure if full normalization would be warranted (e.g. uri down to host, subdomain, registered domain, etc) In thinking about call reporting, and long distance fraud, etc, it seems like many of these would be useful for various buckets in reporting, ml jobs, etc.

[dainperkins]

I went thru a request and 2 responses and came up with what seems like will cover a good bit, there's a couple that might be optional (normalizing uri's & contacts for instance, nonce, and opaque values for auth requests). I didn't really try to match field names, tho the ones that overlap (other than not prefixing each request or response header field and allowing the top level identifier to provide the separation - e.g. to/from/contact - its all more or less the same fields, and easy enough to search for sip.method = register && sip.from.user = John.Doe instead of specifying sip.request.from or sip.response.from...

Sample Packet Capture & ECS implementation in the sheet in this directory:
https://drive.google.com/drive/folders/1yebz9dGyUiQGYOzVSi_iCdE4NuT8ZkzK?usp=sharing

I suspect logs will require significantly more analysis / intelligence in terms of integration, but it should work as well.

Will add a bunch of SIP responses (register failure for auth, proxy auth for invite, etc.) as well as start refining the original invites tonight or tomorrow.

[marc-gr]

awesome @dainperkins will do the changes for packetbeat and see if we can get it in for 7.10 👍

[dainperkins]

Had a call with @jamiehynds and a couple of other SA's today. Final revision on suggested "experiemental" fields can be found here:

https://docs.google.com/spreadsheets/d/1XRHHio_C-yVwc8BARVRhMC5YjU4NK4E8LuznuE2DxYA/edit#gid=398944674

Revised SIP Fields Tab: list of SIP/SD Fields
Breakdown Tabs: started reworking the breakdown of various captures with the fields on the first tab.
Added some thoughts on logic, bytes, etc. here and there.
Will look at working SIP into Sessions as well

@marc-gr if theres anything I can do from a packetbeat perspective to help implement (I know I've went back and forth a few times) let me know.

[ebeahan]

With the SIP RFC proposal PR opened (thanks again @dainperkins!), any objections to closing this issue in favor of any continuing discussion happening in https://github.com/elastic/ecs/pull/1014?