Beats: New protocol support: SIP

[tsg]

Split from #22.

[kputland]

+1

[johntdyer]

:+1:

[marcelog]

+1

[sinkovsky]

+1

[donckers]

+1

[majofi]

+1

[crickes]

+1 for me too

[lukebeer]

I know SIP & RTP very well but not Go. If someone is willing to team up & be on IM to lend assistance with the language. I'm game.

[tsg]

Hi, @lukebeer, I'd be happy to pair program on that. Shoot me an email at [email protected]. Btw, we now have a new protocol dev guide: https://www.elastic.co/guide/en/beats/packetbeat/current/_developer_guide_adding_a_new_protocol.html

[crickes]

Hi, I thought about having a crack at this also but as the majority of SIP clients and all RTP use UDP instead of TCP, don't we have to wait until UDP is supported by PacketBeat?

[tsg]

@crickes UDP is supported now :-). See the DNS protocol implementation for an example.

[lukebeer]

Just to tac on, depending on the network topology, particularly when the
UAs are within the same network, SDP will typically negotiate a direct
media path between the endpoints involved separately to SIP signalling so
capturing the RTP stream won't be possible without anchoring the media
through a server, SBC of some kind or span port switch configuration.

If media is available to the monitor, the best way to store RTP detail
will be with the RTCP reports generated by each end point, if supported.

In short, a SIP module is a great starting point for VoIP support so let's
get this nailed and worry about RTP after. Unless of course a volunteer is
willing :)

Hi, I thought about having a crack at this also but as the majority of SIP
clients and all RTP use UDP instead of TCP, don't we have to wait until UDP
is supported by PacketBeat?

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/packetbeat/issues/152#issuecomment-138465383.

[tsg]

@lukebeer +1, we should start with SIP. I think @crickes meant that SIP is also often run over UDP.

[lukebeer]

Agreed and this brings up a good point, how to track calls - TCP streams
give us the opportunity to just follow the id however with UDP it'll be
call id/branches/cseq's.

That said, should calls be correlated or just treat each message as an
individual? Thoughts?

Perhaps it's time to turn to the RFC....

[email protected]

On 8 September 2015 at 17:48, Tudor Golubenco [email protected]
wrote:

@lukebeer https://github.com/lukebeer +1, we should start with SIP. I
think @crickes https://github.com/crickes meant that SIP is also often
run over UDP.

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/packetbeat/issues/152#issuecomment-138629825.

[crickes]

Hi,

With regards to RTP, they are just flows like anything else. The issue gets more interesting if you want to analyse quality of service. RTCP does give some metrics regarding packets expected and received to give a measure of packet loss, but the extended receiver report has a lot of other QoS metrics in it, but not all endpoints support this.

With @lukebeer comment about tracking SIP calls. In my opinion, it would be better to track SIP dialogues, rather then complete calls. A call can last anywhere from a few seconds to several hours, and tracking the call states purely in the probe is potentially very tricky and very memory intensive. It would be similar to tracking a users visit to a website based on their browser session ID. Perhaps a better approach is to track the individual dialogues, Request to Response but make a note of the Call ID, just as you would with a session ID in tracking a HTTP GET. You could then use the power of Elasticsearch to create an entity centric (Call / session) index to see the call states, rather than trying to track them in the packetbeat probe.

I'm not sure how PacketBeat works as yet, I've just started looking at it, but if you were to wait until the call was completed before it updates ES with the call details, you could potentially be waiting a long time (the call duration) before you were aware of the call at all. Tracking the initial INVITE to response, makes you aware of the call attempt immediately and then based on the response you can infer whether the call is in progress or not. You would then add each additional dialogue to the database using the call-ID to relate the dialogues to the call.

[crickes]

A bit of background information, my colleague and I have already tested sending individual SIP packets into ES and have been able to see rates of message types, ie, 500 INVITES / Sec.. The scale of the solution we are creating makes this individual packet counting technique unscalable (several 10,000s on packets per second being sent to an ES cluster). Using packetbeat to track the dialogues would cut down on the number of inserts into the database and also provide some extra metrics like time to respond, INVITE -> 100 (server response time), or INVITE -> 18x (Post dial delay) for example.

I personally have never heard of 'GO' before reading this thread, but I am very familiar with SIP and RTP, and willing to learn 'GO' and help in the development or at least testing of this module. I work for one of the UK's largest providers of SIP trunks and I can push LOTS of real SIP traffic into a test box and see how it performs under heavy load.

[lukebeer]

@crickes - perfect. Track the dialogs then correlate later, hopefully it'll stand up to the throughput. My concern was specifically how we'd deal with long duration calls. To be honest, it didn't occur to me to just track the dialog, that's a much better approach. Input appreciated.

Also - what a coincidence! I've just signed the papers and will be joining your workforce on the 22nd of October, greetings!

[lukebeer]

Had a thought - a while back I needed a way of handling an insane transaction rate without loss. I ended up using libnfqueue to distribute the traffic amongst multiple nodes running an open source message broker. Each node in this pool acted as a buffer by replaying the packets at a rate the destination could withstand. We could use the same approach here if, when required.

[crickes]

@lukebeer Look forward to working with you on this and potentially other things then! Look me up on linked in.

[urso]

Just some input:

Currently packetbeat has no support for SCTP. Check which transport protocols you will need for your use case. If you can do without SCTP just go ahead.

For how long do you want track the dialog? When and how often do you want to publish dialog states? A dialog is identified by SIP call id, from tag and to tag. For transactions use the cseq in addition. When working on dialogs be aware that transaction (different cseq numbers) might be active at about the same time...

About SIP call leg (hop-by-hop): People sometimes mix TCP and UDP on the very same call leg. Don't trust port numbers, transport protocol, IPs, Mac addresses... there are various combinations of how ethernet, ip, vlan, sip call id are used to make up a leg. Be flexible in configuring the correlation of messages into transactions and or call legs. When handling transactions be prepared to see request via TCP and response via UDP (just saying). Also be aware of Non-RR.

At the moment packetbeat is supposed to be mostly single-threaded. For now you have to solve load-balancing somewhat yourself.

You might want to treat packetbeat mostly as a probe.Consider concentrating on publishing transactions first instead of building a full state machine on dialogs. Transactions are relatively short-lived and you won't tax memory usage that much. Having transactions you will already get registration support. Building call leg state machines based on transactions might simplify the task of handling message reorderings and other weird stuff.

For building up dialogs maybe you can do it in ES based on transactions, maybe some logstash plugin collecting transactions from multiple probes. Plus how often/when do you want to publish call state updates?

About RTP/RTCP: you trust RTCP reports? Or do you plan on collecting similar stats for RTP.

[kputland]

Some thoughts,

• Transactions are better to have sooner rather than later. Think watcher
  based fraud detection routines that can run on say receipt of some
  INVITE/second to high risk destinations.
• Call legs and dialogs can be correlated later
• Correlation can be tricky across SBCs if the SBCs doing topology hiding
  or other HMR that changes callid, or tags.

--Karl

Karl Putland
Senior Engineer
_SimpleSignal_
Anywhere: 303-242-8608
http://www.simplesignal.com/explainer_video.php

On Tue, Sep 8, 2015 at 4:06 PM, Steffen Siering [email protected]
wrote:

Just some input:

Currently packetbeat had no support for SCTP. Check which transport
protocols you will need for your use case. If you can do without SCTP just
go ahead.

For how long do you want track the dialog? When and how often do you want
to publish dialog states? A dialog is identified by SIP call id, from tag
and to tag. For transactions use the cseq in addition. When working on
dialogs be aware that transaction (different cseq numbers) might be active
at about the same time...

About SIP call leg (hop-by-hop): People sometimes mix TCP and UDP on the
very same call leg. Don't trust port numbers, transport protocol, IPs, Mac
addresses... there are various combinations of how ethernet, ip, vlan, sip
call id are used to make up a leg. Be flexible in configuring the
correlation of messages into transactions and or call legs. When handling
transactions be prepared to see request via TCP and response via UDP (just
saying). Also be aware of Non-RR.

At the moment packetbeat is supposed to be mostly single-threaded. For now
you have to solve load-balancing somewhat yourself.

You might want to treat packetbeat mostly as a probe.Consider
concentrating on publishing transactions first instead of building a full
state machine on dialogs. Transactions are relatively short-lived and you
won't tax memory usage that much. Having transactions you will already get
registration support. Building call leg state machines based on
transactions might simplify the task of handling message reorderings and
other weird stuff.

For building up dialogs maybe you can do it in ES based on transactions,
maybe some logstash plugin collecting transactions from multiple probes.
Plus how often/when do you want to publish call state updates?

About RTP/RTCP: you trust RTCP reports? Or do you plan on collecting
similar stats for RTP.

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/packetbeat/issues/152#issuecomment-138717626.

[claygorman]

+1

[howken]

+1

[effeks]

+1

[ESamir]

+1

[damianpfister]

+1

[srpatatas]

+1

[yanou75pro]

+1

[fcastello]

Any progress in adding SIP to packetbeat?
Is there a way I can test what you were working?

[darrenliddell]

+1

[chaunbot]

+1

[Jayranbutong]

so ,how is going

[Haphil]

+1

[sfrique]

+1

[ekoome]

Latest on this?

[kisienya]

+1

[bigtoerag]

@gerardseeto has forked packetbeat and has some good ideas on a way forward but needs help with learning GOlang.

Can anyone with GO skills contact him and see if its easy to progress?

[kisienya]

Try these resources:

https://tour.golang.org/welcome/1
https://gobyexample.com/

Thanks

[kisienya]

One more...https://www.golang-book.com/

[tintoy]

Hi @bigtoerag.

I work for the same organisation as @gerardseeto and will have a chat to him to see if I can help (my Go skills aren't particularly l33t but they're serviceable).

[fatihrd]

+1

[jatsmulator]

We already implement SIP and MSRP protocol plug-ins with Packetbeats, SIP and MSRP packets can be recovered and shown in ELK.

[bigtoerag]

Not sure what you mean as I checked and its not listed yet.
https://www.elastic.co/products/beats/packetbeat

[monicasarbu]

@jatsmulator Happy to hear you already added support for SIP and MSRP in Packetbeat. Would be great if you can share it with the community and create a PR upstream.

[kisienya]

@jatsmulator Where can we get this since its not listed on https://www.elastic.co/products/beats/packetbeat

[jatsmulator]

Hi Kisienya:

Happy new year!
Yes, I am starting create a PR, and add SIP and MSRP support into
Packetbeat.
Please wait for a while.

2017-02-06 16:18 GMT+08:00 kisienya [e-mail]:

@jatsmulator https://github.com/jatsmulator Where can we get this since
its not listed on https://www.elastic.co/products/beats/packetbeat

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/beats/issues/152#issuecomment-277611836, or mute
the thread
https://github.com/notifications/unsubscribe-auth/APOgdeKFRigsQGtnCnN1-GUAdBeXclRsks5rZtdugaJpZM4E01dj
.

[kisienya]

@jatsmulator many thanks for that...

[janprabu]

Hi,

Could you share that plugin code. We wants to try that.

Thanks in advance.

[tsg]

@jatsmulator Hey, any progress on that? Feel free to open a PR even if not all the functionality is in there, we are glad to help with that.

[smokersonly]

any estimate when this would be available?

[luiscarvalho]

Hi!
Any good news about the Packetbeat SIP protocol? I have a pcap SIP source that i would like to index on Elasticsearch.

Thanks

[gauravsaini03-zz]

@jatsmulator Any updates about SIP support? :)

[tj8000rpm]

SIP/2.0 100 Trying

Hi all,
I trying implement the packetbeat SIP protocol.(I could not wait someone implement that.... :sweat: )
https://github.com/tj8000rpm/beats/tree/sip_protocol/packetbeat/protos/sip

In this implementation is not managed the dialog and transaction, because I think that it is little difficult with packetbeat. Is there no problem?
If it is Okay, I am going to send PR after I write more tests and translate the japanese comments.

I am beginner in Golang , github and the open source community...(oh forgot, English too! :sweat_smile: ) , so please give me a little time!

regards

[urso]

SIP can be quite tricky. Especially when it comes to correlating responses to request. Features like call forking don't make it easier to handle (in the wild response might even be send with a very different transport protocol, yay) :)

A request/response can take multiple hops. That is you might also want to define 'legs/hops' when correlating (or just be lazy and use the Call-ID). This protocol is full of edge cases.

Have a look at the packetbeat protocol generator. The generator already adds some common structure to how transactions are analyzed in packetbeat + adds correlation tables (to be enhanced) for correlating requests and responses.

Tip: use goimports to format your source code (can be configured by most editors).

Feel free to open a PR for further discussions on the implementation.

[npenna22]

Hi,
I would like to deploy the packet bit SIP protocol ressources shared on https://github.com/tj8000rpm/beats/tree/sip_protocol/packetbeat/protos/sip.
I'm a newbie in Packetbeat, as in GO and I search for a method to declare such protocols on my node. I have looked on doc pages on Elastic site (Beats Developer Guide [master] » Adding a New Protocol to Packetbeat » Protocol Modules, version 6.2.3) but the Elatsic team indicates that this topic is under construction.
Is there someone who konws the method to deploy those SIP protocol ressources for 6.2.3 Elastic solution please?
Best regards

[tj8000rpm]

Hi, npenna22
Thank you for trying my sip plugin.
Packetbeat is single binary application, so if you want to deploy and use new protocol plugin, you should replace the current(6.2.3) packetbeat binary/setting file to new binary and setting file(my repository is version 7.0.0-alpha1).
_*sample setting is included in my repository._

How to get a new packetbeat binary
Usually you need recompile with new source files yourself. You be clone the github repository and cd to packetbeat directory, then go build .. If you have no problem you can get a new binary on your current directory. (Of course, you should install go compiler in your pc/server.)
However, if you want to run a packetbeat on linux 64bits architecture, you can download compiled binary file form below link(I compiled on Ubuntu 16.04, go version go1.9.2 linux/amd64).
https://github.com/tj8000rpm/beats/blob/sip_protocol/packetbeat/packetbeat.linux_amd64
__*this is NOT official version, so please try under your own risk.__

Thanks.

[npenna22]

Hello,

First of all I would like to thank you for the information you sent me and I wish to apologize for the time taken to answer you. I cannot spend a lot of time on this task right now but if it’s working it could be the case in the future.

Now, as I’m on an Ubuntu 16.04 linux amd64 server, I’m trying to deploy your compiled binary file. I have downloaded the binary and put it in /usr/share/packetbeat/bin directory. I have made some adding in the packetbeat.yml file to declare the SIP port protocol.

I cannot retrieve any SIP data currently but I don’t think to be on the right way to make it working quickly, I’m a newbie on packetbeat... Have you shared some documents which explain how to deploy your binary please?

Have a nice weekend

Best regards

De : てぃーじぇ [mailto:[email protected]]
Envoyé : mardi 3 avril 2018 17:40
À : elastic/beats
Objet : Re: [elastic/beats] New protocol support: SIP (#152)

Hi, npenna22
Thank you for trying my sip plugin.
Packetbeat is single binary application, so if you want to deploy and use new protocol plugin, you should replace the current(6.2.3) packetbeat binary/setting file to new binary and setting file(my repository is version 7.0.0-alpha1).
*sample setting is included in my repository.

How to get a new packetbeat binary
Usually you need recompile with new source files yourself. You be clone the github repository and cd to packetbeat directory, then go build .. If you have no problem you can get a new binary on your current directory. (Of course, you should install go compiler in your pc/server.)
However, if you want to run a packetbeat on linux 64bits architecture, you can download compiled binary file form below link(I compiled on Ubuntu 16.04, go version go1.9.2 linux/amd64).
https://github.com/tj8000rpm/beats/blob/sip_protocol/packetbeat/packetbeat.linux_amd64
*this is NOT official version, so please try under your own risk.

Thanks.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/elastic/beats/issues/152#issuecomment-378295357, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AkRs7ZVrMI3pTAYp5gujRgVfxY4bSAL7ks5tk5fbgaJpZM4E01dj.

---

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

[urso]

@npenna22 @tj8000rpm Let's keep this ticket on discussing SIP protocol implementation in packetbeat itself. On discuss we have a beats developers forum. Maybe more people might be interested in joining the conversation or can help with compile instructions. For testing simple transactions, you can drive packetbeat via pcap files ;)

[npenna22]

Hi,

After few updates on my side, the very first tests seems to be positive. I manage to capture SIP trafic with your packetbeat binary. SDP part is also parsed. I'll continue my test later but it's encouraging. Thanks a lot tj8000rpm

Best regards

[npenna22]

After few tests I would like to share to you two potentials issues in your binary :

• your binary seems to not accept header compact form : have you you think about some evolutions to support this header form?
• If I capture the traffic on a server on which I simulate an inviting and a guest, I only see the INVITE sent by the inviter. The INVITE request received by the invitee seems to be ignore by packetbeat although the INVITE received is different from the one sent by the inviter (callid, sdp offer are different for example). Do you know what could explained this behavior please?

[tj8000rpm]

npenna22

Thank you for testing！
Compact form:

• Yes, it is. I just found on the same day :p. So I supported compact form yesterday.
  my update: https://github.com/tj8000rpm/beats/commit/3fe6626a7816cffd0ee6771010bb06433c73a054

Ignored packet:

• There seems to are no problem in my test lab. Is it correct settings such as SIP port number and ignore_outgoing parameter in your packetbeat.yml.
  https://www.elastic.co/guide/en/beats/packetbeat/current/configuration-interfaces.html

[npenna22]

Great job!

So I try to compil your last update. I take time to setup the compiling environment.

Currently the execution of the "make" command of beats project produce the following traces :
...
Building redis
ERROR: Compose tried decoding the following data chunk, but failed:
u'

It works!

\n'
scripts/Makefile:354: recipe for target 'build-image' failed

And the execution of the "make package" command produce the following traces :
ERROR: Field is duplicated. Please update and try again.exit status 1
../libbeat/scripts/Makefile:294: recipe for target 'update' failed

Have you encountered those issues during the packetbeat project compilation?