Pwnagotchi: use websockets instead of polling

[evilsocket]

currently bettercap's /api/event is polled every second, since bettercap supports websockets we should be using that to receive the events

[dangilkerson]

I'd like to take a crack at this.

[evilsocket]

that'd be great! https://www.bettercap.org/modules/core/api.rest/

ws in bettercap never got tested properly, so they might not work 100% correctly ... if that happens, i'll fix it :D

[dangilkerson]

This is the best way to test it :)

[hslatman]

@dangilkerson: Did you get a shot at it?

I've tried the bettercap WS support and it seems to return an HTTP 200 instead of 101 when accessing ws://user:pass@localhost:8081/api/events. The usual WS headers are set, though 😕 I used the websockets library to connect with Python. The client breaks on the status code not being equal to 101.

[evilsocket]

@hslatman did you set api.rest.websocket to true before starting the API module in bettercap?

[hslatman]

@evilsocket Yes and I've verified it's showing as 'on' in the web interface.

[evilsocket]

mmm i have no idea ... this is the implementation https://github.com/bettercap/bettercap/blob/master/modules/api_rest/api_rest_ws.go

[hslatman]

Got it ... 😅

The parameter does not react to changes that are sent using the API, so I started bettercap with bettercap -eval 'set api.rest off; set api.rest.websocket true; set api.rest on;'. Now I've got something going and will try some stuff with it.

Thanks @evilsocket, your first response was right!

[evilsocket]

oh yeah, in order for parameters to be changed you either set them before the module starts, or restart the module yourself :D glad you figured it out, i'm very looking forward to see what's next @hslatman 👍

[dangilkerson]

@hslatman Looking at it today but if you've got something going I can move on to something else.

[dangilkerson]

Watching events over websockets through a test script now. Going to replace the event polling code.

[evilsocket]

i'm very impressed by the fact this works out of the box XD

[dangilkerson]

Oh don't worry there's issues I'll probably have changes for bettercap as well ;)

[hslatman]

@hslatman Looking at it today but if you've got something going I can move on to something else.

No, I was looking into using the websocket functionality within a different project. I don't have pwnagotchi running, so get your best shot at it ;-)

[hslatman]

Just found out about this issue again. I've created a small POC repository with an example for using bettercap with WebSockets. It's largely based on the Agent/Client setup in this repository, with some WebSocket specific steps included, so it could be an option to integrate the WebSocket support in this repository at some point and when constraints allow for this. Perhaps @dangilkerson made some progress on this too?

It's available here.

[dadav]

Derek assign: dadav

[dadav]

Hmmm. The connection always gets terminated exactly after 1min and shows this msg:
https://github.com/bettercap/bettercap/blob/4eead7eafaec57282ad435473e420d7bad7a602e/modules/api_rest/api_rest_ws.go#L98

[hslatman]

Hmmm. The connection always gets terminated exactly after 1min and shows this msg:
https://github.com/bettercap/bettercap/blob/4eead7eafaec57282ad435473e420d7bad7a602e/modules/api_rest/api_rest_ws.go#L98

That's not the same behavior as I've observed running my small POC. Perhaps the asyncio task keeping the connection is finished / killed without you knowing, resulting in no ping being sent at all.

[dadav]

Hmmm. The connection always gets terminated exactly after 1min and shows this msg:
https://github.com/bettercap/bettercap/blob/4eead7eafaec57282ad435473e420d7bad7a602e/modules/api_rest/api_rest_ws.go#L98

That's not the same behavior as I've observed running my small POC. Perhaps the asyncio task keeping the connection is finished / killed without you knowing, resulting in no ping being sent at all.

Maybe thats it! I will switch to create_task + run_forever instead of run_until_complete. Also I should have added a try/catch around the consume(msg). It's just strange that its exactly after 1 minute, which is exactly the pongWait value of bettercap.
https://github.com/bettercap/bettercap/blob/4eead7eafaec57282ad435473e420d7bad7a602e/modules/api_rest/api_rest_ws.go#L18

Websockets is handling the ping/pong stuff internaly. There was an issue in 7.x versions, which is why I upgraded to 8.1. Yesterday I was just playing with the websocket.connect options, like ping_interval and ping_timeout, but I could not fix it! But today is another day!!

[dadav]

[2020-04-08 08:14:25,107] [DEBUG] client < Frame(fin=True, opcode=9, data=b'', rsv1=False, rsv2=False, rsv3=False)
[2020-04-08 08:14:25,124] [DEBUG] client - received ping, sending pong: [empty]                                                                  
[2020-04-08 08:14:25,179] [DEBUG] client > Frame(fin=True, opcode=10, data=b'', rsv1=False, rsv2=False, rsv3=False)
# some more data
# and then...     
[2020-04-08 08:14:25,216] [DEBUG] client - event = eof_received()                                                                                                                                            
[2020-04-08 08:14:25,321] [DEBUG] client ! failing OPEN WebSocket connection with code 1006                                                      
[2020-04-08 08:14:25,343] [DEBUG] client - event = connection_lost(None) 
[2020-04-08 08:14:25,410] [DEBUG] client - state = CLOSED
DEBUG:websockets.protocol:client x code = 1006, reason = [no reason]

Errorcode 1006 means the peer (bettercap) closed the connection

Bettercap log (always the same log; therefore ts is different):

 mon0  » [08:44:39] [sys.log] [dbg] api.rest Closing websocket reader.                                                                           
 mon0  » [08:44:39] [sys.log] [err] api.rest Error while writing websocket message: write tcp 127.0.0.1:8081->127.0.0.1:59088: use of closed network connection  

I'll try to find out why ws.ReadMessage() is failing...

[evilsocket]

you might want to print the error here https://github.com/bettercap/bettercap/blob/4eead7eafaec57282ad435473e420d7bad7a602e/modules/api_rest/api_rest_ws.go#L98

[evilsocket]

pushed a more verbose log of the error, hope that helps

[dadav]

pushed a more verbose log of the error, hope that helps

thx!!!

seems to fail because of i/o timeout >.>

Edit://
I'll dig deeper ^^

Edit2://
PongHandler never fires!!

Edit3://
The events seem to be read in "bursts" by pythons websockets package. Maybe some schedule-issue of asyncio?

[dadav]

ok, i think i got it! Will push a fix durring the day. Seems like the await consumer(msg) blocks everything, even the pong-functionality.

edit:// at least i thought so...omg, this asyncio is driving me crazy

edit:// its working :)

[dadav]

Shouldnt here:
https://github.com/bettercap/bettercap/blob/4eead7eafaec57282ad435473e420d7bad7a602e/modules/api_rest/api_rest_ws.go#L42

also do:
ws.SetReadDeadline(time.Now().Add(pongWait))
?

Otherwise e.g. for the initial ping (after 54secs), pwnagotchi only has 6 secs (ReadTimeout was set to 60secs and 54secs are already over) to answer it with a pong. This is normaly easily doable, but at the beginning there are a sh!tload of events which keep pwnagotchi busy and therefore it takes longer than 6secs to send the first pong.

edit:// tested the fix and it works like a charm!! :) if you agree i can make a pr

[evilsocket]

possibly