Hammerspoon: Bug: Sending binary data with UDP not possible

[dasmurphy]

I'm writing a local dns resolver for local test domains. I've ported white_dns.py to Lua. I got the problem that the DNS protocol is binary. I receive the data, but sending is not possible.

I use dig for testing the domain from command line.

Running test with my lua dns resolver.

_Output from dig_

❯ dig @127.0.0.1 -p 20560 test50.hs
;; Got bad packet: bad compression pointer
112 bytes
7d 65 ef bf bd ef bf bd e2 88 85 01 e2 88 85 01          }e..............
e2 88 85 e2 88 85 e2 88 85 e2 88 85 06 74 65 73          .............tes
74 35 30 02 68 73 e2 88 85 e2 88 85 01 e2 88 85          t50.hs..........
01 e2 88 85 e2 88 85 29 10 e2 88 85 e2 88 85 e2          .......)........
88 85 e2 88 85 e2 88 85 e2 88 85 e2 88 85 ef bf          ................
bd 0c e2 88 85 01 e2 88 85 01 e2 88 85 e2 88 85          ................
e2 88 85 3c e2 88 85 04 7f e2 88 85 e2 88 85 01          ...<............

_Output from the Lua console_

38 - 7d 65 01 20 00 01 00 00 00 00 00 01 06 74 65 73 74 35 30 02 68 73 00 00 01 00 01 00 00 29 10 00 00 00 00 00 00 00
{ "test50.hs.", { 127, 0, 0, 1 } }
54 - 7d 65 81 80 00 01 00 01 00 00 00 00 06 74 65 73 74 35 30 02 68 73 00 00 01 00 01 00 00 29 10 00 00 00 00 00 00 00 c0 0c 00 01 00 01 00 00 00 3c 00 04 7f 00 00 01
--end---

_Clarifications for Lua_
The first data line is the data the resolver received (byte before - is length, not send). Then the domain which was queried is displayed and the ip address where it should point to. The second data line is the data which is send to the server back (byte before - is length, not send).
Dig receives modified data which is much bigger. It seems that there is some conversation (UTF8?) of the string in the modul for sending data. I think it is done here, but i'm not sure.
https://github.com/Hammerspoon/hammerspoon/blob/master/extensions/socket/udp.m#L375

Running test with white_dns.py

_Output from dig_

❯ dig @127.0.0.1 -p 20560 test50.hs
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.10.6 <<>> @127.0.0.1 -p 20560 test50.hs
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2536
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: Message has 16 extra bytes at end

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;test50.hs.         IN  A

;; Query time: 0 msec
;; SERVER: 127.0.0.1#20560(127.0.0.1)
;; WHEN: Fri Mar 13 10:34:22 CET 2020
;; MSG SIZE  rcvd: 54

_Output from white_dns.py_

❯ python ./white_dns.py (modified for data output)
[INFO] Starting WhiteDNS server..Done.
[INFO] Listening on port 20560
data
09 e8 01 20 00 01 00 00 00 00 00 01 06 74 65 73 74 35 30 02 68 73 00 00 01 00 01 00 00 29 10 00 00 00 00 00 00 00
packet
09 e8 81 80 00 01 00 01 00 00 00 00 06 74 65 73 74 35 30 02 68 73 00 00 01 00 01 00 00 29 10 00 00 00 00 00 00 00 c0 0c 00 01 00 01 00 00 00 3c 00 04 7f 00 00 01
[REQUEST] test50.hs. -> 127.0.0.1
^C[INFO] Shutting down WhiteDNS server..Done.

_Clarifications for white_dns.py_
I modified white_dns.py to display the received data and the packet which will be send. The data received and send is near the same as in lua, except some bytes that are always different. Anyway dig does not receive the correct data from lua.

Why i'm writing a dns resolver in lua?
I have Anvil and Pow running currently and i want to remove it from my system since there is no more development on them. And second reason is: because i can :) Later i will write a small webserver from hs.httpserver to make it possible to use multiple domains with different folders.

Question

• Is it possible to send binary data with the send UDP command?
• If no, is it possible to modify the UDP object to send binary data, and when yes, how?

I'm not very familiar with .m files from the Mac. I used to know C, but the Mac development was not in my learning.

I attached the localdnsresolver.lua as well as my modified white_dns.py and the file that must be put in the /etc/resolver folder on the Mac.

Attachments
localdnsresolver.zip
white_dns.py.zip

[jparoz]

I also have this issue, while trying to send OSC packets from Hammerspoon. Specifically, when I send any UDP message that includes a null byte ("\0"), it is substituted on the other end by "\xE2\x88\x85", the Unicode empty set character (∅). I don't believe there's any good reason for that substitution to be made here, and is a straight bug.

Here is code that reproduces the bug:

-- helper function
function print_bytes(msg)
    s = ""
    for c in msg:gmatch(".") do 
        s = s .. string.format("%02X ", c:byte())
    end
    print(s)
end

sock = hs.socket.udp.new():connect("127.0.0.1", 4444)
listener = hs.socket.udp.new():listen(4444)
listener:receive(function(msg)
    print(string.format("%q", msg))
    print_bytes(msg)
end)

test_msg = "\0\1\2\3"
print(string.format("%q", test_msg))
print_bytes(test_msg)
sock:send(test_msg)

Output:

2020-04-27 22:57:19: Before sending:
2020-04-27 22:57:19: "\0\1\2\3"
2020-04-27 22:57:19: 00 01 02 03 

2020-04-27 22:57:19: After sending:
2020-04-27 22:57:19: "∅\1\2\3"
2020-04-27 22:57:19: E2 88 85 01 02 03 

[asmagill]

Sigh, this was fixed for TCP awhile back but I guess we forgot to apply the same changes to the UDP portion...

It's because the send function is treating the message as a "string" and by default, this means that it "cleans up" the data to make it safe for display in the console.

I'll try and get a patch applied later this week; if someone else wants to jump on it sooner, the fix needs to be applied to ../hammerspoon/extensions/socket/udp.m:

1. Remove lines 381 and 401 entirely:
   ~objc
   sendData = [message dataUsingEncoding:NSUTF8StringEncoding];
   ~
2. Changes lines 366-367 from:
   ~objc
   NSString *message = [skin toNSObjectAtIndex:2];
   NSData *sendData;
   ~
   to:
   ~objc
   NSData *sendData = [skin toNSObjectAtIndex:2 withOptions:LS_NSLuaStringAsDataOnly];
   ~

I think that should be sufficient, but additional testing should be done to make sure.

[latenitefilms]

Added pull request #2347 if you want to build Hammerspoon yourself and test @jparoz?

[dasmurphy]

@jparoz Thanks for creating another test script. And bringing this up again.
@asmagill Thanks for reacting and telling how to fix the source. ;) I was wondering if this was an issue which will not be solved at all.
@latenitefilms Thanks for fixing this bug. I can try to build it myself, would be awesome if it would be merged next days, when it is good.

I will be very happy when this will be working ;) Removes one more Python script from my computer.