dnscrypt-proxy v2.0.10 crash on Alpine linux v3.7.0 x86

[westsuhanic]

Hello:

Thank you for your work.

Here is the core dump from dnscrypt-proxy v2.0.10 under Alpine linux v3.7.0-(grsecurity hardened), x86.

[2018-04-25 15:00:11] [NOTICE] dnscrypt-proxy is ready - live servers: 32
unexpected fault address 0x34846000
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x34846000 pc=0x16c985c2]

goroutine 64 [running]:
runtime.throw(0x16ec4ea1, 0x5)
    /usr/lib/go/src/runtime/panic.go:616 +0x77 fp=0x34405c10 sp=0x34405c04 pc=0x16b50ef7
runtime.sigpanic()
    /usr/lib/go/src/runtime/signal_unix.go:395 +0x1e4 fp=0x34405c34 sp=0x34405c10 pc=0x16b668c4
github.com/aead/chacha20/chacha.xorKeyStreamSSE2(0x34405d34, 0x40, 0x40, 0x34405d34, 0x40, 0x40, 0x34388040, 0x34388000, 0x14, 0x16c97555)
    /home/buildozer/aports/community/dnscrypt-proxy/src/dnscrypt-proxy-2.0.10/src/github.com/aead/chacha20/chacha/chacha_386.s:157 +0x162 fp=0x34405c38 sp=0x34405c34 pc=0x16c985c2
github.com/aead/chacha20/chacha.xorKeyStream(0x34405d34, 0x40, 0x40, 0x34405d34, 0x40, 0x40, 0x34388040, 0x34388000, 0x14, 0x20)
    /home/buildozer/aports/community/dnscrypt-proxy/src/dnscrypt-proxy-2.0.10/src/github.com/aead/chacha20/chacha/chacha_386.go:65 +0x8e fp=0x34405c64 sp=0x34405c38 pc=0x16c975de
github.com/aead/chacha20/chacha.(*Cipher).XORKeyStream(0x34388000, 0x34405d34, 0x40, 0x40, 0x34405d34, 0x40, 0x40)
    /home/buildozer/aports/community/dnscrypt-proxy/src/dnscrypt-proxy-2.0.10/src/github.com/aead/chacha20/chacha/chacha.go:165 +0x1b3 fp=0x34405ca0 sp=0x34405c64 pc=0x16c97233
runtime: unexpected return pc for github.com/jedisct1/xsecretbox.Seal called from 0x8f1f4b36
stack: frame={sp:0x34405ca0, fp:0x34405d90} stack=[0x34405000,0x34406000)
34405c20:  34846000  34247ce0  16b7a981 <runtime.morestack+65>  16b666e5 <runtime.sigpanic+5> 
34405c30:  16c985c2 <github.com/aead/chacha20/chacha.xorKeyStreamSSE2+354>  16c975de <github.com/aead/chacha20/chacha.xorKeyStream+142>  34405d34  00000040 
34405c40:  00000040  34405d34  00000040  00000040 
34405c50:  34388040  34388000  00000014  16c97555 <github.com/aead/chacha20/chacha.xorKeyStream+5> 
34405c60:  16c97233 <github.com/aead/chacha20/chacha.(*Cipher).XORKeyStream+435>  34405d34  00000040  00000040 
34405c70:  34405d34  00000040  00000040  34388040 
34405c80:  34388000  00000014  00000020  00000020 
34405c90:  00000000  00000000  16c97085 <github.com/aead/chacha20/chacha.(*Cipher).XORKeyStream+5>  16ca06e8 <github.com/jedisct1/xsecretbox.Seal+216> 
34405ca0: <34388000  34405d34  00000040  00000040 
34405cb0:  34405d34  00000040  00000040  34388000 
34405cc0:  00000000  00000000  16b77b60 <runtime.(*mcache).nextFree.func1+0>  24006000 
34405cd0:  16b3deb5 <runtime.gcTrigger.test+5>  16b35d64 <runtime.mallocgc+948>  00000001  00000000 
34405ce0:  00000000  00000000  00000000  01014304 
34405cf0:  00000000  00000500  00000002  3459e000 
34405d00:  172235e0  00000000  00004000  34277680 
34405d10:  24006000  00000000  00000060  342608f0 
34405d20:  343723c0  16b7cc11 <runtime.memclrNoHeapPointers+65>  16b67cf7 <runtime.growslice+519>  3459e281 
34405d30:  0000027f  799b09b3  476476e4  ad4a01ac 
34405d40:  a3add054  94aee6ed  85cd259d  949fb27a 
34405d50:  b3a691f3  6dc28ba2  982bd4a0  ad57b292 
34405d60:  767d8824  63c5ec33  e796424c  04ea8fe5 
34405d70:  4f76766f  d6e51325  255a730e  fb3dd7d5 
34405d80:  f6612be9  f82eaff2  ed85cf31 !8f1f4b36 
34405d90: >e035f845  d0ec3743  669815bf  33bcad34 
34405da0:  8cf9ac4a  e82a4ccf  0313d34d  2b227014 
34405db0:  8b668a36  68e7beaf  b8797db0  e9a064f5 
34405dc0:  d51853e4  1a2d2a22  78155d31  06ae1b93 
34405dd0:  792b1791  352f22d7  e64a17e7  75076b81 
34405de0:  1a666551  722ff12f  8cc86c83  1699194a 
34405df0:  a213dca6  16fe22aa  6c8f8ad2  dd88f8b2 
34405e00:  fffe84bd  72d8df77  e99459f9  13b0478a 
github.com/jedisct1/xsecretbox.Seal(0xe035f845, 0xd0ec3743, 0x669815bf, 0x33bcad34, 0x8cf9ac4a, 0xe82a4ccf, 0x313d34d, 0x2b227014, 0x8b668a36, 0x68e7beaf, ...)
    /home/buildozer/aports/community/dnscrypt-proxy/src/dnscrypt-proxy-2.0.10/src/github.com/jedisct1/xsecretbox/xsecretbox.go:31 +0xd8 fp=0x34405d90 sp=0x34405ca0 pc=0x16ca06e8
created by main.(*Proxy).udpListener
    /home/buildozer/aports/community/dnscrypt-proxy/src/dnscrypt-proxy-2.0.10/dnscrypt-proxy/proxy.go:141 +0x88

thank you

west suhanic

[jedisct1]

Sigh. /cc @aead

[aead]

@jedisct1 The only thing I can think of that may cause this is issue is a unsigned comparison. The 386 code compares the length of the slice using JB (jump if below (unsigned)). I change it to JL (jump if less (signed)). Both versions work well on amd64 and amd64 running in 386 comp. mode. I've build dnscrypto-proxy with this change for linux_386.

@westsuhanic It would be great if you can try out the custom release. If this doesn't fix the issue, I'll disable 386 assembly until I can reproduce and investigate it properly.

[jedisct1]

Did you install the apk package?

I installed Alpine Linux in a VM to give it a spin, and it appears to be utterly broken. I couldn't even get it to get to the point where it would start forwarding queries.

Recompiling it in Alpine produced a completely broken binary as well. Using the standard precompiled binaries worked perfectly.

If this is how you installed it, can you try the standard binaries available here? https://github.com/jedisct1/dnscrypt-proxy/releases

[aead]

I basically compiled to the code like done in the .travis.yml file for linux-386. On my (amd64) machine the binary works fine:

[sudo] password for andreas: 
[2018-04-25 22:45:15] [NOTICE] Source [public-resolvers.md] loaded
[2018-04-25 22:45:15] [NOTICE] dnscrypt-proxy 2.0.10
[2018-04-25 22:45:15] [NOTICE] Now listening to 127.0.0.1:53 [UDP]
[2018-04-25 22:45:15] [NOTICE] Now listening to 127.0.0.1:53 [TCP]
[2018-04-25 22:45:15] [NOTICE] Now listening to [::1]:53 [UDP]
[2018-04-25 22:45:15] [NOTICE] Now listening to [::1]:53 [TCP]
[2018-04-25 22:45:16] [NOTICE] [arvind-io] OK (crypto v2) - rtt: 251ms
...
[2018-04-25 22:45:33] [NOTICE] Server with the lowest initial latency: dnscrypt.me (rtt: 4ms)
[2018-04-25 22:45:33] [NOTICE] dnscrypt-proxy is ready - live servers: 31

Let me rebuild and test in a VM...

[jedisct1]

I didn't try the x86_64 version, only the i386 one.

[aead]

I didn't try the x86_64 version, only the i386 one.

Yes, there is no x86_64 version, I've only build the i386. (cross compile). But I've only tested the i386 bin on a amd64 (386 comp. mode). I'll rebuild and setup a 32 bit alpine in a VM to make sure the bin works.

[ianbashford]

I've been running on alpine for sometime and built my own packages.
I have it running in two x86_64 VMs (xen). Can I help out in anyway?

[aead]

Can I help out in anyway?

That would be awesome @ianbashford ! Can you download the:

• https://github.com/aead/dnscrypt-proxy/releases/download/2.0.11/dnscrypt-proxy-linux_i386-2.0.11.tar.gz
• extract it: tar -xzf && cd linux-i386
• mv example-dnscrypt-proxy.toml dnscrypt-proxy.toml
• ./dnscrypt-proxy

I've also setup a VM and I can verify the config but i'm somehow unable to bind the server to :53...
but it resolves e.g. cloudflare-dns.com - So the binary works (to some extend) on alpine_386 (for me ^^)

[ianbashford]

The vm is x86_64. Ran the i386 binary as you described and responds fine. One minor adjustment I made to the config was to listen on
listen_addresses = [':53']

Hope that's an OK mod that doesn't invalidate the test...

[aead]

Awesome @ianbashford , thanks a lot! - I was also able to start the server (forgot configuring loopback ^^)

@westsuhanic Can you please try https://github.com/aead/dnscrypt-proxy/releases/download/2.0.11/dnscrypt-proxy-linux_i386-2.0.11.tar.gz and see whether the issue still exists?

[jedisct1]

@aead Did you try to compile it on Alpine/i386 instead of cross-compile? I suspect the Go compiler to be b0rked on this platform.

[ianbashford]

Sorry that took a while; had to create a new vm with Alpine standard 3.7 x86.

apk add dnscrypt-proxy 
~# dnscrypt-proxy -config /etc/dnscrypt-proxy/dnscrypt-proxy.toml
[2018-04-25 23:54:00] [NOTICE] Source [public-resolvers.md] loaded
[2018-04-25 23:54:00] [NOTICE] dnscrypt-proxy 2.0.10
[2018-04-25 23:54:00] [NOTICE] Now listening to 127.0.0.1:53 [UDP]
[2018-04-25 23:54:00] [NOTICE] Now listening to 127.0.0.1:53 [TCP]
[2018-04-25 23:54:00] [NOTICE] Now listening to [::1]:53 [UDP]
[2018-04-25 23:54:00] [NOTICE] Now listening to [::1]:53 [TCP]

Then from another session

~# nslookup www.google.com localhost
Server:    127.0.0.1
Address 1: 127.0.0.1 alpine32.iso

Name:      www.google.com
Address 1: 216.58.206.100 lhr25s14-in-f4.1e100.net
Address 2: 2a00:1450:4009:810::2004 lhr25s14-in-x04.1e100.net

So the package version is working -- I'll try your tar now I have things on the right platform...

[ianbashford]

Tried the tar from your last post on 32bit VM. That's working too

[ianbashford]

Just built from the latest aports on the x86 VM (2.0.10) and that's working... anything else I can try to help out?

[aead]

@jedisct1 No, haven't tried that. But if I understand @ianbashford correctly he did - and it's working (on his platform). EDIT: I also looked at the generated assembler and machine-code:
go tool objdump ~/go/bin/linux_386/dnscrypt-proxy | grep chacha_386.s - AFAICT it looks fine...

@ianbashford Great, thanks. One question is this a 32 bit VM running on an amd64 host (as you mentioned before) or an x86-i386 host?

anything else I can try to help out?

If you have 386-only hardware it would be awesome to run the tar (I linked in my comment) on it.
Otherwise I think no - thank you very much :+1:

[ianbashford]

Sorry the host _is_ running on amd64... I don't have 386 hardware.

I offered to help the alpine maintainer (who doesn't actually use the software) so I've put in a PR (as I do use it). If you have some ideas for the alpine package let me know and I can at least see if I can get them adopted. (I'm not an alpine committer so have no actual influence but happy to try to help)

[westsuhanic]

Hello Everyone:

I have tried the custom release on my x86 hardware and it works. I am
testing repeatedly using, for example:

sudo ./dnscrypt-proxy -resolve ibm.fr

Running the above command against dnscrypt-proxy v2.0.10-r0 from
edge/community would core every time.

What was changed in order to get this to work?
Is there anything else I can try?
How does this custom release make it into the apk repository?

thank you,

west suhanic

On Wed, Apr 25, 2018 at 2:30 PM, Andreas Auernhammer <
[email protected]> wrote:

@jedisct1 https://github.com/jedisct1 The only thing I can think of
that may cause this is issue is a unsigned comparison. The 386 code
compares the length of the slice using JB (jump if below (unsigned)). I
change it to JL (jump if less (signed)). Both versions work well on amd64
and amd64 running in 386 comp. mode. I've build dnscrypto-proxy with this
change for linux_386.

@westsuhanic https://github.com/westsuhanic It would be great if you
can try out the custom release
https://github.com/aead/dnscrypt-proxy/releases/download/2.0.11/dnscrypt-proxy-linux_i386-2.0.11.tar.gz.
If this doesn't fix the issue, I'll disable 386 assembly until I can
reproduce and investigate it properly.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jedisct1/dnscrypt-proxy/issues/404#issuecomment-384388678,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbU48xoe_qYVDWL_i-7wMIM3NRai9jMlks5tsMC2gaJpZM4Tjngp
.

[aead]

Thanks @westsuhanic - That's great news! However it's very interesting that the unsigned JB instruction seems to be the reason for this issue.
I will fix my chacha20 implementation, than @jedisct1 can revendor and make a new official release?!

[jedisct1]

@westsuhanic Can you try the standard release as well?

[jedisct1]

Revendor'd, but I'd still be interested in @westsuhanic 's results with the standard release (vs a release compiled on Alpine).

[jedisct1]

@aead The issue I have with Alpine is not the segfault reported above, but appears to be chacha.HChaCha20() stuck in an infinite loop.

chacha.HChaCha20(&sharedKey, &nonce, &sharedKey)

Could it be because the key and the output overlap?

[jedisct1]

Answering my own question: nope, it also happens when using distinct arrays.

[jedisct1]

This is using the SSSE3 implementation.

[jedisct1]

And it works perfectly using the SSE2 implementation.

[jedisct1]

This is kinda crazy, but:

        MOVL  $20, Rounds
        MOVOU ?rol16<>(SB), X5
        MOVOU ?rol8<>(SB), X6
chacha_loop:
      SUBL $2, Rounds
      JNZ  chacha_loop

is an infinite loop.

Looks like Go assembly is completely broken on Alpine, especially jump instructions.

[jedisct1]

Mmmm... actually...

        MOVL  $20, Rounds
        MOVOU ?rol16<>(SB), X5
        MOVOU ?rol8<>(SB), X6
chacha_loop:
        SUBL $2, Rounds
        JNZ  chacha_loop

produces an infinite loop, but moving the iteration counter down:

        MOVOU ?rol16<>(SB), X5
        MOVOU ?rol8<>(SB), X6
        MOVL  $20, Rounds
chacha_loop:
        SUBL $2, Rounds
        JNZ  chacha_loop

works.

[jedisct1]

  2179ad:       f3 0f 6f 53 10          movdqu 0x10(%ebx),%xmm2
  2179b2:       f3 0f 6f 18             movdqu (%eax),%xmm3
  2179b6:       b9 14 00 00 00          mov    $0x14,%ecx
  2179bb:       e8 90 bd e8 ff          call   a3750 <__x86.get_pc_thunk.cx>    // <- uh-oh!
  2179c0:       f3 0f 6f a9 70 b7 27    movdqu 0x27b770(%ecx),%xmm5
  2179c7:       00
  2179c8:       e8 83 bd e8 ff          call   a3750 <__x86.get_pc_thunk.cx>    // <- uh-oh!
  2179cd:       f3 0f 6f b1 73 b7 27    movdqu 0x27b773(%ecx),%xmm6
  2179d4:       00
  2179d5:       66 0f fe c1             paddd  %xmm1,%xmm0
  2179d9:       66 0f ef d8             pxor   %xmm0,%xmm3
  2179dd:       66 0f 38 00 dd          pshufb %xmm5,%xmm3

get_pc_thunk.cx may clobber CX ...

[jedisct1]

@aead So, I think this is the bug. CX gets clobbered. Incidentally, you are using it for the rounds count and the length. Which may explain why it went negative.

[jedisct1]

Compiled on Ubuntu, the same code doesn't have that issue:

 81a0084:       f3 0f 6f 0b             movdqu (%ebx),%xmm1
 81a0088:       f3 0f 6f 53 10          movdqu 0x10(%ebx),%xmm2
 81a008d:       f3 0f 6f 18             movdqu (%eax),%xmm3
 81a0091:       b9 14 00 00 00          mov    $0x14,%ecx
 81a0096:       f3 0f 6f 2d 50 f4 4a    movdqu 0x84af450,%xmm5
 81a009d:       08
 81a009e:       f3 0f 6f 35 60 f4 4a    movdqu 0x84af460,%xmm6
 81a00a5:       08
 81a00a6:       66 0f fe c1             paddd  %xmm1,%xmm0
 81a00aa:       66 0f ef d8             pxor   %xmm0,%xmm3

[jedisct1]

Maybe because Alpine (and apparently Android) use -buildmode=pie by default?

[jedisct1]

Yep: https://groups.google.com/forum/#!topic/golang-dev/gRCe5URKewI

[jedisct1]

So, CX is reserved for PIC. We can't use it (at least without being extra cautious).

[jedisct1]

https://github.com/aead/chacha20/pull/23 should fix it.

[jedisct1]

Thanks for merging this, @aead.

Time to tag a new release, then.

[aead]

Wow, that is a really nasty bug. Actually that should be documented for the pie mode somewhere.
Thanks for root cause analysis!

[jedisct1]

That was a nasty one, indeed. And it destroys guarantees that a lot of people may expect from the Go compiler.

[aead]

Okay found the doc^^: https://golang.org/doc/asm

When using the compiler and assembler's -dynlink or -shared modes, any load or store of a fixed memory location such as a global variable must be assumed to overwrite CX. Therefore, to be safe for use with these modes, assembly sources should typically avoid CX except between memory references

The first time I've read the paragraph... Writing assembly is a journey full of pitfalls ¯_(ツ)_/¯

[jedisct1]

Good to know that at least it's always gonna be CX.

[westsuhanic]

Hello:

I have aready tried dnscrypt-proxy v2.0.10-r0 built on 2018-04-17
14:57:36. My error report was based on that release.
Is there another release I am missing?

thank you,

west suhanic

On Thu, Apr 26, 2018 at 2:34 PM, Frank Denis [e-mail]
wrote:

@westsuhanic https://github.com/westsuhanic Can you try the standard
release as well?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jedisct1/dnscrypt-proxy/issues/404#issuecomment-384746031,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AbU48xmTYMbdQ0U01MX45SYqR5iW_qqrks5tshM0gaJpZM4Tjngp
.

[ianbashford]

Sorry to necro-bump.
2.0.11 (last night in releases ) was also merged into alpine aports today.

[jedisct1]

@ianbashford Awesome! Thanks!