Sparkle: generate_appcast doesn't create DSA signatures on High Sierra

[mikepulaski]

I'm having the exact same problem as described in #1129, but without the crash.

I recompiled generate_appcast to include the changes from ec7f55d but the signature is still not being computed. Rather than crashing, I get a rather cryptic error:

Error Domain=Internal CSSM error Code=-2147416063 "Internal error #80010801 at SignTransform_block_invoke /BuildRoot/Library/Caches/com.apple.xbs/Sources/Security/Security-58286.20.16/OSX/libsecurity_transform/lib/SecSignVerifyTransform.c:276" UserInfo={NSDescription=Internal error #80010801 at SignTransform_block_invoke /BuildRoot/Library/Caches/com.apple.xbs/Sources/Security/Security-58286.20.16/OSX/libsecurity_transform/lib/SecSignVerifyTransform.c:276, Originating Transform=CoreFoundationObject}

I'm using keys generated from generate_keys. I'm following the Basic Setup guide.

[kornelski]

Ouch, internal errors are not fun. This file is open-source but I have trouble finding this exact version (the line number doesn't make sense otherwise).

Our previous crash was just from assuming this function never fails. Because it shouldn't :/

What macOS version do you have? Is there anything special about your setup? Does it work on another machine?

[mikepulaski]

@pornel Thanks for the quick response. I've tried it on macOS 10.13.1 and 10.12.4. I don't have anything special about my setup, certainly nothing that would affect system-level APIs.

Interestingly enough, sign_update works, which I'm guessing serves the same function as this routine.

[kornelski]

Yes, sign_update does the same thing, but uses OpenSSL instead of Apple's APIs.

[mikepulaski]

@pornel Ok, good(-ish) news: When using Sparke's "Test Application" private key, it works. I suspect the private key output from generate_keys is to blame, which is a scenario likely to happen when following the Basic Setup.

That said, it's strange; there doesn't seem to be anything alarming in the script (or DSASignature.swift).

[mikepulaski]

I've resolved this by creating a 1028 byte key rather than 4096. For what it's worth, I've tried using 2048 byte keys but they still caused problems.

openssl gendsa <(openssl dsaparam 1028) -out dsa_priv.pem
openssl dsa -in dsa_priv.pem -pubout -out dsa_pub.pem

I'm personally fine with this, but it seems strange that SecSignTransformCreate would not support stronger keys. I tried overriding the attributes of the returned SecTransform but it just made it crash again :)

[mikepulaski]

Ok, I now understand the problem from a high level.

• Sierra: openssl version → OpenSSL 0.9.8zh
• High Sierra: openssl version → LibreSSL 2.2.7

LibreSSL, for whatever reason, generates keys which may be incompatible with libsecurity_transform, the library used to calculate signatures for generate_appcast. If I generate keys on my other machine (running 10.12, OpenSSL 0.9.8), everything works; the keys decode on High Sierra and generate_appcast runs as expected.

A possible workaround could be to rewrite generate_keys natively so that it doesn't use the openssl command.

I'll go ahead and close the issue since it's pretty much outside the scope of this project. I'm just happy to have it working and wanted to document this behavior in case someone else comes across it.

[efef]

I noticed, when still running Sierra, that an old OpenSSL version was used. So I decided to generate the 4096 DSA key on a up2date Debian Stretch linux machine which has a more recent OpenSSL version. When using the generated private key with "generate_appcast", the app gives "Internal error" and doesn't sign.

[efef]

it seems both newer LibreSSL and OpenSSL versions generate 'incompatible' keys for larger keysize. So it isn't @mikepulaski an libreSSL issue. When inspecting the generated keys with 'openssl asn1parse -in key' it shows two integers that are longer. It looks like sha256 might be the reason for this. So the two prime numbers are hashed with sha256 instead of sha1.
When looking into libsecurity_transform I notice a line “Invalid digest algorithm for DSA signature, only SHA1 is supported”. Better not wait if Apple will fix it thought. Let's move to something better than DSA.