Runtime: AppHost signature not being removed on macOS

[externl]

Description

At some point in the last few weeks C# apps using TLS started crashing on a project I work on. After comparing Console.app logs with a colleague who is running with the same system configuration, we noticed one noticeable difference:

code requirement check failed (-67050), client is not Apple-signed

After digging closer we noticed that dotnet apps on my colleague's Macbook were not signed at all while, mine had a bogus signature.

It seems that for some reason, when I created an app using the <UseAppHost>True</UseAppHost> property, the signature is not removed from the apphost app after copying it (https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs#L128 ?). This seems to lead to internal errors when using TLS (see a stack at the end).

How to reproduce

❯ mkdir foo && cd foo
❯ dotnet new console
*** edit project foo.csproj to include <UseAppHost>True</UseAppHost> ***
❯ dotnet build

The generated foo binary now has an invalid signature.

❯ codesign -v bin/Debug/netcoreapp3.1/foo
bin/Debug/netcoreapp3.1/foo: invalid signature (code or signature have been modified)
In architecture: x86_64

Expected result

❯ codesign -v bin/Debug/netcoreapp3.1/foo
bin/Debug/netcoreapp3.1/foo: code object is not signed at all
In architecture: x86_64

Configuration

❯ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.401
 Commit:    39d17847db

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.15
 OS Platform: Darwin
 RID:         osx.10.15-x64
 Base Path:   /usr/local/share/dotnet/sdk/3.1.401/

Host (useful for support):
  Version: 3.1.7
  Commit:  fcfdef8d6b

.NET Core SDKs installed:
  3.1.401 [/usr/local/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Regression?

Yes. Though it works fine for my colleagues. Just failing for me

Other information

• One notable thing that happened recently is that I installed, and then uninstalled quickly after, the Xcode 12 and Apple Configurator beta.

Code sign info detail for foo

❯ codesign -dv bin/Debug/netcoreapp3.1/foo
Executable=/Users/joe/workspace/foo/bin/Debug/netcoreapp3.1/foo
Identifier=client-555549446e4495955e3d30318046040352838d5a
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=832 flags=0x2(adhoc) hashes=21+2 location=system
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

The SSL/TLS exception (probably not relevant)

   ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
    ---> Interop+AppleCrypto+SslException: Internal error
      --- End of inner exception stack trace ---
      at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
      at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
      at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
      at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
      at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
      at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
   --- End of stack trace from previous location where exception was thrown ---
      at System.Net.Security.SslStream.ThrowIfExceptional()
      at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
      at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
      at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
      at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__69_1(IAsyncResult iar)
      at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
   --- End of stack trace from previous location where exception was thrown ---

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[msftbot[bot]]

Tagging subscribers to this area: @vitek-karas, @swaroop-sridhar, @agocke
See info in area-owners.md if you want to be subscribed.

[agocke]

I believe this is expected -- in order to publish a binary, that binary needs to be signed. Since the apphost is now part of your app you'll have to sign it with your own key.

[externl]

Which is expected, the binary not being signed at all or having an invalid signature? Which do you get?

My colleagues who try to reproduce as explained above get an unsigned binary. I get one with an invalid signature.

[agocke]

Invalid signature -- to ship the binary we have to sign it, but once it gets included in your app, the signature will no longer be valid (since it will be pointing at your code).

[agocke]

@vitek-karas to confirm my understanding here

[externl]

From my reading of CreateHostApp it should remove the signature. That's what everyone else who's tried gets.

[swaroop-sridhar]

When you build the app with UseAppHost it is expected that the signature on the apphost is removed and re-written wrt your app. That is, foo should be unsigned.

[agocke]

My mistake! Let's investigate the broken signature.

[agocke]

Also, let's see if this behavior repro's in 5.0. @externl do you happen to have a .NET 5 copy that you can build with?

[externl]

The fact that I'm getting a bogus adhoc signature kind of leads me to think there's something up with the fact that I installed the xcode 12 beta

[externl]

Also, let's see if this behavior repro's in 5.0. @externl do you happen to have a .NET 5 copy that you can build with?

Is dotnet-sdk-preview (dotnet-sdk-5.0.100-preview.5.20279.10-osx-x64.pkg) from brew fine?

[agocke]

Preview 5 is a bit old, but this code hasn't churned much so it's probably fine.

[swaroop-sridhar]

However, I wasn't able to repro the failure -- not sure how to take any further action. Can you please share the otool output for the broken binary? List of load commands to see if there's a signature LC_CODE_SIGNATURE segment? Thanks.

[externl]

Seems to work fine with .NET 5

❯ codesign -v bin/Debug/net5.0/foo
bin/Debug/net5.0/foo: code object is not signed at all
In architecture: x86_64

[agocke]

Huh. @swaroop-sridhar since CreateAppHost ships inside the SDK I assume this is fixed if the .NET 5 SDK is used to build netcoreapp3.1 apps as well?

[swaroop-sridhar]

This issue was fixed in 5, and ported to 3.1 and 2.1 2.2 releases.

[externl]

Hmm.... so, there's no LC_CODE_SIGNATURE segment in the foo binary.

❯ otool -l bin/Debug/netcoreapp3.1/foo
...
Load command 15
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 76632
 datasize 8

❯ otool -l /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-x64/3.1.7/runtimes/osx-x64/native/apphost
...
Load command 15
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 76632
 datasize 8
Load command 16
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 85120
 datasize 34592

[swaroop-sridhar]

OK @externl that's the expected behavior.
Not sure why codesign would report a bad signature on this binary. Could be a bad interaction of codesign tool with XCode?
Is this deterministic on every build of 3.1.4 SDK?

[externl]

Not sure why codesign would report a bad signature on this binary. Could be a bad interaction of codesign tool with XCode?

Possibly, but I don't know how. My codesign binary is the as my coworkers. It's not just codeisgn that's the issue. As the OS in general think it's signed as I get Console warnings about it when using SSL as described above.

EDIT: What changed between 3.1 and 5 that could cause it to work?

Is this deterministic on every build of 3.1.4 SDK?

Yea, every time with 3.1.401 SDK

[swaroop-sridhar]

Hmm.... so, there's no LC_CODE_SIGNATURE segment in the foo binary.

Can you also confirm that the total expected load commands are 15?

[externl]

Not sure if this is what you mean. But 0 - 15 are listed. I assume they're correct.

❯ otool -l bin/Debug/netcoreapp3.1/foo | grep "Load command"
Load command 0
Load command 1
Load command 2
Load command 3
Load command 4
Load command 5
Load command 6
Load command 7
Load command 8
Load command 9
Load command 10
Load command 11
Load command 12
Load command 13
Load command 14
Load command 15

[swaroop-sridhar]

I was asking to check the fact that ncmds value in the header is 16 (at the start of otool -l output)

[externl]

Ah, got it. Yes, it's 16

❯ otool -l bin/Debug/netcoreapp3.1/foo
bin/Debug/netcoreapp3.1/foo:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x80           2    16       2048 0x00a18085

[swaroop-sridhar]

Is there any binary difference between the foo executable generated on this machine vs any other machine where the signature works?

[externl]

I'll have to check with my colleagues in the morning and get back to you.

[externl]

@swaroop-sridhar The two binaries are identical (same sha), which makes it even weirder.

[externl]

Looking closer my foo binary has the same identifier as the other clients that are failing

❯ codesign -dv client
Executable=/Users/joe/src/github.com/zeroc-ice/ice/csharp/test/Ice/acm/msbuild/client/netcoreapp3.1/client
Identifier=client-555549446e4495955e3d30318046040352838d5a
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=832 flags=0x2(adhoc) hashes=21+2 location=system
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

~/src/github.com/zeroc-ice/ice/csharp/test/Ice/acm/msbuild/server/netcoreapp3.1 master
❯ codesign -dv server
Executable=/Users/joe/src/github.com/zeroc-ice/ice/csharp/test/Ice/acm/msbuild/server/netcoreapp3.1/server
Identifier=client-555549446e4495955e3d30318046040352838d5a
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=832 flags=0x2(adhoc) hashes=21+2 location=system
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

❯ codesign -dv foo
Executable=/Users/joe/Downloads/foo
Identifier=client-555549446e4495955e3d30318046040352838d5a
Format=Mach-O thin (x86_64)
CodeDirectory v=20100 size=832 flags=0x2(adhoc) hashes=21+2 location=system
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

For some reason all the binaries created from apphost seem to have the same identifier (client-555549446e4495955e3d30318046040352838d5a).

apphost itself is slightly different

❯ codesign -dv /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-x64/3.1.7/runtimes/osx-x64/native/apphost
Executable=/usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-x64/3.1.7/runtimes/osx-x64/native/apphost
Identifier=apphost-555549446e4495955e3d30318046040352838d5a
Format=Mach-O thin (x86_64)
CodeDirectory v=20500 size=988 flags=0x10000(runtime) hashes=21+5 location=embedded
Signature size=8980
Timestamp=Jul 24, 2020 at 12:31:28 AM
Info.plist=not bound
TeamIdentifier=UBF8T346G9
Runtime Version=10.13.0
Sealed Resources=none
Internal requirements count=1 size=208

The identifier number seems to be backed into apphost (I guess in its signature).

❯ rg 555549446e4495955e3d30318046040352838d5a /usr/local/share/dotnet/packs/Microsoft.NETCore.App.Host.osx-x64/3.1.7/runtimes/osx-x64/native/apphost
Binary file matches (found "\u{0}" byte around offset 5)

[externl]

@swaroop-sridhar @agocke I found the issue and how to reproduce it!

I searched my computer for 555549446e4495955e3d30318046040352838d5a and found an entry in a sqlite db at /private/var/db/DetachedSignatures.

According to this forums post

/var/db/DetachedSignatures is only used if the system has to synthesise a code signature for an app that doesn’t have one (for example, you add a firewall exception for an app that’s unsigned)

I deleted the entry for client-555549446e4495955e3d30318046040352838d5a (well, I deleted then recreated an empty file) and everything started working again. Codesign verification passed.

It's easy to reproduce the issue:

1. Enable macOS firewall
2. Create a binary like foo but one that listens on the network (not loopback but the interface ip)
3. Deny that binary when the firewall asks
4. Now codesign verification will fail for all dotnet binaries created from apphost.

I guess because of the way apphost is used to create binaries macOS is detecting anything crated from it as having the same signature?

Some thoughts:

1. This is not great because adding a firewall rule for one unsigned dotnet binary created from apphost breaks lots of things
2. I wonder if it also means that granting firewall access would allow firewall access for any dotnet binary created form apphost

EDIT: I didn't test but suspect this will also still affect .NET 5

[swaroop-sridhar]

Thanks for the explanation @externl.

Each binary like foo is typically different (not bit-equivalent) from another because it contains an embedded path to the main assembly to execute. It is rather unfortunate if blocking one dotnet app host blocks all other apps. Is it possible to add an exception for a manually-unsigned apphost executable to work-around the problem?

This is something we could discuss with the MacOS firewall team, but not sure there's anything actionable in the .net SDK.

[externl]

Unfortunately it's more than just the firewall issue though as it causes macOS to think that all the binaries are signed with an invalid signature. Even after removing the firewall rule the entry remains in the db. Any code signing verification will fail, which seems to happen indirectly sometimes, such as when using ssl/tls (see my original stack).

I also suspect there are other was than the firewall for binaries to be added to this /private/var/db/DetachedSignatures db.