Runtime: Add Mac Catalyst TargetFrameworkIdentifier or Runtime Identifier

[rolfbjarne]

I'm implementing support for Mac Catalyst in Xamarin, and I've run into a question whether a new TFI should be created or not.

First a few facts:

• Mac Catalyst is a blend of iOS and macOS (much more iOS than macOS though). We currently have two TFIs for these platforms, Xamarin.iOS and Xamarin.Mac respectively.
• The API surface that we would bind in managed code in Mac Catalyst is not identical to neither Xamarin.iOS nor Xamarin.Mac: it would contain parts of Xamarin.iOS, parts of Xamarin.Mac, and potentially API that's in neither Xamarin.iOS nor Xamarin.Mac.
• The native binaries (libmono, coreclr, etc.) have to be built specifically for Mac Catalyst. Binaries built for iOS or macOS won't work.

Adding a new TFI would have a few consequences:

• We'd have to implement support for the new TFI to many other places. NuGet comes to mind, but I'm certain there are many other places I'm unaware of. We'd also have a few TFI for .NET 6 (for instance net6.0-catalyst).
• Any existing NuGet for either Xamarin.iOS or Xamarin.Mac won't work.

Not adding a new TFI would also have a few consequences (say we re-use Xamarin.iOS):

• It would not be possible to create a NuGet that uses any API in Mac Catalyst that isn't in Xamarin.iOS.

Are there any other pros/cons?

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[marek-safar]

@terrajobst @ericstj @mhutch would you be in favour of adding a new TFM for this configuration

[Redth]

If we didn't add a new TFI, we could probably still reason about attempting to use API's with runtime exceptions.

Given that Catalyst is much more similar to iOS, would we be able to swap out API binding references based on a project property? By default you get Xamarin.iOS.dll, but if you set <MacCatalyst>True</MacCatalyst>, we instead give you Xamarin.MacCatalyst.dll which has the superset of available API's? We could then use the existing API annotations on those bindings in roslyn analyzers to advise the user of things unavailable when building with -p:MacCatalyst=False. Then the iOS TFI could pivot based on that variable and build with the correct binding dll. We could also provide #define CATALYST based on the variable. The one disadvantage here is I'm not sure how nugets could provide anything but the MacCatalyst API superset of their assemblies to iOS/Catalyst developers. We could make the attribute available so they could proactively annotate their own API's around incompatibilities, but that feels sad.

For NuGets, this would mean we can still consume existing assemblies targeting iOS, though I'm not sure if there's a great path to surfacing runtime errors if you try and call an API path in one of these existing assemblies that isn't implemented on Catalyst, beyond some unhandled exception lacking much context.

[rolfbjarne]

Given that Catalyst is much more similar to iOS, would we be able to swap out API binding references based on a project property? By default you get Xamarin.iOS.dll, but if you set <MacCatalyst>True</MacCatalyst>, we instead give you Xamarin.MacCatalyst.dll which has the superset of available API's?

Would that work with the IDEs and things like intellisense? Or would you get red squiggles whenever you tried to use a Catalyst only API?

[Redth]

Given that Catalyst is much more similar to iOS, would we be able to swap out API binding references based on a project property? By default you get Xamarin.iOS.dll, but if you set <MacCatalyst>True</MacCatalyst>, we instead give you Xamarin.MacCatalyst.dll which has the superset of available API's?

Would that work with the IDEs and things like intellisense? Or would you get red squiggles whenever you tried to use a Catalyst only API?

I think that is the intention, if you aren’t specifying catalyst, you don’t get the APIs to reference. Or are you thinking the IDE may not handle swapping references based on a property this way very well?

[rolfbjarne]

Given that Catalyst is much more similar to iOS, would we be able to swap out API binding references based on a project property? By default you get Xamarin.iOS.dll, but if you set <MacCatalyst>True</MacCatalyst>, we instead give you Xamarin.MacCatalyst.dll which has the superset of available API's?

Would that work with the IDEs and things like intellisense? Or would you get red squiggles whenever you tried to use a Catalyst only API?

I think that is the intention, if you aren’t specifying catalyst, you don’t get the APIs to reference. Or are you thinking the IDE may not handle swapping references based on a property this way very well?

It's the second part: I wonder if the IDE will handle the reference swap correctly.

[marek-safar]

What about the following? We keep an only a single version of APIs, we could keep calling it Xamarin.iOS. We annotate all APIs which are not available on the Catalyst with [UnsupportedOSPlatformAttribute ("osx")] attribute and rely on existing platform compatibility analyzer to take care of flagging the unavailable APIs to the developers.

[rolfbjarne]

What about the following? We keep an only a single version of APIs, we could keep calling it Xamarin.iOS. We annotate all APIs which are not available on the Catalyst with [UnsupportedOSPlatformAttribute ("osx")] attribute and and rely on existing platform compatibility analyzer to take care of flagging the unavailable APIs to the developers.

That could work.

We'd also have to add all the macOS API (AppKit) that are available in Catalyst but not in iOS, and mark those with [UnsupportedOSPlatformAttribute ("iOS")].

and rely on existing platform compatibility analyzer

I assume there's a way to tell the platform compatibility analyzer which platform we're building for outside of the TFM?

[am11]

fwiw, currently the granularity chosen for shipping artifacts is 'package', such that, there is no concept of 'very similar' when it comes to a consumable package; slightly different is just another package. it has some benefit at the cost of a slightly scaled bandwidth usage (which is not an issue in this day and age). e.g. in .net 5, the difference between dotnet-sdk-linux-x64.tar.gz (linux glibc x64) and dotnet-sdk-linux-musl-x64.tar.gz (linux musl-libc x64) is about 13 binary files; out of total of 3100+ files, yet they are kept separate as it is easy to tag and reason about, imo (and it goes hand in hand with tfm mappings on engineering side of the house).

[filipnavara]

If you keep a single TFM how would I actually multi-target an app for both iOS and macOS in a single build?

[mhutch]

@terrajobst @richlander

My initial feeling is that we need a new TFM. But catalyst doesn't seem particularly descriptive and may end up seeming out of date in a few years. Maybe we should use net6.0-mac-catalyst? Or given we haven't shipped net6.0-mac yet, maybe we should use that for Catalyst and use net6.0-mac-cocoa for Xamarin.Mac?

[mhutch]

We could allow net6.0-mac-catalyst to reference the legacy Xamarin.iOS and Xamarin.mac TFMs, the same way we allow net6.0-ios to reference Xamarin.iOS and allow net6.0-mac to reference Xamarin.Mac. We should _not_ however allow net6.0-mac-catalyst to reference net6.0-ios or net6.0-mac directly; any library that is rebuilt for the net6.0+ TFMs should be expected to use multi-targeting.

When the fallback is used, we should emit a warning that some APIs may not work. We will also need type forwarders so binary assembly references continue to work. And I would strongly suggest that we have separate reference and implementation versions of the catalyst platform assemblies; the reference version should contain supported APIs, while the implementation can include Mac and iOS APIs that are _not_ present on Catalyst and that throw NotSupportedException. This will ensure developers don't end up with weird MissingMethodException or TypeLoadException runtime errors when using legacy references. We could possibly also add link time errors for increased robustness.

[spouliot]

Apple defines Catalyst as a _variant_ of iOS, not macOS (just like the simulators are seen as a variant).

E.g. an .xcframework would have a ios-x86_64-maccatalyst directory for the platform specific binaries

[praeclarum]

Please no, please no, please no :-)

There is no need to fork thousands of nugets. There is no need to manage yet another project and its extensions.

It will be a tremendous amount of busy work for no gains. Take a hint from Apple and keep it as simple as a checkbox.

Instead, Platform=MacCatalyst will give us everything we want.

[spouliot]

There are some divergences between iOS and macOS API (e.g. same type, different members). I think they favour the iOS side (but with Apple it's safer to assume we'll have a mix of both).

We must also be sure that submitted applications do not refer to any symbols not available on the target platform. Otherwise Apple could reject the binaries. As such adding macOS API to Xamarin.iOS.dll can be problematic.

[filipnavara]

@praeclarum How would that work to multi-target both iOS and macOS in a single build?

It's perfectly doable to have TFM net6.0-catalyst (or different name, not a fan personally) that is compatible with both net6.0-ios and xamarin.ios as superset. That would allow the NuGets to continue to work as long as they targeted Xamarin.iOS already.

[praeclarum]

@filipnavara You’re talking about TFMs and net6.0 that don’t work/exist today. To support a scenario literally no one does today.

It wouldn’t work in a single build and doesn’t need to.

[praeclarum]

@mhutch The biggest issue these apps face is compatibility with existing libraries/nugets. You know how long it took people just to get basic Xamarin.iOS in their nugets. If you do a new TFM, it will be years before libraries are recompiled, and you will making life unnecessarily difficult for app developers and library authors.

[filipnavara]

@praeclarum Well, this issue is about TFM for .NET 6.0. I literally use the .NET 5/6 TFMs with the prerelease packages today. I previously used MSBuild.Sdk.Extras to support the same multi-targeting scenario in a similar fashion.

I do want it to work in a single build. New TFM would allow that and it would not break any of the scenarios you mentioned or require new NuGets. In fact, adding Catalyst support would be matter of changing a single <TargetFramework>net6.0-ios</TargetFramework> line into <TargetFrameworks>net6.0-ios;net6.0-catalyst</TargetFrameworks> in the csproj.

I'm operating on the assumption that Mac Catalyst is strict superset of Xamarin.iOS. Small differences can be handled by PlatformNotSupportedExceptions if necessary. As long as the relationship exists in the native code then the same rule can apply to the NuGet TFMs. A package targeting Xamarin.iOS1.0 would be consumable by net6.0-ios or the new hypothetical net6.0-catalyst.

[praeclarum]

@filipnavara I’m just begging you to consider the bigger picture. People put native code in their Xamarin.iOS nugets, with your new TFM those nugets will have to change to work. That’s going to take a long time. Please consider the people that will be using this tech.

[filipnavara]

@praeclarum If they put native code in the NuGet it would not work anyway, would it? You would still need to recompile the native code with the correct compiler flags and architecture, right? I don't see how keeping the same TFM would help you. In fact, I can see quite the opposite of that.

(And if it would work without recompilation then, again, new TFM can be compatible, just like net6.0-ios can consume xamarin.ios1.0 NuGets)

[praeclarum]

It does work! So long as it’s built with Xcode 11.3+

It’s not a new architecture, it’s a new target. It’s the the x86 64 ABI.

Please consider gaining some experience writing Catalyst apps before inflicting this pain on others.

Anyway, I’m just repeating myself and will stop arguing. PLEASE just consider the community before breaking all our apps.

[filipnavara]

@praeclarum I was following all the PRs and the work you did on the Catalyst support. I think you are still missing the point that new TFM doesn't mean that the NuGets need to be republished to support it. Look at the design document for .NET 5 TFMs. You can have a new TFM that is compatible with a previously defined TFM. Any potential net6.0-catalyst would be able to consume NuGets with xamarin.ios TFM that were produced today, as long as the TFM is declared with the correct (backward) compatibility rules. The only thing you would need to change is the TFM of the final application which is exactly one line (in new style .csproj).

[Redth]

Since Catalyst is most close to iOS (it's really just missing a handful of frameworks that work on iOS but not Catalyst), could we move the AppKit API's which _are_ supported on Catalyst into a separate NuGet package, and we leave the iOS API's which don't work annotated with attributes so analyzers from the Catalyst NuGet package would help inform usages of iOS API's that will throw at runtime when Platform==Catalyst? The Catalyst package's existence could also have analyzers to inform the opposite (when catalyst API's are used when Platform!=Catalyst).

[filipnavara]

It’s not a new architecture, it’s a new target. It’s the the x86 64 ABI.

That still sounds to me that the NuGets with native code would have to be rebuilt if they only ship iOS/arm64 native libraries today.

Right now you have a combination of TargetFrameworkIdentifier (TFM) and RuntimeIdentifier (RID). If the API surface was identical to iOS it would make sense to stick to the iOS TFM (net6.0-ios and xamarin.ios1.0). NuGets could then can ship libraries compiled against different ABI (x64/arm64) using different RIDs. And you could multi-target using a set of RuntimeIdentifiers in your app (the modern SDK equivalent of Platform/Platforms/PlatformTarget, if oversimplified).

If the API surface is different enough then it could warrant a new TFM. If you go with the new TFM then it should NOT require modifying every single NuGet out there. At minimum it should be compatible with whatever NuGets target Xamarin.iOS today. Additionally, if the API is superset of the iOS API I would expect it to be a superset on the TFM level as well, ie. hypothetical net6.0-ios-catalyst would be compatible with net6.0-ios and xamarin.ios1.0.

[Redth]

That still sounds to me that the NuGets with native code would have to be rebuilt if they only ship iOS/arm64 native libraries today.

No, most nugets that have actual native binaries in them would be fat architecture with both x86_64 and arm64 at a minimum. Nothing needs rebuilding for Catalyst if that's true (which it almost always is). Most nugets won't even have native binaries like this to begin with as well, just managed .net assemblies calling XamariniOS api's.

[Redth]

Existing NuGet compatibility really just boils down to whether they use any API's in Xamarin.iOS.dll that aren't compatible at runtime with Catalyst. As I mentioned, it's a handful of frameworks for the most part. We've already been taking existing iOS apps built against mono/mono runtime and running them with Catalyst with fantastic success.

[Redth]

Realized I failed to explain the existing x86_64 arch for those who may not know. This is what the iPhone simulator uses, so it’s rather uncommon for native libs to ship without this architecture in them or they wouldn’t work on the simulator. Catalyst apps are very very nearly iPhone simulator apps. They just have a few less iOS APIs available and a few extra ones from appkit that can optionally be used.

[filipnavara]

No, most nugets that have actual native binaries in them would be fat architecture with both x86_64 and arm64 at a minimum.

Fair enough, I forgot about fat binaries. But are the current NuGets targeting Xamarin.iOS published with those? Would that basically be the same binaries that are used with the iOS simulator today? (update: Thanks, got the answer above already while I was writing the comment.)

Note that I am not arguing to drop support for any existing NuGets, quite the opposite. I expect that supporting Xamarin.iOS TFMs without native code would be trivial with or without a new TFM. With native code it may be tricky but it depends on how the NuGets are laid out internally but that would also be problem whether a new TFM is introduced or not.

Existing NuGet compatibility really just boils down to whether they use any API's in Xamarin.iOS.dll that aren't compatible at runtime with Catalyst.

There are several different strategies on how to handle that. Ideally the iOS API would not be monolithic DLL and would be more like per-framework DLLs. Then you could exclude the frameworks from specific targets. Obviously this would be really difficult to pull off with proper backward compatibility (likely not impossible, just very hard). Another possibility is to expose the APIs from the missing frameworks but make them throw PlatformNotSupportedException. This would ideally be combined with some analyzer.

[mhutch]

Apple has the concept of a "targetEnvironment" as an additional dimension of the "target triple" compilation pivot. The API surface can differ depending on the target environment and the app can use conditional compilation (i,e, ifdefs) predicated on the environment,

The only way to do this with .NET is multi-targeting. .NET makes a distinction between the API pivots (the TFM) and the runtime pivots (the RID), and the TFM for .NET 5.0+ has intentionally been kept simple, with only three dimensions: net<netversion>-<platform><platformversion>.

I don't particularly want to add another dimension to TFMs as that would have huge complicated knock on effects, and TargetFrameworkPlatform is the existing one that's the best fit.

@praeclarum you wouldn't need more projects and forked nugets. Apps and libraries could do this:

<TargetFrameworks>net6.0-ios;net6.0-mac-catalyst</TargetFrameworks>

and my proposed back-compat would allow referencing existing NuGets.

IMO the closer we stay to how Apple does things the less likely we are run into complications in the future.

[terrajobst]

I'm not an expert on the Mac side, so I genuinely don't have strong feelings one way or the other.

But what I can do is give you a process by which you can decide whether or not you need a TFM.

1. Would you like to be able to install existing packages that don't target this new TFM into a project that uses the new TFM?
   
   
   
   • If the answer is no, a TFM is fine.
   
   
   • If the answer is yes, a TFM isn't impossible but it's a red flag that you probably shouldn't create one.
   
   
2. How much do the API surfaces differ between Catalyst and iOS/macOS?
   
   
   
   • If you expect the vast majority of iOS/macOS APIs to work in Catalyst, then I think you want to reuse the respective TFM (either macOS or iOS) and simply annotate the APIs that don't work.
   
   
   • If there is very limited API overlap, then having separate TFMs might be fine.
   
   

(1) is about desired policy, (2) is about how much the desired policy makes sense.

Stated differently, I think TFMs work great when you're modeling independent islands, for example Android vs. iOS, because there is no scenario where one needs to reference libraries from the other. TFMs are less useful when you need to model families of the same OS, because code sharing desires are usually complicated and may change over time, which often doesn't play well with TFMs because as @praeclarum outlined earlier, TFMs are very stiff b/c the ecosystem takes years to adopt them.

Assuming Catalyst is very close to a superset of an existing TFM then I'd not introduce a new TFM and simply have a single TFM and use platform annotations to allow users to write adaptive code, i.e. rely on runtime checks. It sounds a bit like as if Catalyst is a superset of macOS + iOS. If that's the case, then we're a bit screwed because reusing either TFM would still disallow the other. In that case, you have two options: declare one TFM the winner, add a compat fallback for the other, and moving forward only invest in the winner. And if you really dislike the name, you could introduce a third, deprecate the existing ones and add a fallback for both. But either way what we can't do is shipping multiple TFMs that can reference each other. The only way we can model sharing is "shared with all" (net5.0) and "shared with same TFM only" (net5.0-<os>) . Anything else is PCLs and we know it doesn't scale and is extremely complicated.

If you don't create a new TFM this would exclude multi-targeting, but I'm working on a spec to enable multi-targeting with RIDs. Regardless of your choice for TFM, Catalyst should get a separate RID. And with this feature, people would be able to write #if to provide different behaviors between them. However, the TFM choice dictates whether our not libraries can offer different API surface between macOS, iOS and Catalyst. However, library authors would still be able to indicate which of their API surface works on Catalyst/iOS and which one doesn't (via the new support attributes in .NET 5).

My gut feel (again, I'm not an expert here) is that it sounds like you may want to be able to consume existing iOS libraries, in which case I'd propose to not introduce a new TFM and simply reuse the one for iOS.

In fact, we may want to think about this scenario for all the Apple platforms. The current plan is having different TFMs for WatchOS vs iOS vs tvOS. But maybe having a joint TFM with annotations for most people, with the ability of #if with RID for advanced scenarios/native code, is a better approach?

[mhutch]

Yeah, if we have the ability to multi-target across RIDs and annotate APIs as only working on certain environments/devices (rather than just OSes) that expands the options.

[mhutch]

My _gut feel_ (again, I'm not an expert here) is that it sounds like you may want to be able to consume existing iOS libraries, in which case I'd propose to not introduce a new TFM and simply reuse the one for iOS.

Developers may also want to consume also some existing Mac libraries, hence my earlier back-compat proposal.

[Redth]

My _gut feel_ (again, I'm not an expert here) is that it sounds like you may want to be able to consume existing iOS libraries, in which case I'd propose to not introduce a new TFM and simply reuse the one for iOS.

Exactly ^

Developers may also want to consume also some existing Mac libraries, hence my earlier back-compat proposal.

It’s certainly possible but in practice I don’t anticipate that scenario to be very common. The subset of macOS appkit APIs available in Catalyst is _much_ smaller the iOS APIs and are completely optional (you don’t need any of them to build a functional catalyst app, but they can be used to enhance your app’s experience on macos)

It does sound like RID gives us the ability to have the #define we’d want and be a way to pivot the additional macos APIs available (and help annotate the iOS ones which are not).

[Redth]

In fact, we may want to think about this scenario for all the Apple platforms. The current plan is having different TFMs for WatchOS vs iOS vs tvOS. But maybe having a joint TFM with annotations for most people, with the ability of #if with RID for advanced scenarios/native code, is a better approach?

The line of supported iOS APIs becomes more blurry with watchOS and tvOS but is not totally different than catalyst. Would having RID allow us to package assemblies in nuget for each RID too?

[terrajobst]

RIDs are already available and packages can use them for multi-targeting, but today it's rocket science to make that work. Hence, we generally don't want to build experiences that hinges on 3rd parties being able to multi-target across RIDs. However, in many ways that applies to multi-targeting in general, because the complexity jumps up from a single output by quite a bit. That's partially due to lack of tooling in the UI, but also because developers need to consider compatibility between their own multi-targeted outputs, for which we don't have any validation today. The goal of my spec is to lift multi-targeting for RIDs to a point where it's as easy (or hard) as multi-targeting between TFMs. In a sense, RIDs are simply a more restricted form of multi-targeting (can't differ API surface, can't differ dependencies, etc).

Would having RID allow us to package assemblies in nuget for each RID too?

Yes, with the restrictions I mentioned above. And we have many more RIDs than we have frameworks. Basically, we have RIDs for anything that is relevant for native assets, which is all operating system and CPU architectures. So from my point of view, having a RID for Catalyst seems to be a given.

[rolfbjarne]

One thing that would complicate re-using the Xamarin.iOS TFM is if Apple adds API to Catalyst that's incompatible with iOS.

I looked through the Catalyst API, and I found at least one instance that affects the API bindings: the SslCipherSuite enum is a uint on Catalyst and ushort on iOS (https://github.com/xamarin/xamarin-macios/blob/d7bb2f2d9f2d33735b69e83dd0e8fb2425673168/src/Security/SecureTransport.cs#L278-L282): https://developer.apple.com/documentation/security/sslciphersuite?language=objc

It would not surprise me in the least if Apple were to do this in more places in the future.

[mhutch]

One thing that would complicate re-using the Xamarin.iOS TFM is if Apple adds API to Catalyst that's incompatible with iOS.

Yes, that's exactly the kind of problem I was referring to in "IMO the closer we stay to how Apple does things the less likely we are run into complications in the future."

If we use a single TFM then we're essentially building an abstraction that may not hold up in the future,

[rolfbjarne]

@praeclarum

Please no, please no, please no :-)

There is no need to fork thousands of nugets. There is no need to manage yet another project and its extensions.

It will be a tremendous amount of busy work for no gains. Take a hint from Apple and keep it as simple as a checkbox.

It's only as simple as a checkbox for the actual app project. If you want to build your library project for Mac Catalyst in addition to the iOS Simulator/device, things get complicated fast. If you want to ship your library project for multiple platforms (say Mac Catalyst, iOS Simulator + iOS device), you can't even use Xcode, you have to drop down to the terminal.

People put native code in their Xamarin.iOS nugets, with your new TFM those nugets will have to change to work.

People will have to rebuild/repackage their native code anyway, it's not possible to re-use native libraries built for the iOS Simulator on Mac Catalyst.

You'll get this error if you try:

error: Building for Mac Catalyst, but the linked library 'libsimlib.a' was built for iOS Simulator.

I understand it can be painful to rebuild NuGets for yet another TFM, but at this point I'm not sure we have a choice, because as far as I understand from other people's comments, that's the only way to offer an API that's (ever so slightly for now) incompatible with Xamarin.iOS.

[coolbluewater]

[Edited for brevity.]

Let's note that a "Xamarin.iOS" project today includes iOS as well as iPadOS. iPadOS was introduced late in the game and I don't think there are API differences yet, but there will be in the future. Nevertheless, Xamarin.iOS has always exposed and supported options to build for iPhone, iPad, or both (Universal.) The Xamarin build system knows how to package the binaries as well as iPhone/iPad-specific assets and Info.Plist to create a universal binary.

About Mac Catalyst: According to Apple it lets you Create a version of your iPad app that users can run on a Mac device.. In other words, it is a variant of iPadOS that includes some macOS features.

It is correct and consistent with Xcode, therefore, to further extend the meaning of "Universal" to include Mac Catalyst apps. And it should be exposed via a single checkbox. That is, a single .csproj that can build an app that deploys to the iPhone and/or iPad and/or Mac Catalyst. Why would we want to make it more heavy weight than this? We shouldn't. The existing .csproj has all the information we need. We just need to add a setting for enabling Mac Catalyst.

I understand it can be painful to rebuild NuGets for yet another TFM, but at this point I'm not sure we have a choice, because as far as I understand from other people's comments, that's the only way to offer an API that's (ever so slightly for now) incompatible with Xamarin.iOS.

Is the result going to be a single csproj and a single nuget package with all the needed binaries, including separate binaries for Mac Catalyst? If that's the case then it's a detail that is invisible to the developer. On the other hand if you mean that there will be separate nugets for vanilla Xamarin.iOS and for Mac Catalyst, I don't understand why. If the API being exposed by the library is consistent across these platforms, then there shouldn't be the need for a separate nuget. We have plenty of examples of nuget libraries that have platform-specific implementations but yet offer up the same API across the platforms they support. I don't think nugets with different APIs between Xamarin.iOS and Mac Catalyst is a valid use-case.

[Mogikan]

It does not sound great to rebuild all nuget packages. Let's take as an example abandoned https://github.com/xamarin/SignaturePad. Owner can not push through pull requests with WPF and MacOS support and unlikely to rebuild nuget with new TFI.

[dalexsoto]

If the API being exposed by the library is consistent across these platforms, then there shouldn't be the need for a separate nuget.

Unfortunately this does not hold true for Apple SDKs, for example this was reported in our discord channel NSTextAlignment the value depends on the platform it is run and @rolfbjarne already pointed out above another case, it seems that the only way to safely future proof is really having a different TFI, the vast majority (if not all) the nuget packages that contain any native libraries will need to be rebuilt anyways unfortunately.

[coolbluewater]

@dalexsoto - that's a platform difference, not a library difference. By library, I mean the one that the developer is creating.

For concreteness, we have three "platforms" - iOS, iPadOS, and Mac Catalyst. The first two are served by the current Xamarin.iOS as noted. What does the proposed csproj look like in the case where I want to build a) a new nuget library and b) an app, that both deploy to all three "platforms"?

[dalexsoto]

For concreteness, we have three "platforms" - iOS, iPadOS, and Mac Catalyst.

Kind of, you are leaving away tvOS and watchOS which both have their own Xamarin.*.dll all of these platforms share some APIs that is true but tvOS and watchOS have some platform specific frameworks. iPadOS is just a marketing thing from Apple, it still identifies itself as iOS but Mac Catalyst isn't and it has kind of moved away from just a checkbox this year:

One of my fears is that Apple goes beyond next year and they could have a Mac Catalyst only framework or introduces Mac Catalyst platform APIs or removes already exposed API (for example) from UIKit only in the MacCatalyst context, these would pollute Xamarin.iOS.dll API surface which we could annotate but as developers already found out even if the current API is exposed it does not mean it works so nothing stops Apple on creating breaking changes. If I understand correctly by having a different TFM and a Xamarin.MacCatalyst.dll would allow us to accommodate such changes.

There is a big warning there already from their docs

Mac apps built with Mac Catalyst can only use AppKit APIs marked as available in Mac Catalyst, such as NSToolbar and NSTouchBar. Mac Catalyst doesn’t support accessing unavailable AppKit APIs.

About how csproj would look I am not completely sure, this reminds me a lot about the "Shared Projects" we used to have to share code between iOS or Android back when Nuget profiles was a thing...

@coolbluewater nice points btw :)

[coolbluewater]

One of my fears is that Apple goes beyond next year

@dalexsoto - why is this a problem? I think its a matter of realizing that msbuild and nuget already allow for all this variation.
Nothing new needs to be invented. From the point of view of the machine, it is completely different code that runs on the simulator and device, for example. The constructs of "library", "app" and "project" are fluid and should correspond to how the developer thinks of these, or the result is cognitive dissonance.

When I think of a Xamarin.iOS project I know that there are features that are unsupported on the iPhone that are supported on the iPad, so my code adapts appropriately. It is fine even for APIs to differ in signature, such as NSTextAlignment - msbuild and nuget have no difficulty with compiling & packaging different outputs for each of the platforms that live under the umbrella of a) the same nuget library or b) the same "app" that can be built for different targets. If a build break occurs as you describe, it will occur only for the platforms on which compilation failed. Then the developer knows to adapt their code to platform differences, by #if statements or any other conditional construct, including conditional statements in the .csproj file.

In other words, we should look at nuget libraries and apps as being umbrella concepts, which is exactly what they are. No new mechanism is needed for multi-targetting, IMHO. In fact this ought to have always included other platforms as well, so that a single csproj could be used for a library or app that targets desktop, Xamarin.iOS and Xamarin.Android. That's what MAUI is moving towards, and what the rust world already has in cargo.

I really think that we can do this today. The underlying tooling has the necessary smarts, but the top-level tooling - the Xamarin msbuild scripts and Visual Studio - prohibit this because they don't duck-type what it means to be a library or an app. We're just not using all the expressive power that the underlying msbuild platform and nuget allow.

[marek-safar]

NSTextAlignment the value depends on the platform it is run ..... it seems that the only way to safely future proof is really having a different TFI,

I don't think the change in enum value requires new TFI, it's an implementation detail that can be covered by different RID for catalyst platform and there will be one.

In fact, we may want to think about this scenario for all the Apple platforms. The current plan is having different TFMs for WatchOS vs iOS vs tvOS. But maybe having a joint TFM with annotations for most people, with the ability of #if with RID for advanced scenarios/native code, is a better approach?

I think having a single TFM would give the developers the most value. I understand this would require more work on our side but having a uniform way to target all apple "modern" platforms would be a big benefit to developers, especially libraries authors.

[rolfbjarne]

I understand this would require more work on our side but having a uniform way to target all apple "modern" platforms would be a big benefit to developers, especially libraries authors.

Won't that require library authors to build their libraries (even the purely managed ones) for each RID instead of for each TFM?

[coolbluewater]

@rolfbjarne, as an example, what would the csproj(s) look like for a) a Xamarin.iOS app and b) a Xamarin.iOS nuget library, each of which runs under iOS, iPadOS and Mac Catalyst, under the two approaches (RID vs TFM)?

[marek-safar]

Won't that require library authors to build their libraries (even the purely managed ones) for each RID instead of for each TFM?

No, it's similar as you build .net5 libraries today. You build for .net5 TFM even though we have hundreds of RIDs for os/arch combinations.

[rolfbjarne]

@marek-safar I don't see how building once can work when you have incompatible API between RIDs:

NSTextAlignment the value depends on the platform it is run ..... it seems that the only way to safely future proof is really having a different TFI,

I don't think the change in enum value requires new TFI, it's an implementation detail that can be covered by different RID for catalyst platform and there will be one.

Enum values can't be an implementation detail, because they're read from the reference assembly and baked into the compiled assembly.

[rolfbjarne]

@coolbluewater There's not much difference. Note that there's no separate RID or TFM for iPadOS, it's just iOS.

It would be something like this (for .NET 6):

Different RIDs, same TFM

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net6.0-apple</TargetFramework>
    <RuntimeIdentifiers>ios-x64,ios-arm64,maccatalyst-x64,maccatalyst-arm64</RuntimeIdentifiers>
    <OutputType>Exe</OutputType>
  </PropertyGroup>
</Project>

Different TFMs (exactly how to match RIDs with TFMs hasn't been figured out yet afaik, so there may be slight changes).

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net6.0-ios,net6.0-maccatalyst</TargetFrameworks>
    <RuntimeIdentifiers>ios-x64,ios-arm64,maccatalyst-x64,maccatalyst-arm64</RuntimeIdentifiers>
    <OutputType>Exe</OutputType>
  </PropertyGroup>
</Project>

for a library project change OutputType to Library.

[mhutch]

@coolwaterblue Xamarin "Universal" apps don't allow you to use #if. They perform a single C# build, then run the AOT managed->native compiler multiple times and package up the resultant output.

When Xamarin is part of .NET 6, its frameworks and targets will be usable in modern "SDK-style" project files. That means you'll be able to use these frameworks with .NET multi-targeting. Multi-targeting allows you to build code for multiple platforms froma single project, and is handled transparently by NuGet.

For example, a csproj with <TargetFrameworks>net6-ios;net6-android</TargetFrameworks> will be built twice, once using the iOS targets and APIs and once using the Android targets and APIs, producing two dlls (or apps). When it's packed with dotnet pack, both dlls will be packed into a single NuGet. A project that references this NuGet will get the appropriate version of the dll.

Adding a new TFI for Catalyst would allow libraries that target Catalyst to participate in this, e.g. net6-ios;net6-catalyst.

TBH we should also have a TFI for iPadOS, but multi-targeting wasn't a viable option at the time iPadOS support was shipped.

[mhutch]

I would also reiterate that no-one is advocating having to rebuild existing iOS NuGets. We can add a back-compat code path to allow consuming old iOS NuGets from Catalyst builds, as described in this comment: https://github.com/dotnet/runtime/issues/44882#issuecomment-730492281

[akoeplinger]

no-one is advocating having to rebuild existing iOS NuGets

... for NuGets that don't contain native binaries, those that do will need to recompile to support Catalyst because the native binaries aren't compatible as Rolf mentioned in https://github.com/dotnet/runtime/issues/44882#issuecomment-732427710

[coolbluewater]

@coolwaterblue Xamarin "Universal" apps don't allow you to use #if. They perform a single C# build, then run the AOT managed->native compiler multiple times and package up the resultant output.

To support Mac Catalyst before .net 6 are there any workarounds for this? Also @rolfbjarne mentioned:

Enum values can't be an implementation detail, because they're read from the reference assembly and baked into the compiled assembly.

That indicates that nugets have a distinguished reference assembly. I'm assuming that in .net 6 this is no longer the case, correct?

From the pent up demand as well as the work by @praeclarum waiting for .net 6 to enable Mac Catalyst builds would be highly disappointing. It would also say much about the agility of the .net/Xamarin platform. So can we make something happen in a week or two? It has already been a year and a half since Catalyst was released, and Xamarin's motto used to be about having bindings available on day one of Apple's releases.

[rolfbjarne]

Let's say we re-use the Xamarin.iOS TFM for Mac Catalyst.

• We already know there are APIs that are incompatible between iOS and Mac Catalyst. Let's assume we can figure out a way around those, and hope that Apple doesn't make this too difficult in the future.
• It would likely not be feasible to add the API specific to Mac Catalyst to a different assembly, because the Mac Catalyst specific API isn't limited to new types, there are also added members to existing types. Having some Mac Catalyst-specific API in a different assembly and some in the existing Xamarin.iOS.dll just becomes confusing (and quite complicated if the API in Xamarin.iOS dll needs to reference types in the other assembly, because then you end up with circular references). This means we'll have to put all the API, for both iOS and Mac Catalyst, in Xamarin.iOS.dll.
  
  
  
  • This has consequences for builds published to the App Store: Apple will most likely reject iOS apps that contains Mac Catalyst-only API. We'd have to implement a way to detect and remove any usage of API that's not available on the target platform (depending on the linker is not feasible: it's too easy to end up referencing API that isn't available - and this may happen without direct usage from the developer, so there would be no compiler warnings).
  
  
• NuGets built for Xamarin.iOS with only managed code should work as-is (assuming they don't use API that is iOS-only of course).
• NuGets built for Xamarin.iOS that contain native code must be rebuilt and republished.
  A. What happens if a new project that targets Mac Catalyst tries to use a NuGet that doesn't have native code for Mac Catalyst (i.e. a project builds for a RID that a referenced NuGet doesn't have)? Is there a helpful build error ("NuGet 'MyLibrary' doesn't contain native bits for "?) Or would it just fail mysteriously at runtime?
  B. Is it possible to search for NuGets that support a specific RID? Or would I have to try them out to see which ones work and which ones don't?
  C. A related question would be if it's possible to declare a NuGet as iOS-only or Mac Catalyst-only (if I knew that it would only work on one of them)?

[coolbluewater]

@rolfbjarne,

Let's say we re-use the Xamarin.iOS TFM for Mac Catalyst.

I'm assuming you mean in the pre-.net 6 era, is that right?
It looks like .net 6 will have the features needed for a clean implementation of Mac Catalyst in Xamarin. Therefore, we'll need to be clear which release - .net 5 or .net 6 - we're referring to at any moment.

For .net 5, I'd personally be OK with many simplifications, including no extra support for nuget at all. That is, Just getting a Mac Catalyst .exe project building will work for me. But others might differ on this point. I don't know how much this affects Xamarin Forms, for example. Catalyst should be able to translate the standard Xamarin Forms code into a vanilla experience. (More on this below)

I'm also ok if in .net 5 we need a separate .csproj for a Mac Catalyst app, assuming it can coexist in a solution with the iOS/iPadOS app, and that Mac Catalyst will appear in the device drop-down when the Catalyst project is selected.
If we have a separate csproj for the Mac Catalyst app in .net 5, then we can use conditional #if's to light up Mac functionality. In particular developers can add custom renderers for Xamarin Forms controls for Mac Catalyst right into their app's source code. (This code might be supplied by Xamarin as source). Not pretty, but it will work in the short term until .net 6 is released.

This means we'll have to put all the API, for both iOS and Mac Catalyst, in Xamarin.iOS.dll.

I was starting to realize this as well. The NSTextAlignment issue is kind of tricky, what workaround do you see?

To recap, the train of Mac Catalyst related features in the .net 5 timeframe might be:

1. Very near term: Allow Mac Catalyst .exe's to be compiled via a separate .csproj, with no special library support. This means that only all-managed apps will work at this stage. But it's a good first step and allows something real to be delivered soon.

1. Longer term but still .net 5: Support for nugets that have Mac Catalyst specific code, and native code. (Depending on the demand for this feature.)

And in .net 6 we can look forward to a single multi-targetted .csproj for .exes and libraries/nugets that works across iOS, iPadOS and Mac Catalyst and supports conditional code as well as arbitrary API differences.

Thoughts?

Edit: the above assumes that the .net 6 multi-targetting build system will be a long time coming. On the other hand if it is already in good shape and can be made available to .net 5 in the near term, then of course that is the way to go.

[Redth]

There is no Xamarin support in .NET 5, and there will not be. The existing runtime for Xamarin will continue to be supported into .NET 6 where Xamarin support will actually land.

[coolbluewater]

There is no Xamarin support in .NET 5, and there will not be. The existing runtime for Xamarin will continue to be supported into .NET 6 where Xamarin support will actually land.

@Redth - .net 6 is a year out, is it not? That would make Mac Catalyst support occur 2.5 years after the fact. Or in other words, 30 months late.

I beg to ask, what led to this unfortunate decision? This is not the time to remain silent.

https://github.com/xamarin/xamarin-macios/issues/6210 was filed in June 2019, and went largely ignored despite people pleading desperately for this. @rolfbjarne did note in September 2019 that

it's the top request from the new iOS 13 features, so it will be prioritized accordingly.

Since then, nothing.

@praeclarum went so far as to produce a working script for mono. In an interview he had with James Montemagno, Montemagno continued to be opaque about why Xamarin had ignored this.

Open source is not just about having the code on github. It ought to be about transparency and clear roadmaps, should it not? Why is this the first time we're hearing a delivery schedule, one that is completely at odds with customer expectation and all prior Xamarin messages about day one delivery?

Apple has done the hard work of adapting iOS apps to run as Mac apps. @praeclarum did what nobody in Xamarin cared to do.

Adding Mac Catalyst to Xamarin is relatively simple now, given that it is standing on this work. The customer benefit exceeds anything else in Xamarin at the moment (or please prove this wrong.)

A terse one line "press release" just does not cut it, I'm afraid. What’s more, the high handed tone of your reply is a slap in the face to all who took a bet on Xamarin over the years and have waited as their competitive advantage slips steadily away.

@migueldeicaza, is this what your vision for Xamarin has turned into? No Mac Catalyst, no SwiftUI, no WidgetKit, and on and on? And employees who haven’t the slightest idea about participating in a community? if I’ve said anything incorrect, please correct me. But don’t let this once-promising technology slip into incompetence.

[rolfbjarne]

There is no Xamarin support in .NET 5, and there will not be. The existing runtime for Xamarin will continue to be supported into .NET 6 where Xamarin support will actually land.

@Redth - .net 6 is a year out, is it not? That would make Mac Catalyst support occur 2.5 years after the fact. Or in other words, 30 months late.

Mac Catalyst is being implemented for both the current Xamarin.iOS, as well as .NET 6.

[coolbluewater]

@rolfbjarne, so @redth is not a Xamarin employee? Understood.

Mac Catalyst is being implemented for both the current Xamarin.iOS, as well as .NET 6.

Here's how https://github.com/xamarin/xamarin-macios/issues/6210 went:

@mrwcjoughin on Jul 8, 2019: Any news on the Catalyst support? This is urgently needed please

_Wait two months..._

@gdignard on Sep 15, 2019: Following up on this. It's over 3 months since the announcement and as I write this we are now only a couple of days from public release of iOS 13 and I still have no answer for my clients who are (reasonably) wanting to know whether I'll be able to deliver to them applications with all features and functions of the new iOS release.

@rolfbjarne on Sep 17, 2019: @gdignard Our intention is to implement this, but unfortunately we've not had enough time to complete the work for the initial public release of iOS 13. Neither am I able to say when it will be implemented, but it's the top request from the new iOS 13 features, so it will be prioritized accordingly.

@Uncommon on Oct 4, 2019: Any updates on this, now that iOS 13 support is done and the macOS 10.15 GM seed is up?

@rolfbjarne on Oct 5, 2019: @Uncommon I have no further news than in my last comment, our intention is to implement this, but we have no timeline yet.

_A month later..._

@sichy on Nov 2, 2019: @rolfbjarne what is needed for the catalyst binding? am happy to do it, as we need it ourselves. Let me know mate.

@rolfbjarne on Nov 5, 2019: This is a rather complicated task, here's a rough outline of the bare minimum of what needs to happen in order to make something that can be used to create Catalyst apps:
Build the mono runtime for the uikitformac platform. I don't know how to do this, the mono runtime people would probably have to be involved somewhat.
Build the code in xamarin-macios/runtime for the uikitformac platform (this won't fully work without the previous step - but I have this almost done already, it just needs the previous step completed first)

_Wait another 45 days_

@mandel-macaque on Dec 18, 2019: @rolfbjarne have we talked with the mono team already?

@rolfbjarne on Dec 18, 2019: No, I have not.

_Wait another two months_

@praeclarum on Feb 15, 2020: I have been waiting patiently for this feature and am still shocked that work hasn’t even begun.

_Wait 9 months, because why not?_

No more comments from @rolfbjarne until the current issue was opened 15 days ago, after @praeclarum had submitted his script that does the heavy lifting to build mac catalyst apps in Mono.

Is it any wonder that we take these promises with a blue whale unit of salt?
Dates, please.

@migueldeicaza

[Redth]

@coolbluewater sorry, let me be more clear, and then let's try to keep this issue related to the original intent of the conversation, which is, do we need another TFM or RID or ? for .NET 6 support of Catalyst.

First of all, I wanted to be sure it was clear there is no Xamarin support coming to .NET 5 whatsoever. We will of course continue to support Xamarin.iOS, Xamarin.Mac, and Xamarin.Android using the existing mono based runtime and BCL in the interim. Originally it was planned for Xamarin to support .NET 5 but due to various external and internal factors it was decided and communicated some time ago to wait for .NET 6.

As for Catalyst, we know there is a desire for it to be supported. I have been eager to be able to use Catalyst myself since it was announced. I've also been a very big advocate (aka real big pain!) for its support internally and have been working hard to see it happen, as I believe it makes a LOT of sense to use this as our approach for Xamarin.Forms to be supported on macOS going forward.

@rolfbjarne has been working on adding support for Catalyst via the existing mono runtime and BCL which allows us to work on it in parallel to things needed in .NET6 for the effort (like this TFM/RID decision), however at this time we are not committing to making Catalyst a fully supported feature of the product until .NET 6. I don't see a reason why we wouldn't continue to release the work Rolf is doing for it as is, and perhaps even gate it as a preview or experimental feature so that anyone who cares to try it is welcome to, and may very well be successful with it, but fully supporting the scenario is another conversation with other implications. This may or may not change in the future.

[coolbluewater]

@Redth, first off could you please describe your role?

Edit: Ah, I see - I was using the term .net 5 generically to include everything pre-.net 6, while you are referring specifically to the non-mono implementation. A case of too many .nets. Does that help clarify what I meant?

[filipnavara]

.Net 5 is already released, and Xamarin runs atop it. So whatever you are saying needs to be more descriptive.

No, it does not. At least not Xamarin.iOS/Mac/Android.

[coolbluewater]

@filipnavara, see my edit above.

[Redth]

@coolbluewater I'm the engineering lead/manager of the Xamarin.Forms team (Maui, Essentials, SkiaSharp, and others). I'm also coordinating the Xamarin SDK effort for .NET 6 support. I'm also not a product manager. @davidortinau would be a good person to comment further on any Catalyst support pre .NET 6.

So, yes I think using .NET 5 to indicate pre-.NET 6 is confusing especially in the context of Xamarin.

Again, at this point there's no real commitment to release Catalyst support before .NET 6 as a supported feature. But of course since it's being worked on in the open, on top of the mono based Xamarin runtime/BCL, it may be usable with the disclaimer of "as-is".

[coolbluewater]

@Redth, I understand your role now.

Again, at this point there's no real commitment to release Catalyst support before .NET 6 as a supported feature.

That's no more useful than your previous one-liner. And this really is the right place and time to discuss why engineering was unable (or chose not to) to deliver a Mac Catalyst implementation, and more importantly why it will take another whole year to have something usable.

If you don't wish to discuss this that's your choice. But you've said nothing about the issues raised regarding the response to https://github.com/xamarin/xamarin-macios/issues/6210.

What's being discussed here is completely on-topic, and is the context of the particular issue. Is quibbling about where the discussion occurs really more important to you than to just engage with this discussion as-is?

Also - the build system for building Xamarin.iOS apps at this point is in fact .net 5, correct? That's what I was referred to when I stated this:

Edit: the above assumes that the .net 6 multi-targetting build system will be a long time coming. On the other hand if it is already in good shape and can be made available to .net 5 in the near term, then of course that is the way to go.

So I think that was the right terminology after all. What you're saying is a categorical "no" to having a dot release that includes what might be needed in the build system for Xamarin to support Mac Catalyst. But you haven't explained why.

Microsoft (and now Xamarin) have made the mistake time and again of these gigantic releases that everyone has to wait for and that slip with near certainty. .net 5 => .net 6 was a case in point. (And please don't bring up the pandemic, Apple had its most innovative year in this period. Just anticipating your response; I am not given to sarcasm.)

It seems to me that the build system ought to be a small, tight component that can ship on a dime, and that in itself needs only the most basic features of the runtime. If this isn't the case it seems that a high tech debt has built up that is crippling the rate at which you can move forward.

[filipnavara]

Also - the build system for building Xamarin.iOS apps at this point is in fact .net 5, correct?

Nope. Not at all. It is the Mono desktop MSBuild based system that is totally unrelated to the .NET 5 code base.

What you're saying is a categorical "no" to having a dot release that includes what might be needed in the build system for Xamarin to support Mac Catalyst.

I am reading it differently. It could very much be shipped as part of classic (non-.NET 6) Xamarin packages as a preview feature.

[gdignard]

I would point out obvious that there's a very real value of having a feature like this appear as early as possible, even with disclaimers of "preview" / "as-is" / " WARNING: Operating this software can expose you to unimplemented features which are known to the State of California to cause machine crashes."

Having it in a non-publishable state is still of enormous value as the development process can begin even if things aren't stable. As was noted, as part of the .NET 6 release, it puts Xamarin developers at a 30 month disadvantage. Having access to Catalyst sooner than that would at least put us in a position of having software ready to go when there's a GM rather than having the race start then for us.

I'm very happy to see the work underway and for the traction it's finally starting to receive.