Core: Manage .NET's native dependencies

[mthalman]

Note: This is a proposed experience for .NET 6 and not yet committed.

Situation

There are a set of .NET-related customer workflows that require the use of native libraries contained within the operating system. These are divided into two categories:

• Execution of a .NET process: Runtime components defer some of their logic to native libraries. Examples include ICU libraries for globalization support, GNU C Library, and others.
• Building .NET from source code: The toolchain required to build the .NET source consists of native dependency prerequisites on the build machine. Examples include CMake, curl, Python, and others.

Clearly, these native libraries need to be installed in order to execute these workflows. This would be a reasonable expectation of developers if the set of dependencies was small, static, and well-documented. The problem is that these dependencies vary based on usage, they change as the product and the dependent ecosystem evolve, and the documentation is often out-of-date with the product [1]. This leaves the community with few options other than an iterative and error-prone approach of attempting to execute their workflow and finding what breaks due to missing dependencies [2]. Even worse is when there are cases where the community has discovered incompatible dependencies in environments that are supported [3].

Objective

We need to provide a better user experience for those developers that continually encounter issues of missing native dependencies in their systems. By providing .NET developers access to self-service tools that consume an up-to-date and comprehensive database of native dependency information, we can help these developers to accurately configure their systems.

We also need systems in place to guard against changes to native dependencies that either break workflows or cause inconsistencies in product assets.

While these experiences are distinct, there is enough commonality in the requirements that allows us to achieve a single comprehensive solution that addresses both of them. This can be done by producing a layered set of multi-purpose deliverables.

Execution

The first priority is to get an accurate accounting of what native dependencies the runtime and build toolchain workflows have [4]. This information will be encoded in a machine-readable format to provide an accurate picture of the native dependencies across all supported platforms and versions. Along with tooling, this will help to ensure we keep .NET product assets up-to-date wrt to those dependencies [5]. An example of this would be tooling which transforms this data into documentation to published to dotnet/docs.

Next, we will assist .NET contributors by alerting them when changes have been made to the native dependencies. This can be implemented in a crawl-walk-run fashion as the system evolves with the following examples:

• Crawl: Define a written policy for how native dependencies are to be managed for the product. Contributors are made aware of the policy.
• Walk: Bot automatically adds a "Native Dependency Change" label to a PR when it detects a change. Contributor must manually update native dependency description to reflect the change.
• Run: Updating of the native dependency description is generated automatically by merged PRs. Stakeholders are automatically notified when dependency changes are proposed.

Finally, we will assist .NET developers by providing them the ability to determine their application's native dependencies through a website and tooling. The website would allow users to search and browse the database of dependencies with relevant links to the various package repository websites. A dotnet global tool will be made available that can analyze .NET projects to determine their native dependencies.

Footnotes

1. While the primary set of runtime native dependencies are documented, it doesn't provide a full accounting of all dependencies. Even worse, the documentation can become out-of-date compared to what the product actually requires (see https://github.com/dotnet/docs/pull/18989).

2. Developers are often left to discover what dependencies are required by running their app and seeing what breaks (examples: https://github.com/dotnet/runtime/issues/36888#issuecomment-633220620 and https://github.com/dotnet/dotnet-docker/issues/1767). This is a tedious and error-prone process. This experience could be improved if the developer could know exactly what native dependencies are required without even needing to run their app.

3. Source build toolchain dependencies are another set of dependencies that require careful management. There were recent changes that are made which caused the required version of cmake to be updated such that .NET could not be source built in many Linux distros (https://github.com/dotnet/runtime/issues/38755). This required significant effort to revert this dependency. Capturing these kinds of dependency changes early can help to reduce unnecessary work and hopefully not release such changes in the wild.

4. A recent breakdown in understanding what native dependencies we have is https://github.com/dotnet/aspnetcore/issues/27950 where it wasn't known we even had a dependency on the procps package.

5. In addition to documentation, there are a variety of product assets that need an accurate reflection of the required dependencies:

• deb packages (https://github.com/dotnet/installer/tree/master/src/redist/targets/packaging/deb)
• RPM packages (https://github.com/dotnet/installer/tree/master/src/redist/targets/packaging/rpm)
• Snap packages (https://github.com/dotnet/runtime/tree/master/src/installer/snaps)
• Docker images (https://github.com/dotnet/dotnet-docker)

Keeping these up-to-date is key to maintaining a functioning product. When product dependencies are changed or changes are made externally by the operating system or ecosystem (such as Linux package versions), these changes need to be coordinated across all product assets.

User Stories

User Stories under this Epic:

• [ ] [Describe native dependencies](https://github.com/dotnet/core/issues/5646)
• [ ] [Client library for reading native dependency data](https://github.com/dotnet/core/issues/5647)
• [ ] [Native dependency change detection](https://github.com/dotnet/core/issues/5648)
• [ ] [Website to browse native dependency data](https://github.com/dotnet/core/issues/5649)
• [ ] [Global tool for discovering an app's native dependencies](https://github.com/dotnet/core/issues/5650)

[dleeapho]

/cc @shawnro

[danmosemsft]

@mthalman @dleeapho this was missing a parent -- I put it under https://github.com/dotnet/core/issues/5437. Feel free to correct if I'm wrong.

[mthalman]

@danmosemsft - There's not yet a theme for this and may not be. It's part of the Engineering Services work. @shawnro can provide more info.

[danmosemsft]

Ah, OK. If it doesn't connect to a theme you may want to label it "bottom up work". Please do detach from #5437 if its not part of it

[omajid]

Thanks for creating this! I had a couple of thoughts in no particular order:

• For things like the C library, there are choices: users/developers can either use glibc or musl. But those are exclusive and shouldn't be mixed for one build. We will probably need some way to express these mutually exclusive dependencies.

• There's a note about updating stakeholders when dependencies are changed. I wonder if it would be better to automate even more: define the dependencies available in platforms and warn if updating dependencies breaks compat with those defined platforms.

• For platforms that are not supported - such as a Linux distribution not in the "supported list" - maybe the tooling can offer generic advice? Like OpenSSL 1.1 instead of a distro-specific package name?

• After skimming https://github.com/dotnet/runtime/issues/36888#issuecomment-633220620, I am wondering how we might go about "full" vs partial dependencies. Which dependencies are listed as required and which are lightup or optional-depending-on-use-case?

[mthalman]

After skimming dotnet/runtime#36888 (comment), I am wondering how we might go about "full" vs partial dependencies. Which dependencies are listed as required and which are lightup or optional-depending-on-use-case?

@omajid - Yes, that's been a goal of mine as well. I think this is particularly relevant for scenarios like diagnostics where a developer wants to make use of diagnostic tools in which case LTTng would be listed as an optional library, for example. My intention was that requirements would be described for the core runtime as a whole and then other NuGet packages would each have their own set of distinct requirements. So while it's optional to make use of the System.DirectoryServices.Protocols NuGet package, libldap would be listed as a requirement of that package.