Source-build: [release/5.0] Drive prebuilts down to 0

[omajid]

For .NET 5.0, we would like to get to a place where there are 0 prebuilts.

At that point, a source-build built .NET 5.0 SDK can be used to build a subsequent .NET 5.0 SDK without using any prebuilt binaries (SDK, nuget packages, tools, etc).

This allows source-build consumers (such as Linux distributions) to bootstrap .NET 5.0 SDK once, and then use that to build further versions of 5.0 without needing any prebuilts. This makes it possible to package .NET 5.0 SDK in Linux distributions where prebuilt binaries are disallowed (such as Debian and Fedora).

This is similar to what's already available for the release/3.1 branch in source-build.

I know that this is a shared goal/understanding among many of the folks working here, but I couldn't find it explicitly documented. So I am filing this issue for awareness.

---

Latest list of prebuilts: https://github.com/dotnet/source-build/wiki/Driving-to-zero-prebuilt-dependencies-in-5.0.

If you'd like to own any, please comment with the prebuilt(s) you'll investigate.

[omajid]

cc @tmds

[dagood]

I think https://github.com/dotnet/source-build/issues/1577 was intended to track this, but it's not totally clear. /cc @dseefeld

[dseefeld]

I think #1577 is to add back the check to ensure that the prebuilts that we have don't grow or change. This issue is more explicit about getting to zero. Let's keep this one to drive the prebuilt removal work.

[tmds]

@dagood @dseefeld I'd like to understand this better. Can you explain how you do this? What tools/steps are involved? Is there some documentation about this?

[dagood]

Since we expect 0 prebuilts, we basically need to turn this back on, and follow the errors:

https://github.com/dotnet/source-build/blob/26531ea7074211d64f4ff2a696a0b483319aecf7/.vsts.pipelines/jobs/ci-linux.yml#L119

There is a decent amount of useful info that the build saves to track down individual prebuilt usage errors, in addition to the usual binlogs. This is done in the production build (where some baseline of prebuilts is always expected due to circular references) and tarball (where 0 are expected):

#> find artifacts/prebuilt-report/ -type f
artifacts/prebuilt-report/prebuilt-usage.xml
artifacts/prebuilt-report/annotated-usage.xml
artifacts/prebuilt-report/generated-new-baseline.xml
artifacts/prebuilt-report/snapshots/PackageVersions.props.pre.application-insights.xml
artifacts/prebuilt-report/snapshots/PackageVersions.props.pre.sourcelink.xml
...
artifacts/prebuilt-report/snapshots/PackageVersions.props.pre.templating.xml
artifacts/prebuilt-report/snapshots/PackageVersions.props.pre.installer.xml
artifacts/prebuilt-report/baseline-comparison.xml
artifacts/prebuilt-report/all-project-assets-json-files.zip

Removing prebuilt usage can be accomplished by:

• Adding patches to remove parts of the build that shouldn't be running at all (e.g. tests)
• Adding packages to dotnet/source-build-reference-packages if they are only compiled against, never run
• Fixing parts of the build that should be producing a certain package but aren't
• Fixing parts of the build where packages aren't flowing properly to a downstream repo (Package Version Props issues)
• Adding new repositories to source-build to produce packages that must be source-built

We did have long-term tooling set up to track progress in the initial burndown for 3.1, but I don't think that will be necessary: we know 3.1 works with 0 prebuilts, so I think this will mostly be going after small regressions. (And adding Cecil https://github.com/dotnet/source-build/issues/1537.)

[omajid]

@dagood Is there something I can do here without stepping on others' toes? I want to help out but I have seem commits that indicate others are already working on this too. Is there some low hanging fruit that others are not already working on?

[dagood]

I think adding jbevain/cecil is pretty independent and would be very useful: https://github.com/dotnet/source-build/issues/1537. Thanks for the offer. 😄 PR https://github.com/dotnet/source-build/pull/1758 is similarly adding a repo, as an example. I think Cecil might have a somewhat more challenging .proj because it's not using Arcade--I expect it'll need dotnet-based build commands more like linker, newtonsoft-json, or nuget-client.

I'm not super familiar with the status of the overall prebuilt fixes that @dseefeld and @crummel are working on (in particular regressions in the existing repos), but maybe there's something that can be carved out there as well if adding Cecil doesn't fit.

[omajid]

Adding https://github.com/jbevain/cecil/ as a repo turned out to be unnecessary: https://github.com/dotnet/source-build/issues/1537

Is there any other lower hanging fruit I can take?

[dagood]

We talked about this in triage, and IIRC @dseefeld is planning to update here with some more info and ideas once the big rock of using a "real" previously-source-built archive is done, as that's expected to clarify where we're actually at right now.

[dseefeld]

I added the current status on prebuilts to the description. Prebuilts that aren't being investigated have no owner. If you'd like to investigate any, comment here and I'll update the table. I'll update the table with current prebuilt status weekly.

[tmds]

The SDK includes the Microsoft.CodeAnalysis.NetAnalyzers analyzer. That means we should built it. The analyzer lives in https://github.com/dotnet/roslyn-analyzers repo.

[dseefeld]

https://github.com/dotnet/source-build/pull/1935 was the final PR to get to zero prebuilts.