Source-build: Analyze prebuilts for corefx

[dseefeld]

The following prebuilt packages are included as dependencies in source-build by corefx. These need to be analyzed to determine how to avoid requiring packages that source-build doesn’t build.

1. Change to use the source-built version
2. Package is not critical to product and can be removed. (i.e.test-specific, windows-specific, etc)
3. The package is used as reference only
4. The package should be built in source-build
5. Other – please elaborate on what you think should be done.

For more detailed usage information for these, please see the _Usage Exploration_ tab of the SourceBuildPrebuiltReport.

Prebuilt Status Updated 2019-06-24

Up to date prebuilt status can now be found in the SourceBuildPrebuiltReport in the _Repo Status Data_ tab.

Prebuilt Table Updated 2019-05-07

PackageName | PackageVersion | SourceBuiltVersion | Category
-- | -- | -- | --
Microsoft.DotNet.ApiCompat | 1.0.0-beta.19205.6 | | Add'l validation - can be disabled for source-build
Microsoft.DotNet.CoreFxTesting | 1.0.0-beta.19205.6 | | Test-only
Microsoft.DotNet.GenAPI | 1.0.0-beta.19205.6 | | Will be turned off for source-build
Microsoft.DotNet.GenFacades | 1.0.0-beta.19177.11 | | Will be getting from Arcade
Microsoft.DotNet.SourceRewriter | 1.0.0-beta.19205.6 | | Will be removed
Microsoft.DotNet.VersionTools.Tasks | 1.0.0-beta.19205.6 | | Will be conditioned out
Microsoft.TargetingPack.NETFramework.v4.7.2 | 1.0.0 | | Ref-only
System.Memory | 4.5.1 | | Ref-only (use version below)
System.Security.Cryptography.Cng | 4.5.0 | | Ref-only (use version below)
System.Security.Cryptography.OpenSsl | 4.5.0 | | Ref-only (use version below)
System.Security.Cryptography.Pkcs | 4.5.0 | | Ref-only (use version below)
System.Threading.Tasks.Extensions | 4.5.1 | | Ref-only (use version below)

Original Prebuilt Table

PackageName | PackageVersion | SourceBuiltVersion | Category
-- | -- | -- | --
Desktop.Analyzers | 1.1.0 |
Microsoft.CodeAnalysis.Common | 2.9.0 | 3.1.0-beta1-19067-14
Microsoft.DiaSymReader.Pdb2Pdb | 1.1.0-beta1-62506-02 |
Microsoft.DotNet.ApiCompat | 1.0.0-beta.19115.1 |
Microsoft.DotNet.Build.Tasks.Feed | 2.2.0-beta.19115.1 |
Microsoft.DotNet.CoreFxTesting | 1.0.0-beta.19115.1 |
Microsoft.DotNet.GenAPI | 1.0.0-beta.19115.1 |
Microsoft.DotNet.GenFacades | 1.0.0-beta.19115.1 |
Microsoft.DotNet.SignTool | 1.0.0-beta.19067.6 |
Microsoft.DotNet.SourceRewriter | 1.0.0-beta.19115.1 |
Microsoft.NETCore.Compilers | 3.0.0-beta2-final | 3.1.0-beta1-19067-14
Microsoft.Net.Compilers | 3.0.0-beta2-final |
Microsoft.SourceLink.GitHub | 1.0.0-beta2-18618-05 |
Microsoft.SourceLink.Vsts.Git | 1.0.0-beta2-18618-05 |
Microsoft.SymbolUploader.Build.Task | 1.0.0-beta-63604-05 |
Microsoft.TargetingPack.NETFramework.v4.7.2 | 1.0.0 |
System.Memory | 4.5.2 |
System.Runtime.Analyzers | 1.1.0 |
System.Runtime.InteropServices.Analyzers | 1.1.0 |
System.Security.Cryptography.Cng | 4.5.0 |
System.Security.Cryptography.Hashing.Algorithms.Analyzers | 1.1.0 |
System.Security.Cryptography.OpenSsl | 4.5.0 |
System.Security.Cryptography.Pkcs | 4.5.0 |
System.Threading.Tasks.Extensions | 4.5.1 |
sn | 1.0.0 |
vswhere | 2.5.2 |

[ericstj]

Looks like this chart is out of date and the report is not accessible. @dseefeld

[dseefeld]

@ericstj I updated the chart from the most recent completed source-build build and gave you permissions to the report.

[ericstj]

Updated the table and fixed a lot of these with https://github.com/dotnet/corefx/pull/36387.

[ericstj]

Hi @dseefeld I was taking a look at the report after my changes went in and I don't see a difference reflected. How often do you do SourceBuilds and run the report? Thanks!

[dseefeld]

@ericstj We've been updating source-build when prodcon has a coherent build (usually around a release). There's work in progress to get updates from DARC on a more frequent basis, so our build frequency should go up.

[ericstj]

It'd be great if there was a way to drive this down that wasn't so much "guess and check". Any pointers for how to do a single-repo build that would replicate a portion of this?

[dseefeld]

Unfortunately, we don't have anything to build a single-repo to see how prebuilts are affected. The way to do this would be to clone source-build, replace the commit for corefx with the desired commit and then build using /p:RootRepo=corefx. This will build everything up through corefx.

[ericstj]

Hi @dseefeld do you have new data for us here? I checked the report and I'm still seeing all the same data as before.

For example, if I look at https://msit.powerbi.com/groups/me/reports/64e989dd-8072-4d84-8268-140bde0cbc7d/ReportSectionc64da6d8334068f41e23

I see it complaining about packages like System.Runtime.Analyzers, but those are all under '$(DotNetBuildFromSource)' != 'true' since 3/26:
https://github.com/dotnet/corefx/commit/7732d6b2569f5ffa44ecca1755d0135d300e8569

[dseefeld]

We're working on updating source-build to 3.0 preview 4. That should pull in the changes you've identified. See https://github.com/dotnet/source-build/pull/1022

[ericstj]

@dseefeld could you add the following as reference-only packages?

PackageName | PackageVersion | SourceBuiltVersion | Category
-- | -- | -- | --
System.Memory | 4.5.3 |   |  
System.Numerics.Vectors | 4.5.0 |   |  
System.Security.Cryptography.Cng | 4.5.0 |   |  
System.Security.Cryptography.OpenSsl | 4.5.1 |   |  
System.Security.Cryptography.Pkcs | 4.5.2 |   |  
System.Threading.Tasks.Extensions | 4.5.2 | |

We pick these up from servicing branches and use their reference assemblies when compiling netstandard assemblies.

Also I had a question about desktop framework reference assemblies: do we have a source-built copy of desktop refrence asemblies? In corefx we use these to build our .NET framework compat facades.

[dseefeld]

@ericstj Yes. I'll add those reference-only packages. Regarding the desktop framework reference assemblies, we're using these in source-build, but we don't have a source-built copy of these, yet.

[ericstj]

I've added one more reference package. Please take note of these versions as many are different from your current report since the packages have serviced since past source-builds. @joperezr

[ericstj]

@dseefeld we're still being flagged for the packages listed here: https://github.com/dotnet/source-build/issues/884#issuecomment-489268532

[ericstj]

Also the current report is flagging us for GenAPI, but we discussed before that this would only be needed for the first phase of sourcebuild and we've conditioned it to be off for the offline portion: https://github.com/dotnet/corefx/blob/1841042b99062de13dc80098cede9413be569238/eng/Tools.props#L21-L25

[dseefeld]

@ericstj 4 of the packages from above still need to be added as reference-only packages. I'll add them as soon as I can.

PackageName | PackageVersion | Status
-- | -- | --
System.Memory | 4.5.3 |  Reference-only - to be added |  
System.Numerics.Vectors | 4.5.0 |  Reference-only - to be added |  
System.Security.Cryptography.Cng | 4.5.0 |  Already included as a reference-only package |  
System.Security.Cryptography.OpenSsl | 4.5.1 | Reference-only - to be added |  
System.Security.Cryptography.Pkcs | 4.5.2 | Reference-only - to be added |  
System.Threading.Tasks.Extensions | 4.5.2 |  Already included as a reference-only package |

WRT the GenAPI prebuilt, I see what the issue is. The condition in eng/Tools.props is checking for OfflineBuild, but it should be checking for DotNetBuildOffline. I should have caught that in my review of the PR. I'll add a patch to correct this, so we can see that change in the report. Can you change the code in corefx so we can pick it up the next time we pick up a coherent build?

[ericstj]

I see, I thought we specifically agreed upon that property name. I submitted change to corefx: https://github.com/dotnet/corefx/pull/39926

Thanks for looking into this.

[dseefeld]

I see, I thought we specifically agreed upon that property name. I submitted change to corefx: dotnet/corefx#39926

Thanks for looking into this.

I did state that this was the property name to use, but I was wrong. Internally, source-build uses OfflineBuild, but the official property passed to the repos is DotNetBuildOffline. Sorry for the confusion

[dseefeld]

All prebuilts removed.