Arcade: Why does execute-sdl.yml download all artifacts? Fails in Core-Setup due to disk space

[dagood]

https://github.com/dotnet/arcade/blob/ab99af9da2605b56f8720278a889bb0d2d711645/eng/common/templates/job/execute-sdl.yml#L21-L27

Why does this download everything, rather than the set of "final" artifacts Arcade uses in other situations? (PackageArtifacts/BlobArtifacts?)

The above along with the (I think relatively small) hosted agent that's it's hard-coded to use causes failures for Core-Setup:

https://github.com/dotnet/arcade/blob/ab99af9da2605b56f8720278a889bb0d2d711645/eng/common/templates/job/execute-sdl.yml#L16-L17

Core-Setup publishes a lot of artifacts to the pipeline that aren't signed yet and/or won't be published, so that other stages in the pipeline can work with them. This creates a large enough download that the SDL job fails due to lack of disk space in this prototype build:

https://dev.azure.com/dnceng/internal/_build/results?buildId=402553

@sunandabalu @dotnet/dnceng

[dagood]

The SDL task fails on NetCoreInternal-Pool buildpool.windows.10.amd64.vs2019. This is probably due to lack of prereqs, and I imagine running it on that pool is not supported, but mentioning it for the record and in case increasing disk size ends up being a recommended workaround:

https://dev.azure.com/dnceng/internal/_build/results?buildId=404535

(Not copying error in case info should remain internal.)

[JohnTortugo]

FWIW: Last time I checked with @sunandabalu I believe she said downloading all files were a requirement. Anyway, I think we are overdue to enabling the post-build scripts to execute on non-hosted agents.

BTW, @sunandabalu is OOF today.

[dagood]

I see, if that is the case, then Core-Setup is blocked adding automated SDL tooling until https://github.com/dotnet/arcade/issues/3700 is complete and the SDL tooling is made compatible with some pool that has enough space.

[dagood]

Leaving this issue open to get a definitive answer on this:

Last time I checked with @sunandabalu I believe she said downloading all files were a requirement.

Opened https://github.com/dotnet/core-eng/issues/8180 to track getting the SDL tooling runnable on a pool with more disk space.

[sunandabalu]

The reason we download all the artifacts for SDL step, is to be able to run tools that target artifacts like BinSkim. BinSkim runs on all the final binaries technically that we would ship like pdbs, exe and dll. So if we have a pattern where these bits are published to, we can restrict to downloading only those. But again, the artifacts vary from repo to repo and how / what gets published is up to the repos. So how could we determine that only Package and Blob artifacts is what we care about?

[dagood]

how could we determine that only Package and Blob artifacts is what we care about?

Does Arcade need to make this determination?

On the flip side, there may be build assets that are published without being uploaded as pipeline artifacts. For example, VS insertion packages are generally published to the VS feed on the machine they're built on, I believe due to limitations in Arcade. They currently are uploaded as artifacts, but this "weak" dependency means it could be disabled without realizing the effect on SDL runs.

My point is that the repo already has to be aware they need to publish more artifacts than they may otherwise need to. It doesn't seem like Arcade should assume either way: either it downloads too much, which blocks and/or slows down the build, or it misses some assets.

[dagood]

The ultimate solution is to have Arcade handle all publishing. Then Arcade can safely scan only the artifacts it knows it will publish. However, this requires Arcade to support all the types of publishing we need to do.

Supporting custom publish steps in a way that ensures all build assets pass through a known set of artifacts would also be a way to know what needs to be scanned.

[chcosta]

Does Arcade need to make this determination?

It doesn't seem like Arcade should assume either way

agreed. The onus should be on the repo owner to ensure they are configured properly.

[mmitche]

@dagood I chatted with @sunandabalu on this. There are a ton of other artifacts downloaded here which are not interesting. Specifically "IntermediateUnsignedArtifacts" as well as "PreparedArtifacts". Both of those are unnnecessary in core-setup's case, since BlobArtifacts and PackageArtifacts are the only bits that are ever published.

The suggestion here is to plumb through a filter for the artifacts to download, rather than to always download everything. Default to the current behavior, while core-setup would set it to PackageArtifacts/* and BlobArtifacts/*.

Can you try this solution out before we implement in this in arcade by just altering execute-sdl,yml in your branch? If it works, we will make the change in arcade. I am also interested in how long SDL takes.

[chcosta]

This should already be mostly plumbed through - https://github.com/dotnet/arcade/blob/master/Documentation/AzureDevOps/TemplateSchema.md

[dagood]

I did already try this, which worked:

-   - task: DownloadBuildArtifacts@0
-     displayName: Download Build Artifacts
-     inputs:
-       buildType: current
-       downloadType: specific files
-       matchingPattern: "**"
-       downloadPath: $(Build.SourcesDirectory)\artifacts
+  - task: DownloadBuildArtifacts@0
+    displayName: Download Package Artifacts
+    inputs:
+      buildType: current
+      artifactName: PackageArtifacts
+      downloadPath: $(Build.ArtifactStagingDirectory)\artifacts
+
+  - task: DownloadBuildArtifacts@0
+    displayName: Download Blob Artifacts
+    inputs:
+      buildType: current
+      artifactName: BlobArtifacts
+      downloadPath: $(Build.ArtifactStagingDirectory)\artifacts

[mmitche]

Great.

[dagood]

This _should_ already be mostly plumbed through - https://github.com/dotnet/arcade/blob/master/Documentation/AzureDevOps/TemplateSchema.md

Is work being tracked to have the SDL template follow this schema, or whatever it is that we need to plumb in the change above? (Not completely clear to me how the schema would be applied to this case, maybe just the DownloadArtifact schema as a param for execute-sdl.yml?)

Should someone be assigned to this issue?

We need this to turn on automatic SDL validation and satisfy 3.1 compliance requirements.

[sunandabalu]

Should be a straight forward change to plumb this through, putting in the FR backlog so it will get addressed soon.

[adiaaida]

@dagood Have you gotten a chance to try out this change? I'm hoping to close this issue soon.

[dagood]

Nope, kicked off https://dev.azure.com/dnceng/internal/_build/results?buildId=424848 to try it.

Edit: Failed due to combination of underlying bugs, rerunning at https://dev.azure.com/dnceng/internal/_build/results?buildId=425206.

Edit: Failed due to signing service falling over. Rerunning at https://dev.azure.com/dnceng/internal/_build/results?buildId=426149.

Edit: Finally got a success! At https://dev.azure.com/dnceng/internal/_build/results?buildId=426399

[dagood]

Will this be ported to release/3.x? This issue blocks Core-Setup from implementing automated SDL runs in 3.1 servicing releases. (3.1 had the compliance requirement that kicked this all off.)

@vivmishra, now that adding automated SDL validation missed 3.1.0 GA for Core-Setup, we do still want it for 3.1.1, correct?

[sunandabalu]

I don't see why not, @adiaaida could you send out a PR to merge this in 3.x branch for approval?

[adiaaida]

Sure thing

[vivmishra]

Yes, we want this working in release/3.1 eventually.

[MattGal]

@adiaaida anything left here?

[adiaaida]

I'm going to close this out. @dagood, let me know if anything doesn't work as expected.

[dagood]

Was it ported to 3.x?

[adiaaida]

Yes, in this commit: https://github.com/dotnet/arcade/commit/2485d69f4aebb147ab4ec1755555ffa24a3c6b1b

[dagood]

Cool, thanks for the link.