Arcade: Make Core-SDK download blobs from private location when public location fail

[JohnTortugo]

Similarly to what is described in this issue and implemented in this PR, Core-SDK will need a similar work, but for downloading _.zip blobs and implemented in MSBuild_. The logic, I believe, should be the same: builds will initially lookup the file in the public location and, if the download fail and the build is internal, try download from the private location.

Link to sample failing build: https://dnceng.visualstudio.com/internal/_build/results?buildId=393720&view=logs&j=695ed18d-4f96-5f5b-a547-692c59452cfc&t=581b320b-1054-5daf-6073-fa89a72ce14b&l=63

The build that updated Core-SDK eng\version.props file in the above build's branch is public and thus published the blob to a public location. However, the current logic in core-sdk is to whenever the current build is an internal one look for the blobs in MSRC only, and thus it doesn't find the file published to the public location.

The implementation I'm planning is this:

• Augment the DownloadFile task to receive a PrivateUri parameter which should be set only if the build is internal.
• DownloadFile task will only try to download files from PrivateUri if the download using Uri fail with error code in the 400 range.

What do you think? @mmitche @livarcocc @dagood @riarenas @markwilkie

[riarenas]

Could you point at which pieces of the Core-SDK build are failing without this?

(Just looks like we keep making more and more one-offs to download from private locations and I'd like to get an idea of what is the requirement here to see if there's something we can improve about the whole thing in the future)

[JohnTortugo]

Updated the description

[riarenas]

Looking at the build, it looks like it's already trying to download from the private location. The failure is due to trying to compose a hybrid of internal + public build, which I thought was out of scope?

@markwilkie @mmitche do you think it's worth trying to get this scenario of hybrid builds working? (The question is just for 3.x servicing, we should definitely see how to improve the hybrid build scenario for 5, but wondering if it's the right moment to work on this)

[riarenas]

Also worth noting that this particular issue has a workaround (produce an internal build of toolset, and the blobs will all be published and restored correctly)

[markwilkie]

Pretty sure that by definition, most security builds will be "hybrid". Unless you mean hybrid at the repo level.... Meaning, I agree that a repo's build can be fully open or private - but the servicing builds will always be a combination of open and private repos/branches as far as I know.

[riarenas]

Hmm, then I think I misunderstood previous conversations. I believed we wouldn't have to worry in core-sdk about only some blobs being in the private locations.

This will have to be fixed for servicing then, so that core-sdk doesn't fail if toolset is not involved in the internal build.

[mmitche]

@markwilkie @mmitche do you think it's worth trying to get this scenario of hybrid builds working? (The question is just for 3.x servicing, we should definitely see how to improve the hybrid build scenario for 5, but wondering if it's the right moment to work on this)

Generally you should expect that the runtimes, when we're done, are either all public or all private. But there will be times when we haven't propagated the private version of the runtime around yet, and you might be building an internal branch but pulling a public runtime.

[markwilkie]

So perhaps the question is whether or not we should the work now to support a mixed inter-repo state. The example here is probably a good one where core-sdk is bringing in toolset. In a security situation, toolset is unlikely to need changes, so one would imagine that bringing it in from the open is advantageous.

So....the idea of "look first in the open, and if not found, look private" - is that something we should do now or later? (or perhaps never I suppose, but that sounds odd)...

[mmitche]

So....the idea of "look first in the open, and if not found, look private" - is that something we should do now or later? (or perhaps never I suppose, but that sounds odd)...

I think that's what @JohnTortugo is currently implementing. But yes, we have to do that now.

[dagood]

Generally you should expect that the runtimes, when we're done, are either all public or all private. But there will be times when we haven't propagated the private version of the runtime around yet, and you might be building an internal branch but pulling a public runtime.

The scenarios I had in mind aren't temporary: anything where the previous public release is inserted into VS, in particular stuff that makes up the toolset like Roslyn, MSBuild, and NuGet. We can't rebuild them because we need the bits included in the product to match VS, and it wouldn't be reasonable to force every single component to put out a new release just because an unrelated .NET Core component happened to need a security release. (IMO.)

Hmm, then I think I misunderstood previous conversations. I believed we wouldn't have to worry in core-sdk about only some blobs being in the private locations.

Maybe we were thinking this along with the plan to be able to add arbitrary channels to an existing release? Then we could add the existing public released bits to internal locations after the fact, on demand. My understanding is this is still not working and needs some AzDO features, so we can't depend on it for serviceability.

[riarenas]

I think that's what @JohnTortugo is currently implementing. But yes, we have to do that now.

The way I understand the state of the world now is:

1. When built from an internal/* branch, Core-SDK will look for all its input blobs in the private location (https://github.com/dotnet/core-sdk/pull/4753). I understood this would be enough, as we wouldn't try to give public builds for inputs during a servicing release.
2. When building from the internal project, Arcade will try to install the specified additional runtimes by first attempting using the public location, if that doesn't work, try looking in private storage. (https://github.com/dotnet/arcade/pull/4122)

What this issue asks for, is that we give item 1 in that list the same treatment as item 2, as my understanding regarding how the servicing SDK would be constructed was incorrect, and we will need to implement this in the core-sdk repo.

[JohnTortugo]

I think the issue is now clarified - please raise your hand if not :-). So I'll move forward with the plan outlined in the issue's description.

@livarcocc can you take a quick look at the plan and let me know if it makes sense to you?

[JohnTortugo]

We are waiting 3.0.1 01 to be out of the door before merging the PR associated with this issue. Moving the issue to tracking them.