Arcade: How do I upload one-off packages to the dotnetfeed dotnet-core blob feed?

[Chrisboh]

@dagood commented on Fri Jul 27 2018

TL;DR: I think we should make a VSTS build def that publishes packages to the central blob feed, or make sure FR can do this.

@jcagme, @MattGal, what do you think?

/cc @sfilipi @eerhardt

---

Recently I've had a few requests to upload some nupkgs to MyGet and the central blob feed. They're data packages (for ML) that are produced as needed, not by a build. I've been using hacked-up ProdCon v1 infrastructure to push to the blob feed, but I think either FR should be able to handle these requests or I should be able to show people how to push packages themselves in a reasonable way.

The Sleet command-line configuration is daunting, but I think FR could use it if doc'd: https://github.com/emgarten/Sleet. However, I don't think we should tell people to use it: it uses write access to the entire account to do anything. It would be easy for someone to do something weird by accident and destroy the feed, causing dotnet repos to be unbuildable until it's fixed. (As far as I know, there's no backup.)

We use build defs for a few other scenarios like this, and I think it makes sense here, too.

---

@jcagme commented on Mon Jul 30 2018

I agree this can be an FR issue, but will leave the decision to @MattGal.

When a one-of upload is required, do we need to upload to both MyGet and blob feed? For blob feeds do we need the package to be in dotnet-core or we could use a different feed? If the second we could be more "relaxed" on the uploads.

From Sleet, code to do this should be straight forward. Deciding where to create the task (buildtools?) and configuring it might be the time consuming part, but nothing hard.

---

@dagood commented on Mon Jul 30 2018

When a one-of upload is required, do we need to upload to both MyGet and blob feed?

Yes, it would be nice for the definition to be able to do both. (In theory people can get individual permission to push to MyGet, but using a centralized def would be great.)

For blob feeds do we need the package to be in dotnet-core or we could use a different feed?

Only dotnet-core. Once it's reasonable to make a private blob feed (https://github.com/dotnet/arcade/issues/321) that seems like a useful target, too, but not now.

Deciding where to create the task (buildtools?) and configuring it might be the time consuming part, but nothing hard.

This definitely seems like something for Arcade. But how to use a task simply... I don't know about that.

---

@jcagme commented on Mon Jul 30 2018

While I think Arcade is the ultimate goal, it might take some time to get it there. If there is not big rush then we should aim to build it there.

---

@dagood commented on Mon Jul 30 2018

I'm a little surprised to hear that (mostly because the feed package is already in Arcade) but it doesn't particularly matter to me where it ends up.

---

@alexperovich commented on Mon Jul 30 2018

This should just be a single csproj with a <PackageReference Include="Microsoft.DotNet.Build.Tasks.Feed"/> in it. It becomes a bit awkward but you can create some FilesToPublish items from the packages then run the "PublishFilesToBlobFeed" target.

---

@dagood commented on Mon Jul 30 2018

Ah, yeah, and creating that csproj using a build def step worked fine for the signing def, seems like it would be nice here, too.

(And the MyGet dotnet-core publish should be fine with the built-in step.)

[dagood]

I set up a build def that does this (including MyGet) here: https://devdiv.visualstudio.com/DevDiv/_build/index?definitionId=9767.

Used it to help with https://github.com/dotnet/coreclr/issues/12898#issuecomment-410023868.
(Other scenarios: TestData packages, ML data.)

[jcagme]

This only publishes at one feed per execution right? Meaning I either push to MyGet or blob feed not both in the same run?

For the blob feed case, do you use Sleet?

[JohnTortugo]

Hey, I was assigned to take up the work on this issue. :-)

Are we already settled on how this should be implemented? From the discussion above seems there are at least two options:

• Use a build definition
• Call a MSBuild task from command line (for some reason I like this one better)

Just a rant..: This feels like the work for a command line tool. Can't Darc/Maestro++ implement this?

[JohnTortugo]

This issue seems quite related to #101

[dagood]

I documented the build def for FR usage at https://github.com/dotnet/core-eng/tree/master/Documentation/Tools/dotnet-core-push-oneoff-package (https://github.com/dotnet/core-eng/pull/4097). Functionally, that's enough for me. @markwilkie, @maririos, and @MattGal might have further input, especially around security.

About using the Sleet command line (or an MSBuild task), I want to bring this back up:

However, I don't think we should tell people to use it: it uses write access to the entire account to do anything. It would be easy for someone to do something weird by accident and destroy the feed, causing dotnet repos to be unbuildable until it's fixed. (As far as I know, there's no backup.)

[JohnTortugo]

After talk with @jcagme and @dagood we decided to keep on using the build definition for pushing one-off-packages. I then went to:

• Patched the build definition to publish to one of the feeds or both at the same time
• Tested it publishing packages on MyGet or Azure Central and both worked - with an exception of a network problem on my end
• Updated the documentation to reflect the changes

I'll move this issue to "In PR": can you guys PTAL at the changes?

[dagood]

A few comments:

1. I wouldn't expect splitting CopyLocal and PublishToCentralBlobFeed into different steps of the build def to work: the CopyLocal step creates ToPublish items which is what PublishToCentralBlobFeed publishes. I think just doing CopyLocal again in the blob feed publish task will work ok. (You could do something more with the conditions to prevent duplicate work, but I don't expect the copy to take long enough for that to matter.)
2. The account key variables which were redirecting to the variable groups were marked secret (makes sense for manual queueing) so I can't tell what the values are anymore. Are they still redirecting to the variable group by default?
3. The doc doesn't mention what the default values for the account keys do.
4. I think the doc should mention again when describing the feed urls that setting one to blank means it won't be published to. This will make the info easy to see in context.

[dagood]

Is it reasonable for people who create and use these one-off packages to use the build definition to unblock themselves?

If not, I think a nice middle ground would be to change this to a release definition with an approval step that goes to core-eng and maybe a few other trusted individuals who do this often and would need to approve for themselves.

If so, we should probably add something to the doc like "file a core-eng issue to let us know why you're using the def" to keep a handle on it.

@ViktorHofer was able to use this def to push Microsoft.DotNet.UAP.TestTools/1.0.20 to MyGet. (That did confirm that the def is currently broken and didn't push to the blob feed, though.)

[jcagme]

If not, I think a nice middle ground would be to change this to a release definition with an approval step that goes to core-eng and maybe a few other trusted individuals who do this often and would need to approve for themselves.

Do we know how common is the need of this? I don't want everyone uploading packages daily. I see this like a "patching" tool. At the same time, I don't want folks to be blocked and waiting for approval in case none of the "trustees" is around.

@markwilkie what do you recommend?

[markwilkie]

It seems there are two objectives here. One is a public publishing step, the other is a product composition piece (or local/private publishing)

For "release", or public publishing, @leecow should be involved. I'm interested to know what @leecow thinks about any "one off" myget and/or public blob location publishing events - specifically around approvals. Perhaps another RM pipeline??

For "product composition", @mmitche should weigh in. Some of these scenarios seem outside of prodcon - and it's unclear to me what any sort of central approval is needed.

[dagood]

Well, 6 packages have been pushed using the def in the 24 days it's existed, to give some idea of rate.

Note for @leecow, this is the current definition of a one-off package from the doc:.

A one-off package is a package that wasn't created as part of a repo's build. If a package is generated during a build, running that build should automatically publish it to feeds and this manual process shouldn't be used.

These don't get pushed to nuget.org, just dev feeds, usually for testing infra or in one case a machine learning model payload for an infra service.

Agreed, this seems completely unrelated to prodcon.

[markwilkie]

If it's always internal - then I don't see the need for approvals. However, it does feel like a slippery slope, especially with myget.

[dagood]

I wouldn't call it internal, since these feeds are public and the packages can be required to build/test our repos. The impact of this def could be fairly big IMO.

[markwilkie]

@leecow - thoughts?

[leecow]

I don't have any particular concerns regarding these one-offs to MyGet feeds. NuGet would be a different matter.

[markwilkie]

Thanks @leecow - so for now, you're cool w/ "open season", so long as the target customer is devs and the bits are being published to myget or our blob storage?

[dagood]

The build definition is still currently broken, by the way.

FYI @artidoro who just used it to push an ML package: it'll be available on MyGet, but it didn't get pushed to the central blob feed.

[JohnTortugo]

Just reverted the build-definition to the status I got it initially.

[dagood]

@mmitche, it looks like you added a condition back in for disabling--skipping MyGet publish looks fine, but if you disable the blob feed push step, it looks like it will fail to CopyLocal the packages for the MyGet step. (See above for more info/caveats about fixing this.)

[mmitche]

Okay you can leave it as is then...it will just fail if the upload to myget has no feed but it's not a big deal.

[dagood]

Agreed, I'm fine either way, mostly just want to make sure it's known and status is somewhere.

[dagood]

A couple changes:

• I moved the project code to https://devdiv.visualstudio.com/DevDiv/_git/DotNetCore?version=GBone-off-nupkg-publish so it isn't tied to my name (in case that discouraged maintenance or something).
• So far, I've been looking at each build and marking them to be permanently retained (not deleted by retention policy) when they push to MyGet or the dotnet-core central blob feed. I'm going to turn off my email notifications and stop keeping track of this because this def is seeing increasing usage.
  
  
  
  • The risk I was mitigating is being unable to retroactively understand weird publishes after 30 days, the maximum automatic retention time.
  
  

[ViktorHofer]

Because of https://github.com/dotnet/core-eng/issues/5070 the myget upload step is currently broken. Still I wanted to update the uap tools package in the blob feed but it seems that isn't working either, the new package version isn't added to the feed: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=2362686.

[dagood]

If you look at the central blob feed publish log, it didn't find anything to publish. ("No items to push found in the items list.") I suspect the build agent doesn't have access to that share. I suggest creating your own based on https://github.com/dotnet/core-eng/tree/master/Documentation/Tools/dotnet-core-push-oneoff-package.

[ViktorHofer]

The access rights to my share are correct, I just double checked them.

[ViktorHofer]

Also this was working fine last year (till December), maybe users changed?

[dagood]

Oh, actually, the "pkgs" artifact actually does have your package in it, so the agent is definitely able to access it. I wonder if PassIfExistingItemIdentical is misfiring and ignoring your package. This project has an old version of the blob feed push tasks--maybe something has changed in the blob feed format that's affected this. I suggest filing a core-eng FR issue.

[riarenas]

What is the work remaining here?

[mmitche]

None, there is a build def that is around that does this, but we should not really be doing this at all any longer. The dotnet3* and dotnet5 feeds in azdo are the right places for these types of things from now on.

[dagood]

I'd recommend working with common users of https://devdiv.visualstudio.com/DevDiv/_build?definitionId=9767&_a=summary to make sure nothing's missed. But agreed, as things move to AzDO feeds the need will drop, and it seems unlikely that anyone will do anything to improve the old def at this point.