Dependabot-core: .NET - Production Ready

[deinok]

This is a meta issue.

This issue is facing a general discussion about the future of .NET support of dependabot.

So, true issues related to .NET should be added by its own.

Things to discuss:

• [x] SLN support.
• [x] Remote repositories.
• [x] Packages.config
• [x] <Import Project=""/> aka *.props
• [x] Handle property updates
  
  
  
  • [x] Handle properties that affect a single dependency
  
  
  • [x] Handle properties that affect multiple dependencies
  
  
  • [x] Handle inherited properties
  
  

Older conversations:

212

[greysteil]

Thanks for this @deinok!

• packages.config support should already be working - see https://github.com/greysteil/fsharp/pulls for some examples
• <Import Project=""/> should be working too, although I don't have an example to test on. Any ideas?
• Similarly remote repositories - I'm pretty confident this is working 👌, even when authentication details are required (they can be provided in the Dependabot dashboard).

[deinok]

<Import /> To take a look see this -> https://github.com/graphql-dotnet/graphql-client/blob/master/tests/tests.props

All the projects inside the tests folder import tests.props, and tests.props also imports root.props.
(Notice that tests.props have all the package reference for run tests)

A more complex props can be https://github.com/dotnet/corefx/blob/master/dir.props
So, every lib in src//src|pkg|tests/.csproj calls the parent dir.props, and dir.props calls its own parent until the last one that have a condition

[greysteil]

OK, just ran on https://github.com/greysteil/graphql-client and it's pulling down everything correctly (imports and all). It also generated a PR for a project file it found in the .sln - https://github.com/greysteil/graphql-client/pull/1.

A couple of things I've seen that I'm still not sure about:

• <ProjectReference Include="..\..\src\GraphQL.Client\GraphQL.Client.csproj" />. What's the deal with this? Similar to ?
• I've seen $(...) used for properties in a few files. How do these work? Can they be used for versions (we see that a lot in Maven)? It seems like they're often used in paths so I think we're going to need to support them...

[RohanNagar]

ProjectReference Include="..." /> is saying that the listed .csproj file is a dependency of the current project. So, <ProjectReference Include="..\..\src\GraphQL.Client\GraphQL.Client.csproj" /> means that the current project depends on the GraphQL.Client project. When building, the GraphQL.Client project will be built first so that the current project can use those binaries when building itself.

I suppose what this means for dependabot is that it should fetch the .csproj files given as ProjectReferences and update any external dependencies in those projects as well.

[greysteil]

👍 - OK, <Project Reference Include="..." /> tags work now. I think the next biggest improvement would be supporting variables that use $(...) syntax. Anyone have any experience with them?

[greysteil]

Just noticed lots of <Reference ...> tags in this file. Should we be updating those, too?

[deinok]

That is the old csproj format. I can take a look at how to handle that

[greysteil]

👍. I think handling variables (the $(...) syntax I've seen) is probably more important to start with?

[deinok]

Important, yes. But supporting old csproj will be more easy

[greysteil]

Haha, yeah, make sense. Well if the only syntax is straightforward to handle then no reason not to add it 🙂

[RohanNagar]

for the $(...) syntax, I'm pretty sure that those are referencing variables in a PropertyGroup section, either within the same csproj file or in a parent csproj file.

Example:

<PropertyGroup>
  <BaseFolder>c:\base\folder</BaseFolder>
  <SettingsFile>$(BaseFolder)\settings\settings.xml</SettingsFile>
</PropertyGroup>
<Message Text="Using settings file found at $(SettingsFile)"/>

Output:

Using settings file found at c:\base\folder\settings\settings.xml

For dependencies, I think using a property as part of the path is common, and I don't see why you wouldn't be able to use it for the version number too.

[greysteil]

@RohanNagar @deinok I could use a hand with these property versions, and have an example repo with them now: https://github.com/jyotirmaya12/MusicStore

The csproj for that repo is here, and I'm not sure how Dependabot could have figured out that the constants are defined here. Perhaps from the .sln file (here)?

Any guidance greatly appreciated!

[deinok]

A fast view dont reveals how are they linking the props with the csproj.
I will take a better look ASAP

[greysteil]

Hero, thanks masses @deinok

[deinok]

@greysteil Seems like is using something called KoreBuild https://github.com/aspnet/KoreBuild
I don't see how is it taking the versions.
Seems really complex for its logical simplicity.
Seems more like an edge scenario than a normal case.

[josesimoes]

Just noticed that the version update on Nugets is not checking the dependencies of the updated package, which will results - inevitably - on a build failure.

Example:
This repo/project depends on nanoFramework.CoreLibrary and nanoFramework.Runtime.Events.

After triggering a bump I've received 2 PRs updating both packages (which is correct, in principle, because both were updated). nanoFramework.CoreLibrary is now on 1.0.0-preview073 and nanoFramework.Runtime.Events is now on 1.0.0-preview188.

The fact is that nanoFramework.Runtime.Events depends on nanoFramework.CoreLibrary _AND_ the current version (188) requires nanoFramework.CoreLibrary-1.0.0-preview073.

Any of this PRs fails the build because of wrong dependencies on the other referenced package.

[deinok]

Nice to know about the NanoFramework, I mostly work with embeded devices with C# so I'm glad to discover this framework.

Nvm, for what I can see, the actual published version of:
https://www.nuget.org/packages/nanoFramework.Runtime.Events/1.0.0-preview183 is 183 that referenciates
https://www.nuget.org/packages/nanoFramework.CoreLibrary/1.0.0-preview062

So, Its not a big problem.
Also, I see that you are using a .nfproj -> Probably a NanoFrameworkProj
I'm not sure that dependabot can handle this files, so it is taking the packages.config, and here comes the issue.

Package.config tracks the definitive versions after nuget have merged all dependencies, etc...

[josesimoes]

Nice to know about the NanoFramework, I mostly work with embeded devices with C# so I'm glad to discover this framework.

Great! I'm sure you'll like it. Head there and give it a try 😉

On the versions you are referring to: that's on Nuget.org. For testing preview versions we are using a MyGet feed. There you'll see the versions I'm referring to.

As for the nfproj, I'm aware that this file extension is not being processed now. @greysteil has been kind enough to look into this. See the thread here.

[deinok]

@josesimoes Sorry, Im not able to see the feed, is it private?

Can you explain me the diferences between *.nfproj and *.csproj?

[josesimoes]

🤔 it's a public feed... maybe I copied the dashboard URL... sorry about that...

json feed
gallery

Basically nfproj is very similar to a csproj. Because of the build system (and other details) besides clearly distinguish the project type we are using that extension.

[deinok]

So, we can considerate nfproj to be the old school csproj (CsProj and Package.config | No SDK CsProj)?

[josesimoes]

I would say that statement is correct. 😃

[deinok]

@greysteil Could you tell me what is the environment of the Dependabot Servers, and how is deployment orchestated?

[greysteil]

@deinok of course. It has two components:
1) The main backend, which has a database and keeps track of who is using Dependabot, what PRs have been created, etc.. The main backend kicks of update runs which run in the "updater"
2) The updater, which is essentially a script wrapping Dependabot Core. The updater runs on isolated one-off Heroku dynos (we spin up a new one for each update run). It's Dockerised, with dependabot-core's Dockerfile as its base image.

As a result of the above, it's OK for Dependabot to run unsafe code in the updater, which doesn't have access to any sensitive details. However, Dependabot doesn't clone your repos, it just uses the File Fetcher to pull them down, so running arbitrary code can be tricky (we might not have all of the bits we need, but we can sanitize stuff).

I'm interested in how Dependabot might avoid creating PRs that have version conflicts in them for .NET (or whether doing so is a good / bad idea). Any advice on that would be super valuable.

[deinok]

Okey, from the Dockerfile I supose we can add a program. I think that there is a way to the tell nuget to update a package. So, using this aproach could probably save us time doing the logic to avoid incompatibilities.

I'm not completely sure of this aproach but, adding dotnet and mono to the dockerfile should not be a very big problem

[deinok]

Just to referenciate: https://github.com/dependabot/dependabot-core/pull/663

[josesimoes]

@deinok leveraging on nuget executable to do the hard work on finding the updates (which in the process gets the correct dependencies) seems a very cleaver approach. 👏

It shouldn't be that hard to having it process the project(s)/solution(s) and then committing those difs as the bump PR. Actually this can solve several of the issues/suggestions currently open (regarding .NET).

[greysteil]

Makes sense to me. We can actually remove Java, Groovy and Gradle from the Dockerfile, so I was planning to cut a new version of it today. I'll do that and get #663 in this morning.

[deinok]

Before merging it let me test a thing related to this:

However, Dependabot doesn't clone your repos, it just uses the File Fetcher.

Let me look what happenz if this aproach is just done with the skeleton with just *.csproj, *.sln, *nfcsproj, etc...

[deinok]

@greysteil Also, if you want, I can create something like https://github.com/dependabot/php-dummy-pkg-a but for .NET

[greysteil]

Before merging it let me test a thing related to this:

However, Dependabot doesn't clone your repos, it just uses the File Fetcher.

Let me look what happenz if this aproach is just done with the skeleton with just *.csproj, *.sln, *nfcsproj, etc...

If that causes any problems then hopefully it's possible to create a "mock" application that keeps nuget happy.

[greysteil]

FYI, I started work on supporting property updates today. It's very similar to the process we use for Maven, so not too tricky. Dependabot can now update versions stored in properties as long as only a single dependency is affected, and the property isn't inherited from a parent.

[greysteil]

Update: Dependabot can now update properties that are inherited. There's still a restriction on the property only affecting a single dependency - I'll make it handle properties that affect multiple dependencies later today / tomorrow.

Update: Dependabot can now update properties that affect multiple dependencies, too, meaning support for property versions is complete. 🎉

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[Mobrockers]

Hey! What's the progress on package lock files? We transitioned to package lock files so we can utilize dependency caching in the pipeline but now dependabot can no longer auto-update our packages resulting in tens of pull requests to manually fix up. Would be awesome if package lock file support can be added! Would rather like to not have to build it into the pipeline myself.

[feelepxyz]

@Mobrockers 👋 we haven't made any progress on this yet. It will require quite a bit of re-architecture to actually run the native tools to generate the lockfile. Unfortunately we have lots of work planned already to integrate fully with GitHub so will take a while before we can get to this. We're hoping to grow the team working dependabot-core over the next few months.

[Mobrockers]

Aight, that's a shame. Perhaps someone on the nuget team or the azure pipelines caching team can provide some assistance, this should be pretty relevant to them 😜

[infin8x]

Duplicate of #1303

[Mobrockers]

@infin8x How can this be a duplicate of an issue that was created like one and a half years later :p