Monogame: On external online dependencies

[mrhelmut]

Hello there,

Following #7086 and #7088 (which are very welcome moves), I see some nuget gymnastic going on and I thought it would be interesting to open a question over how to manage dependencies (both internal and external), just for future reference.

On some projects I've worked on, there was an architectural policy of "no online dependencies in the repository" because it was a requirement to be able to deploy new build environments in a context with limited access to the internet (some game companies have build stations cut from the internet to avoid breaches), to prevent any failures (e.g. internet outages, nuget.org outages, external dependencies deprecating or being removed), to handle third parties (e.g. private nugets), to provide a standalone repository (i.e. all dependencies are internal and build out-of-the-box), or simply to avoid starting a dependency hell.

Most of these concerns can be handled by setup'ing a nuget cache local to the repository, like having all nugets pushed to the repo with a setup file for MSBuild/Visual Studio (Nuget.Config):

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
        <add key="local-packages" value="../offline_nugets" />
    </packageSources>
</configuration>

In the case of #7086, this is fine.

In the case of #7088, I don't know if Cake can fetch from a local source? (I assume it uses MSBuild for its setup and can read the config while restoring packages)

Independently of these PRs, the question I'd like to open here, is whether we should endorse external online references or not?

My personal concern is that starting to allow them exposes MonoGame little by little to adding any kind of nuget references everywhere and making the project less and less resilient to being built offline/out-of-the-box.

What are your thoughts?

@tomspilman @Jjagg @harry-cpp

[Jjagg]

NuGet uses a package cache on the user machine. Any package that is downloaded is put in the cache and if a cached version is required it can be restored without an internet connection. Since users have to download the MG packages anyway at some point, the requirements for an internet connection do not change by switching to NuGets. I.e. you need an internet connection to get the packages once, then you can build offline (as long as you do not clear the package cache).

For #7086 we use a ProjectReference internally to reference the Core project so internally things don't change. For game projects, the packages are restored at the same time and if possible are retrieved from the package cache, so it does not affect requirements of an internet connection.

Both for maintainers and consumers I think NuGet packages offer a better experience and we should not shy away from it.

[mrhelmut]

Since users have to download the MG packages anyway at some point, the requirements for an internet connection do not change by switching to NuGets.

It does change things when it comes to administrating an offline environment with a package manager. Calling your administrator every time you need a new package to be added to the offline repo can be complicated at times (especially when some cascading deps are resolved at build time and you couldn't figure that prior to calling the administrator the first time).

My question is not at all against nuget or any package manager, but it is about being cautious about somehow hidden or cascading dependencies that can hardly been figured out if anyone wants to setup an offline repo at some point.

It's not just about "downloading once".

[Jjagg]

Calling your administrator every time you need a new package to be added to the offline repo can be complicated at times (especially when some cascading deps are resolved at build time and you couldn't figure that prior to calling the administrator the first time).

So this is about trust/security rather than availability?

Regarding the two packages we're talking about now:

• If companies pull the MG package into their offline repo as a NuGet they won't need NBGV. If they build from source they don't care about MG versioning in the assembly and can remove the dependency in the MonoGame.props file.
• With the Core package if they're building from source it's a project reference so it doesn't go through nuget. If referencing a runtime, yes you need the extra package locally.

In general for NuGet dependencies there's great tooling to figure out dependencies and download them to a local folder. E.g.

• dotnet-list-package to list packages (flag to list transitively)
• dotnet-outdated to report outdated packages
• dotnet-depguard to check for blacklisted packages
• nuget-deploy to install restored nuget packages locally
• sleet to create and manage a nuget feed.

Though probably the simplest way to set up a local feed is to

• temporarily override the nuget cache location
• restore MG project
• validate packages
• register temp cache location as feed and remove default nuget.org feed

I agree we should not have npm style dependency graphs and we should always consider the downsides and different usage scenarios when we include NuGet dependencies. However I also think we should take into account what's easiest to maintain and what's the nicest experience for the majority of MG users that do not have offline build requirements.

And to get ahead of myself a little, for something like #6144 I do not see much difference in companies trusting dependencies we directly include in the repo vs trusting nuget packages we reference (apart from perhaps company security policy, but I'd argue if those cases are treated differently it's badly written policy and that's not our responsibility to account for), especially considering nugets can't be removed or overwritten on nuget.org.

[tomspilman]

"no online dependencies in the repository"

In general that is my preferred way to work. This is why i disliked Protobuild calling git to download stuff. It is also why i generally dislike using NuGet.

As long as we can still use the C# projects and assemblies locally and NuGet isn't mandatory i'm happy.

So this is about trust/security rather than availability?

It is all of those things.

what's the nicest experience for the majority of MG users

Yes. This is why i'm not as grumpy about all this NuGet stuff.

For most of our users it is totally fine.

[stromkos]

As, @tomspilman, has said:

As long as we can still use the C# projects and assemblies locally and NuGet isn't mandatory I'm happy.

I am going to take this slightly off topic @Jjagg @mrhelmut, my apologies.

The problem, as I see it, is the project is on divergent paths for delivery: traditional shared libraries and project based solutions (Nuget). The latter, while good for most users, fails when you have hundreds of projects.

Put yourself in the shoes of a new user:
I have download the only easily accessible version of Visual Studio, to most users the only option, Visual Studio 2019 Community.

When you click on the get started link on the website, you are brought to the Downloads page listing installers. I would expect that clicking on the latest build, then the Visual Studio download, that it would just work. The installer didn't do anything.

Assume an old fashioned advanced user:
I went through everything above, without success, and so, I try the following:
I will just download the source, attempt to build it, and manually reference it. (Please note this pre-dates COVID-19, I have not tried since), the build fails, I cannot download one of the dependencies, removed do to DCMA request.

I give up, your program does not work.

Nuget is great, but it does not address the waste of not reusing DLL files (the entire purpose to their being) or the question of: what packages do I need? Think of the Content Builder, MGCB.exe.

The decision to abandon the installer, was premature in my opinion. While Microsoft may be telling you the future is elsewhere, something about command line?(I did not get that memo), the average new user wants a simple way to install the program. The installer worked for this purpose, and the simple one section addition for vs2019 to the installer package would solve the issue completely.

Either update the webpage to accurately reflect the new way to install the templates in VS 2019, and/or include them in the installer.

As a side note, those of us attempting to install your software on 30 computers at a time in a lab, must have an installer to run, (I have created a custom modified installer to install the templates for all users). It is not feasible to store a copy of the entire compiled product, per student per project.

This has worked perfectly, until the pandemic moved my students to use their home computers.

I would expect the XNA compatibility to extend to the installers as well as the API, until the break at 4.0, as planned.

I do not wish to be a Debbie downer, but you should not abandon the installer, without offering a viable alternative on the webpage.

[tomspilman]

Hi @stromkos .

you should not abandon the installer, without offering a viable alternative on the webpage.

To be fair we have installers in the latest stable builds.

It is the development installers that are currently broken and likely to be removed.

I will just download the source, attempt to build it, and manually reference it. (Please note this pre-dates COVID-19, I have not tried since), the build fails, I cannot download one of the dependencies, removed do to DCMA request.

You just happened to try at an unfortunate time. We got hit by a dumb DCMA problem and we have been transitioning to a new way to deliver MG to users. It is not like this normally.

If you build from source today... or any other day in the last 10 years of the project... it would have worked fine.

As a side note, those of us attempting to install your software on 30 computers at a time in a lab, must have an installer to run,

The plan is for all of it including the templates to work via nuget and be automatically available in VS.

We discussed maybe having some sort of installer than pre-installs the nugets and stuff... but that has been low priority.

@Jjagg @harry-cpp ???

[Jjagg]

Thank you for your feedback @stromkos.

In my interpretation is there are two things to address:

• the downloads page instructions do not work with VS2019 and are outdated
• we need to support some form of silent installation in VS2019

The downloads page will get updated. It is an issue that we never updated the installer to install the templates in VS2019.
In the develop branch we build a vsix (VS extension) package with the templates. We plan on automatically publishing this package to the VS marketplace so it can be downloaded from within VS2017/VS2019 from the extensions menu. Docs on how to do that are incoming in #7117.

As for silent install, apparently Visual Studio ships a CLI tool to install vsix packages. There doesn't seem to be any official documentation. See https://stackoverflow.com/a/30574978/3130776. Would that be sufficient for your use case?

[mrhelmut]

To continue on the "nugets referencing nugets" and the "need to be online to build" discussion, I'm quoting Streets of Rage 4 lead dev from https://github.com/dotnet/corert/issues/7200

What I do not like with CoreRT is that it uses .Net Core, that uses nuget. I had a lot of problems with nuget. It slows compilation down, is hard to configure, badly documented, and unreliable (connecting to the internet to build, really ??). I ended up finding a way to include the nuget packages in the repository and compile against them to have the game compiling on all machines, and not have package updates breaking the game.

My suggestions are:

• MonoGame shouldn't rely on nugets to build from source
• Produced artifacts should be monolithic and not reference other nugets, so that users can safely setup local packages (which is not possible on UWP, this platform is plagued with references)
• We should provide backup nuget download links outside of nuget.org

The idea is that whenever nuget restore needs to be called to either build MonoGame or a project using it, users should be able to be offline and resolve all packages locally without errors.

I wonder if, we should also ship Cake in the repository as well (a working executable, not just a bootstrap install script).

[Jjagg]

Apart from constrained environments, I really don't understand why shipping assemblies is better than using nugets. Every argument against using NuGet packages I've read is countered by the local package cache and pinning versions. If you can download the repo you are connected to the internet and you can pull down NuGets too.

What I do not like with CoreRT is that it uses .Net Core, that uses nuget.

.NET Core does not 'use NuGet' in and of itself, though NuGet is a first-class citizen in SDK-style projects.

It slows compilation down

Once packages are in the local cache there's only the discovery impact, which is minimal.

is hard to configure

Need more detail to counter this one, but I don't agree. Especially just referencing NuGet dependencies is as easy as it gets.

badly documented

I'd say it's pretty well documented.

and unreliable (connecting to the internet to build, really ??)

Only if packages aren't in the local cache. If bad/no internet connection worries you, for MG's purposes, run dotnet restore when you pull/update the repo.

not have package updates breaking the game

NuGet packages don't automatically update unless you use wildcard versions.

MonoGame shouldn't rely on nugets to build from source

I disagree. NuGets make our lives and that of MG users easier. Take SharpDX for example. We used to reference the assemblies in the Dependencies repo. I've upgraded it a couple times and the process is incredibly tedious. I had a script to pull down nugets and for 4.0.1 had to set up fixes multiple times (1, 2. With NuGet upgrading couldn't be easier and it handles picking the appropriate target for us so we wouldn't have run into any of those issues if we were referencing nugets before.

Now we've switched to PackageReference for SharpDX for both UWP and WindowsDX. Changing the version is just a config change and if users have SharpDX 4.0.1 in their NuGet package cache before building MG, there's no need to redownload the packages.
If people want to get all assemblies when pulling the repo, we can provide a script to restore nugets.

Produced artifacts should be monolithic and not reference other nugets, so that users can safely setup local packages (which is not possible on UWP, this platform is plagued with references)

I don't understand what you mean by monolithic in this context. The assemblies are always in seperate files and they're a dependency in the same way regardless of how the assembly is referenced, directly or through nuget.

We should provide backup nuget download links outside of nuget.org

We can set up a feed on the web server.

I wonder if, we should also ship Cake in the repository as well (a working executable, not just a bootstrap install script).

7076 adds Cake.Tool to the dotnet local tool manifest so it gets installed when users run dotnet tool restore with the nuget being cached of course.

[tomspilman]

I really don't understand why shipping assemblies is better than using nugets.

I've had to struggle to get a nuget to restore properly in the past. Delete this folder, delete this other folder. Call restore from VS. Use restore on the command line. Convert this .json file to something else. Etc.

I compare that experience to consuming an assembly... which i've never had an issue with.

So i totally understand the hate for nugets.

But i don't care because i usually build MG from source and don't really care how MG is delivered beyond that. :)

[tomspilman]

Another little thing that we have had experience with.... assemblies are not subjected to DMCA takedowns. But i bet a NuGet could be taken down.

[harry-cpp]

I'm with @Jjagg on this one.

he plan is for all of it including the templates to work via nuget and be automatically available in VS.

The idea is that the user just installs the templates for their preferred IDE, and that's it, they can now make a new MG project and build it.

[mrhelmut]

I'm not against nuget at all. It sure makes things more maintainable and easier to distribute, and I truly believe that the latest progress on MonoGame are very necessary and robust. Using and maintaining MonoGame has never been this easy. I would just like to suggest to be mindful of some scenarios.

My point can be summed up like this: being able to deploy a project in an offline context with minimum friction.

To be more specific, as of now, the distribution model and structure of MonoGame is cool and suited to offline use (it's not too complicated to download/build the nugets and have them at hand to create and build projects with a local nuget source or offline).

My main, and mostly sole concern, is to be mindful of future developments and cascading references.

For instance, #7086 (which in concept is very useful, not debating that) may tempt us to add the "Core" references to every target nugets (e.g. DesktopGL nuget referencing Core nuget, etc.). I would rather ship the Core assembly with all target nugets than having cascading references (that's what I meant with "monolithic": shipping assemblies in one nuget instead of two).

In the same fashion, it makes perfect sense to have SharpDX referenced via nuget, but from a distribution perspective it would be interesting to pack SharpDX assemblies with MonoGame WindowsDX (I know that dotnet pack can't pack references like nuget.exe used to allow, but it's still possible with a few "copy to ouput" directives).

And that's pretty much it.
It's no big deal, it's merely about a design guideline I'd like to suggest (i.e. "nuget things only if necessary"). If we don't agree, that's fine, let's just list the dependencies somewhere for manual download.
It's just about being mindful and having a reasonable limit on "nuget and reference all the things" to eventually be more liked by sys/build admins in the long run.

Things with Cake or NBGV are very minor nitpicking (it's always possible to just fire up the solutions in VS and calling it a day).

[harry-cpp]

Just adding a conclusion here from maintainer only chat: We'll make an automated script that will search all the .csproj files in the repo for PackageReference and automatically fill up another repo with all the needed nugets for offline building.