Monogame: Discussion for removing Protobuild and migrating to new(er) style csproj

[Jjagg]

Tasks

I don't think we should migrate the installer. Instead we should replace it with nugets (and possibly another way to distribute the Pipeline Tool).

To make things usable, but not offer a full replacement we need to fix any failing tests for MGCB on .NET Core, making sure its dependencies all work. Other than that we just need to publish NuGets:

• nugets for Core projects
• nugets for dotnet new templates
• MGCB global tool nuget

We could already publish nuget 1 and 2 without fixing 3, because they use cra0zy's Content.Builder nuget to pull in MGCB (running on .NET Framework).

To fully migrate we need to:

• Distribute the pipeline tool someway
• Finish DesktopGL VideoPlayer: #6860 (so we can drop the Mac platform)
• Provide a VSIX that installs the dotnet new samples (mentioned in #6842)

Bonus:

• F# and VB templates
• Let the CAKE script upload develop nugets to a feed (myget or GitHub nuget feed)

---

This issue is for tracking and discussing how to migrate from Protobuild generated project files to sdk-style csproj files.

Related:

• #6207 Issue for removing Protobuild
• #6650 PR that moves source files for globbing with sdk-style csproj

We will set up pull requests separately to add the csproj for each platform.

---

Additionally we should look at the following issues once we remove Protobuild:

• #5339 #4975 Supporting .NET Standard will be easy
• #6137 Support dotnet build
• #6144 Switch to nugets for dependencies; this would make dependency management so much easier. We could set up a nuget feed on GitHub's Package Registry, on teamcity or on myget to have our custom-built stuff hosted.
• #6649 can be closed when Protobuild is gone
• #5751 and #5192 Publish development builds and pull request builds to a NuGet feed.

[Jjagg]

@tomspilman Is there anything we need to look out for or hold off on because of consoles or other limitations?
Also pinging @mrhelmut because he might know as well (and is probably excited for this :wink:).

I don't think we should let this hold us back, but SDK-style projects can only be open in VS2017+ (and we should of course document that).

[Jjagg]

We'll need to fix up the NuGets as well and I have some suggestions.

I would really like to remove the DX and OpenGL nugets added in #5799. To me, it makes so much more sense to have a package per platform (and so per .csproj after migration) because we can't be consistent in supplying nugets that contain multiple platforms. With the new csproj files we can include nuget information in the .csproj files and publish easily with dotnet CLI.
In #6650 I dropped Framework from the project names. I would love if we could also use those names for the NuGets. So:

• MonoGame.WindowsDX
• MonoGame.DesktopGL
• MonoGame.DesktopGL.Mac
• MonoGame.UWP
• MonoGame.Android
• MonoGame.iOS
• MonoGame.Web

The current packages would get deprecated as the corresponding updated package is merged and hidden when we have a new major release.
Thoughts?

[harry-cpp]

+1 for droping DX and OpenGL nugets
-1 for removing Framework from the name
+1 for removing Framework from the name for 5.0 version

[harry-cpp]

PS. nuspec files will get removed, give me a sec to make DesktopGL PR, I know exactly how to properly set it up :)

[Jjagg]

Why 5.0 (and not 4.0)?

[harry-cpp]

Why 5.0 (and not 4.0)?

Because of the namespace, for now I wouldn't change the nuget name or the .dll name as its breakage for no good reason, however for 5.0 we are breaking stuff so we might as well shorten it.

[harry-cpp]

DesktopGL csproj is ready, I just need to fixup nuget library copying and I'll be ready to submit it.

[harry-cpp]

DesktopGL done: #6772

[mrhelmut]

I'm all in for this, especially for dotnet core/netstandard support.

Also pinging @tomspilman because I am unsure if consoles support the VS2017+ compiler yet (it produces IL code that may not be compatible), though I guess this can be addressed by having consoles using old-style csproj to get them running on VS2015. We'll just have to make sure that the MG code base keeps being compatible with Net4.5.

[harry-cpp]

Well consoles already have their own csproj files so thats not an issue.

[mrhelmut]

Hurrah then!

[harry-cpp]

We still have to use "old" C# tho.

[YTN0]

@Jjagg @cra0zy Are you guys going to update the main github page to reflect the new process for building from source? It's still showing the protobuild process.

Also, are the solution files still valid? After syncing to the latest changes, I tried building the windows dx version using the csproj file as well as the solution file, and neither worked.

I am guessing you guys are still merging the necessary changes, or is it supposed to be buildable at this point?

[Jjagg]

Are you guys going to update the main github page to reflect the new process for building from source? It's still showing the protobuild process.

Yes, we are.

Also, are the solution files still valid? After syncing to the latest changes, I tried building the windows dx version using the csproj file as well as the solution file, and neither worked.

Yes, they are. I just tried a fresh clone and succesfully built MonoGame.Framework.Windows.sln (generated by Protobuild) with MSBuild and succesfully built MonoGame.Framework.WindowsDX.sln with both MSBuild and dotnet build. What went wrong for you when building? @YTN0

[harry-cpp]

I know whats the problem with build failiure, its the fact that it pucks up assembly info from obj folder, I fixed that in my CAKE PR.

[tomspilman]

Well consoles already have their own csproj files so thats not an issue.

Currently they don't... the console versions used Protobuild to generate their C# projects for the MG framework assembly. And because of dumb cross compiler issues we need to avoid using VS2017's C# compiler as it generates some unsupported IL.

That said moving forward yes we need to generate our own C# projects in the console repos. The complication is hopefully not needing to change every project file each time new code is added upstream.

For now console work from the develop_console branch so none of these changes affect it yet. But i should fix all the console branches in the next month to use the latest.

[YTN0]

@Jjagg I was building in the IDE.... I'll sync and give it a try again....

[harry-cpp]

the console versions used Protobuild to generate their C# projects for the MG framework assembly.

Doesn't it still use it? Thats what I'm talking about when I say they technically already have their own project files. It pretty easy to add custom project files that just use wildcards.

[mrhelmut]

Will the new structure allow to work like Protobuild does with components? The console repos basically act as add-on to the main repo. For instance, when generating the Pipeline proj, it detects the presence of any add-on repos and automatically add the present targets (which means that, if you have multiple console versions, you don't have to build a separate Pipeline for each of them, you can have one to rule them all with a single Protobuild call).

The Pipeline/MGCB can be built with VS2017, so I guess the wildcards will do the trick here?

And then having separate VS2015 csproj files per consoles?

For now console work from the develop_console branch so none of these changes affect it yet. But i should fix all the console branches in the next month to use the latest.

Since we're using some of the latest stuff from develop, I'll most likely be up to help. I used to maintain local console versions on par with develop, though these changes will require a repo refactor.

[Jjagg]

it detects the presence of any add-on repos and automatically add the present targets

If I understand correctly this can be done by importing .targets files for each platform with the condition that they exists in the locations where you'd expect it if you have the console repos cloned (e.g. MonoGame/Switch/pipeline.targets).

[mrhelmut]

Right now, console repos are to be cloned as sub-directory to the main repo. It's something like this.

⤷ Build
⤷ MonoGame.Framework
⤷ MonoGame.Framework.Content.Pipeline
⤷ Tools
⤷ Protobuild.exe
⤷ Switch
    ⤷ Build (with its own protobuild component definitions)
    ⤷ MonoGame.Framework
    ⤷ MonoGame.Framework.Content.Pipeline
    ⤷ Tools
    ⤷ Protobuild.exe
⤷ XBone
    ⤷ Build (with its own protobuild component definitions)
    ⤷ MonoGame.Framework
    ⤷ MonoGame.Framework.Content.Pipeline
    ⤷ Tools
    ⤷ Protobuild.exe

And when you run Protobuild -generate Windows from ./MonoGame, it scans the definitions from the subdirectory to add them to the main Windows target (and therefore to the Pipeline/MGCB csproj).

So yes, I think this should totally be doable with conditional targets files.

[harry-cpp]

As I said above, technically they are separate projects so we don't need to worry about them, we just have to add csprojs at the end(that use wildcards), as well as some bonus wildcards for the content pipeline tools.

[lloydjatkinson]

Would this be a good opportunity to drop the ancient XNA namespaces?

[Jjagg]

Not particularly, removing Protobuild won't cause any breaking changes in the assemblies.

[mrhelmut]

It would be cool that this is a separate process, removing Protobuild is already a heavy enough change.

There's also the question of keeping an XNA compatible version of MonoGame (which Tom will likely want). E.g. having a "MonoGame 4.0" branch for the XNA 4.0 compatibility and 5.0+ for breaking changes.

@Jjagg is there anything left to do? Is the first post up to date? I might have some time to help.

[Jjagg]

@mrhelmut I'm currently having some trouble getting #6824 to work, I definitely could use some help with that one. The issue is with the tests. The old tests don't work because

• On Windows old MSBuild is used.
• On Mac we get DllImportExceptions that I haven't been able to fix yet (I can't test on Mac, so iteration is really really slow).

I think the best way forward is to use the SDK-style projects for the tests and drop the old ones altogether. In the process we'll fix any remaining issues with the SDK-style projects. Right now I'm running into native crashes running MonoGame.Tests.DesktopGL on Linux. I think it has to do with incorrect native dependencies. So I think if you want to help, the best thing you could do right now is check out my branch from #6824 and help get the MonoGame.Tests.DesktopGL and MonoGame.Tests.WindowsDX` projects passing.

Once we fix up that one, only the Pipeline Tool is left. To complete the migration, we need to finish the CAKE build script so it fully replaces the current build system.

[vpenades]

I've been checking the nuget packages, and, althought the latest MonoGame.Framework.DesktopGL is a multiplatform project targeting both Net45 and NetStandard, the corresponding nuget package is still being packaged as supporting only Net45.

[Jjagg]

The DesktopGL project that gets packed by the build bot is still the old project generated by Protobuild.

[hach-que]

Heads up folks, given the improvements made in the .NET ecosystem over the last few years with regard to cross-platform support (particularly through .NET Core and .NET Standard), I intend to retire and archive the Protobuild project as of Jan 1st 2020. I noticed there's already work in this thread underway to replace it, but I thought I'd give you a heads up anyway.

[Jjagg]

@mrhelmut Check out the original comment. Basically we should just publish the NuGets. The most important thing to fix is MGCB's dependencies on .NET Core. To fully replace the old setup we also need to distribute the Pipeline Tool, and provide VS templates and the DesktopGL VideoPlayer (so the Mac project can be dropped).

[mrhelmut]

I'm trying to catch up, please let me know if I'm missing anything.

So for now, the Content.Builder package is not built within the main repo, right? @cra0zy could you merge that and make the Cake script to package it? (or point me to the nuget definition)

Then I believe it's going to be about updating this package to use global tools, right?
MGCB already works as a global tool and installs/runs fine.

As far as I have looked into this, core templates are not packaged as nugets yet?

So, if I get this right, tasks are:

• updating content builder to use global tools (and eventually making it to fetch the tools if needed)
• packaging it as nuget
• make sure the templates get this right (update the references to MG's own nugets)
• publish templates nugets, libraries nugets, builder nugets, tools nugets

And we should be good for the Core part?

Then figuring out how to distribute the Pipeline tool and fix the tests.

[Jjagg]

So for now, the Content.Builder package is not built within the main repo, right? @cra0zy could you merge that and make the Cake script to package it? (or point me to the nuget definition)

Agreed, would be great to get this in the main repo.

MGCB already works as a global tool and installs/runs fine.

I said this in other discussions already, but there are some dependency issues with it. We should make sure all dependencies work correctly on the three desktop platforms. I know for example that nvtt (texture compression) crashes on Linux.

As far as I have looked into this, core templates are not packaged as nugets yet?

Not by the build script, but to packages them you can simply run dotnet pack on the template project.

So, if I get this right, tasks are:
...

Yeah, that's the things we need to make the SDK-style projects easily consumable.

[Jjagg]

Closing since all main things from this issue are either done or have a separate issue for them.