Msbuild: DSL based generic project format with remotely acquirable plugins

[gulshan]

TL;DR: __Gradle is good. Be like Gradle.__

Modern build tools are not just build tools, but they require complete project management tooling capability. This includes-

• Project description
• Build procedure
• Test and other task running capabilities
• Integration with package management tools (like nuget or npm)
• Support plugins (and support for pulling plugins on-demand over Internet from some repository)
• Versioning, deployment and continuous integration (Specialized build and task management)
• Basic integration with version control tools (like querying current state)

Currently MSBuild has most of these capabilities. But those capabilities are obscured behind it's XML based file format IMHO. I do not see these as short term goals. But as a long term vision, I think MSBuild can look at other successful build tools like __Gradle__ and incorporate some of their good things. I shall try to describe a few such points comparing MSBuild with Gradle-

• __DSL Based format:__ Most of us programmers use procedural techniques for programming. So, procedural languages always clicks for us. So, a procedural DSL will be a lot easier to our eyes than current declarative XML. Specially when we try to tweak the build process a bit, or add some little extra step, XML becomes an obstruction. But a DSL will be much easier. Many a times, current .*proj files are like opaque binary files to developers- no hand-editing, only edit through Visual studio dialog. Hand-editing a DSL will be trivial. Ant and Maven was XML based and Gradle choose a DSL, which was good for them. With __Powershell__ going cross-platform, I feel DSL can be based on it. DSL from Cake project can be another option. But others may have different choices. And this can be discussed later.
• __Generic project format and Plugins:__ I think MSBuild should become more generic and avoid coupling with other tools. There need not to have different project formats or *.*proj file extensions for different kind of projects. Even the full project file name can be generic and constant like project.msbuild or project.build. And plugins can be used to manage project types. Gradle uses project.gradle files and plugin specifies the project type. And, these plugins should reside in a repository in web with a qualified name and a specific version can be pulled by MSBuild over Internet on demand- just like Gradle. And the good thing is- multiple plugins can be applied to same project. So, different language files like C# and F# will be able to share same project. And MSBuild should not be tightly coupled with Nuget or even Visual Studio. I should easily be able to use MSBuild without any IDE, by just hand-editing project files and invoking commands from terminal. Visual studio will be just internally using MSBuild for its project management and provide a GUI for editing project file. Some more thoughts about project format in this comment.

I know many of the things I am proposing are long shot. Some of these capabilities are currently managed by others tools like Dotnet CLI or Common Project System of Visual studio. But this is what I feel the future of MSBuild should be like. And this may solve many current dissatisfactions with MSBuild.

What I am proposing may look like this project.build file (which replaces ConsoleProject.csproj file)-
``` C#
using Microsoft.CSharp;

Plugins.Apply(Microsoft.CSharp.ConsoleApplication, "1.0.1");

CSharp{
TargetFramework : "netstandard1.6",
Compile.Include : "***.cs",
Information{
ApplicationName : "Console App",
VersionName : "1.0",
Author : "Mr. Awesome"
}
BuildTypes {
Debug {
LogEnabled : true,
LogLevel : 3
},
Release {
LogEnabled : false,
Native : true
}
}
}

Dependencies{
Project{
"ExternalProject"
},
Library{
"ExternalLib" : "../ExternalLib.dll"
},
Nuget{
"Microsoft.NETCore.App" : "1.0.0",
"Microsoft.NET.SDK" : "1.0.0"
},
Myget{},
NPM{}
}

Tasks{
CleanBuild(){
if(BuildDirectory.Exists()){
Clean();
}
Build();
}
}
```

Another format using .net class can be found here.

[Mike-E-angelo]

Solid post, @gulshan. Thank you for your contribution to the community. 👍

[MaximRouiller]

@gulshan Would love to see a sample of this DSL that you are suggesting.

Not as in "design me a DSL" but more of a "Here's how it could look like".

Would help gather support for your cause.

Thanks to @Mike-EEE for moving me from one issue to the other. 😉

[Mike-E-angelo]

Haha right, @MaximRouiller? My baby has already grown a little long in the tooth for only being around 6 months, haha. But, it's nice to capture the history, sentiment and dialogue, IMO. Hopefully along with issues like this one it will lead to consensus, direction, and most importantly _action_ towards improving MSBuild.

[MaximRouiller]

@Mike-EEE

My belief is, it will end up like ASP.NET. It will get rewritten. 😉

Thing is... you don't want to rewrite the build system while Xamarin has just been bought and you just rewrote ASP.NET. From scratch. Too many moving pieces and too many customers who will have a "Taco Bell bathroom experience" but with bricks.

So you change one piece at a time. First, ASP.NET. Once everything is relatively stable and you stop re-inventing the framework, then you can move to a bigger project. I already see investments being made to the project system with dotnet/roslyn-project-system.

Now, whether my prediction about an MSBUILD rewrite is to hold true or not... that's a WHOLE different story. May never happen. The only thing I know is that there were always one constant in our domain.

Change.

[gulshan]

@Mike-EEE Thanks for the support!
@MaximRouiller The project file project.build can look something like-

[ Example moved to [top post](#issue-186506848) to accumulate votes 😄 ]

And I am not proposing to fully rewrite MSBuild with all its internals. I am just proposing a DSL frontend with some new features. Let's see what happens.

[Mike-E-angelo]

@MaximRouiller I have a little more faith in this than you do, I suppose. We are not asking for sweeping changes to the internals, but how _input_ is fed into those internals. @gulshan you have inspired me. Along with @jnm2's challenge in the other thread I am going try to assemble something that looks remotely close to what I am after here. 👍

[MaximRouiller]

@Mike-EEE Thread carefully. We may find ourselves with a 25th build system.

Something similar already exist with Cake.

[Mike-E-angelo]

Yeah, but with MSBuild idioms and entities. Like @gulshan says, let's see what happens. 😄

[jnm2]

Hey actually guys, Cake is not like msbuild. Cake is build orchestration, exactly like the new TFS build definitions. Both Cake and TFS call into msbuild, or vstest, or NuGet publish, etc. Neither Cake nor TFS build actually call csc with .cs files or anything like that.

http://www.cakebuild.net/dsl

(Cake is frickin awesome though...)

[jnm2]

If you're writing a new build system, the most important factor to me is that you cleanly isolate the project definition from the build script. The project definition must _absolutely_ be treated as data and be readable and writable by tooling. The build script will then gather the information it needs through a project definition format adapter which reads the project definition. (You should have multiple pluggable format adapters so that the build system can gather the same information from an XML csproj or a project.json or your own custom format that you like. That way people can continue using Visual Studio csprojs which is an important capability for the new build system.) The build script itself could be a full DSL though.

[Mike-E-angelo]

You should have multiple pluggable format adapters so that the build system can gather the same information from an XML csproj or a project.json or your own custom format that you like...

That pretty much sums up my position on this as well. The project file is simply an input _data_ file (which is described in POCOs, that way it can leverage tooling greatness) which is then used by the build _process_.

In my ideal world, the process file(s) are also comprised of serialized POCOs and can also be described in different formats as well. Figuring out how to do this would involve some work, as there would have to be a registry of sorts to know all supported formats and to select the appropriate serializer to load it into memory. But, for the sake of discussion we can keep it simple and say that it is simply XML. Or simply JSON.

I lied. Nothing is simple here. 😛

[jnm2]

Just to be clear... I know what you mean, but I would consider POCO an _implementation detail_ of the format adapter. If I was writing such an adapter, I would _not_ want to use POCOs because they would be slow and heap overhead. I'd prefer to do fast forward-only reading of the XML or JSON or other text, as or if the build system needs it, rather than construct a massive single universe of every possible setting in the linked project files.

[gulshan]

If you're writing a new build system, the most important factor to me is that you cleanly isolate the project definition from the build script.

@jnm2 Initially, I was also in that camp. But, this is what changed my mind. Generally there are a lot of types of projects supported by a build system like MSBUILD. So, it can be a C# asp.net app, a VB Windows Forms app, a F# Xamarin app or C++ server app, but will have different project properties. If we have separate project definition file, be the format JSON, XML or POCO, either you go with a generic and lenient schema, or you have multiple project types with strongly enforced schema. (Current MSBuild is probably worst of both worlds- multiple project formats- .csproj, .fsproj, .vbproj etc and yet all follow same generic structure and schema. Thus ItemGroup and PropertyGroup things cannot be removed and nobody is happy.) Consider some type of project needs some new properties/attributes. Then both of these approaches may not be so user-friendly.

Here I am proposing not only a DSL but also a plugin based architecture. So, all the project types are actually plugins of MSBuild. And they will define their own format of project description. And DSL will be used to provide that description, with strongly-typed support. Plugins can easily update or even change their project format, which will be carried to the developers by errors or warnings in DSL. BTW, plugins are all versioned, so no worry of breaking projects if plugin version not updated. Multiple plugins can be used in same project. Using Typescript with asp.net will be just setting up plugins. This kind of flexibility is very hard(even if possible) with simple data formats.

Another thing is, most of the projects do not use custom build logic. So, even if build logic is allowed in DSL based format, very few people will use/see it. And one can always separate the custom build logic into a separate file and include that in the project file.

And lastly, Gradle is using DSL very successfully for project description. You rarely hear or see any complain about the project format. So, as I have said, MSBuild can just be like Gradle. Oh, MSBuild also allows project description from build logic in same file!

[danrien]

I would prefer something like the above, except idiomatic C# rather than something gradle-y. Better yet, why not just ask for a C# file (with a special name, i.e. build.csc) that implements a spec? It could even just support any old CLR language! This would allow a developer to be as flexible as needed.

This would also provide an avenue for easy backwards-compatibility with existing build systems like the existing MSBuild csproj files, which could ship with an csproj compatible implementation of the spec.

[Mike-E-angelo]

I would not want to use POCOs because they would be slow and heap overhead.

Ah that's the thing, the XML and JSON are describing objects that the build process understands, correct?

Alright, I took some time to provide a _VERY_ rough sketch of what I am thinking of, and have posted that here: https://github.com/Microsoft/msbuild/issues/613#issuecomment-262339197

Please let me know if you have any questions. Sorry for the cross-posting. Now I'm confused more than usual. 😛

[gulshan]

@Mike-EEE Making model the default namespace and removing unused namespaces will make the file cleaner I guess.

[Mike-E-angelo]

Yes @gulshan, Xaml is chatty and verbose, and that is a valid criticism of the format. There are new flavors emerging such as OmniXaml that are tackling that aspect. However, what is important here is the features that are gleaned from this, which I hope the screenshot successfully captures.

Again, the idea here is that the content that you see in Project.xaml would also be described in a Project.json (or Project.xml or Project.ini) as well -- as well as being compliant in the format in which it is described.

[gulshan]

I got a comment on this proposal on Twitter saying-

I think we already have too many scripting languages for this. Another DSL that I edit infrequently is a no-go.

And I that quite true- a language already known to user will be better than a new DSL. In fact, Gradle do not made a new DSL, they used Groovy and now Kotlin scripts. So, I thought, how about using C# Script in this case. So, my DSL example in the top post became this in C# Script project.build.csx-

``` C#

r "nuget: Microsoft.CSharp/1.0.1"

using Microsoft.Build;
using Microsoft.CSharp;

Project = new CSharp.ConsoleApplication{
TargetFramework = TargetFrameworks.Netstandard16,
FileIncludePattern = "***.cs",
ProjectInformation = new Information{
ApplicationName = "My Console App",
VersionName = "1.0",
Author = "Mr. Awesome"
},
BuildTypes = new List{
new BuildType{
Name = BuildType.Debug,
LogEnabled = true,
LogLevel = Log.Debug
},
new BuildType{
Name = BuildType.Release,
LogEnabled = false,
Native = true
}
},
Dependencies = new List{
new ProjectDependency{ Name = "ExternalProject", Path = "../ExternalProject" },
new AssemblyDependency{ Name = "ExternalLib", Path = "../ExternalLib.dll" },
new NugetDependency{ Name = "Microsoft.NETCore.App", Version = "1.0.0" },
new NugetDependency{ Name = "Microsoft.NET.SDK", Version = "1.0.0" },
new NugetDependency{
Name = "Microsoft.NET.Tools-pre", Version = "1.0.0",
SourcePath="https://www.myget.org/F/Microsoft.NET.Tools-pre" // For Myget
}
//new NPMDependency{}
}
};

Task async CleanBuild()
{
if (BuildDirectory.Exists())
{
await Clean();
}
return Build();
}
```

The pros in this case is-

• This is just regular C#, no need to learn a new language. The intellisense and other tooling is there to help.
• As MsBuild itself is build with C#, there will be very fluid connection between the tool and the script.
• No separate repository is needed for plugins, Nuget can be used in this case. I am using a planned C# script feature to load plugin from Nuget.
• If someone wants, they can use other .net scripting like F# script with a project.build.fsx file. After all, it's .net API, can be used by any .net language.
• It's kind of similar with the POCO based project format suggested in #613 , but come with APIs also, which is more straight forward I guess.

[Mike-E-angelo]

Yessss.... you got it, @gulshan. This is why POCO is the best-o. Ha ha. Once you are using plain ol' CLR objects, you can do anything with them, which then means:

• You can _serialize_ them in any format in which you have a supporting serializer. Which THEN means
• You can design those objects in a _designer_ that is built for that format. Which THEN means
• _Cheaper development and maintenance costs for your solutions_.

Lower TCO is the way to go, yo. 😛

but more straight forward I guess.

Yeah, that issue has unfortunately been butchered by conversation and misunderstandings. I've actually been working to flesh out the POC above that demonstrates more of what we're after here. I will try to incorporate more of your modelling efforts. 👍

[jnm2]

@gulshan My only problem with this is how does the NuGet package manager work? How do plugins like ReSharper add or remove references automatically? And how do PowerShell scripts work that are intended to have the same effect on each project, like installing NuGet packages or adding files to the compilation?

In every case the changes would not be persisted without editing the script you wrote, which could be as hard as solving the halting problem.

[gulshan]

@jnm2 Again borrowing from Gradle. Yes, other tools do have to edit the script to persist changes. For them, there has to be a tooling API, like Gradle has one.

[gulshan]

Looks like #16 had some discussion about scripting and to use C# script. It also seems, Microsoft is(was?) cooking something regarding the build system in private-

We don’t have anything to announce right now, but you might be interested in some things that have been posted externally.

[jnm2]

This API allows you to execute and monitor builds and to query Gradle about the details of a build.

@gulshan I read the whole page and see nothing about modifying the script, only about executing and monitoring and querying.

I'm not sure what their approach would be, but you can't get around the halting problem without severely limiting the freedom of the language.

[gulshan]

@jnm2 Actually, as I have used, IDE do change the script, if I change something from GUI. May be some custom API is in use.

After some searching it seems Gradle do have the possibility of Halting problem. Some links about it-

• http://wiki.apidesign.org/wiki/Gradle
• http://delivervalue.blogspot.com/2015/03/so-what-about-this-halting-problem-in.html
• http://blog.cacoethes.co.uk/gradle/comments-on-recent-gradle-criticisms

And it seems, Gradle went the way of using existing language instead of building a new restricted one.

As mentioned in the first link, limiting the language from indefinite looping is the solution. Can that be done using Roslyn analyzer?

[jnm2]

As mentioned in the first link, limiting the language from indefinite looping is the solution. Can that be done using Roslyn analyzer?

Not without solving the halting problem. :D

[jnm2]

To be fair, you can warn if you encounter for, while, goto or recursion (or delegate invocation, any could lead to recursion) but you can't tell whether it's dangerous or not.

To be even fairer the halting problem isn't the immediate problem, though it could become part of it. The immediate problem is along the lines of, how does a tool add references if I'm generating the references list using linq off some JSON.NET object I read in from another whole file?

It's the same problem any time you move from a declarative format to a scripting format.

[gulshan]

I would rather make these constructs(recursion, goto, while and for) outright illegal to prevent halting problem, by raising compilation errors. foreach will be the only looping construct then. If possible, the IEnumerable (on which foreach is looping) has to be immutable inside foreach block, using Roslyn.

And I fail to see the problem with adding reference within a script. Is it impossible or source of other problems like halting problem? If it were impossible, Gradle could not have done that.

[jnm2]

Well, here's what I'm thinking. How does NuGet package manager or command line deal with installing a new package into this?

```c#
Project = new CSharp.ConsoleApplication
{
Dependencies = ReadDependencyFile("customformat")
};

IReadOnlyCollection ReadDependencyFile(string path)
{
using (var file = File.OpenText(path))
return new JsonSerializer().Deserialize>(file);
}
```

[gulshan]

I think, tools should be working with convention. In this case the convention is- defining the dependencies within actual project file. There can even be multiple conventions. Then a tool has to build support for each of the conventions separately. So, keeping the dependencies in a separate file can be another convention, which has to be supported by tools separately. But if user goes out of convention, then s/he cannot expect support from tooling and has to assume responsibility for that.

For example, now MSBuild is introducing new features in the project file/format. So, other tools like Resharper has to build support for them. There is no automatic support.

[gulshan]

1392 , #1436 , #1439 and dotnet/sdk#436 all are discussing SDKs and other build time tools and how they can plug into MSBuild. These SDKs and tools are similar to my proposed plugins. Just there shuold be a lot more scope of programmability for them.

[laurencee]

A very nice clean suggestion by @galich

https://gist.github.com/danfma/c45087b2464cfb10e4f9935921f1acd7#gistcomment-1907390

c# Project .UseCSharp(6) .WithPackage("Microsoft.NETCore.App", "1.0.0") .WithPackage("Microsoft.NET.SDK", "1.0.0") .WithFramework("netcoreapp1.0", dependencies: framework => framework.AddSomething()) .Compile("**/*.cs") .OutputExe();

[gulshan]

Following different discussions, I am starting to envision the high level architecture of my proposal-

• MSBuild should supply an interface (or abstract class) like IProject and extension class(es) based on the interface supplying various functionality.
• Among the public APIs, an attribute like [MSBuildTask] along with a special signature will be used to select methods, which will be available as "MSBuild Tasks" and can be invoked from command line or other tools.
• There will be plugins to MSBuild. Plugins are regular .net libraries, which can provide many things on top of MSBuild supplied API-
  
  
  
  • Implement the IProject for a certain type of project like CSharpProject or FSharpProject.
  
  
  • There can be child project inheriting other projects defined by other (or same) plugins like FSharpLibrary or ASPNETWebApp. That means, a plugin can depend on other plugins.
  
  
  • Add extension classes/methods for IProject and implementation.
  
  
  • And any other crazy stuff you can think of.
  
  
• Sorry, I think some crazy stuffs should be disallowed in plugins to prevent unpleasant outcomes like indefinite looping (aka the halting problem). The restrictions will be applied by roslyn analyzer. One such restriction is no recursion, goto, while loops and for loops are allowed. Only foreach over a read-only/freezed collection will be allowed.
• The main MSBuild has to be installed as an app/tool. But plugins will be acquired using Nuget. That will facilitate version and dependency management.
• Plugins can be acquired on project basis. But there should be aggressive caching in use to avoid fetching same version of some plugin multiple times.
• I prefer plugins to be hosted in a separate repository other than nuget.com, so that all the plugins will be rigorously tested, manually reviewed by the a responsible team as well as rated by users.
• The end user or developer will be using a regular .net class file implementing the IProject interface to define/describe a project. But commonly it will be inheriting a project class supplied by a plugin. A partial class definition scheme can be used along with XAML files in this case like being used in WPF. (Hey @Mike-EEE )
• The project file will refer to the plugins it needs using C# script nuget package reference syntax mentioned here- dotnet/roslyn#5654.
• The project class will define/describe itself in the public default(parameter-less) constructor. So, a project/plugin can provide a constructor, which actually build the project reading from a xml, json or any other type of file. (But there is a concern for external tools here.)
• End developer will also be able to define MSBuild tasks, matching the attribute and signature already mentioned.
• Other abilities and restrictions of plugins would also be applied to project class files using roslyn analyzer.
• There will be a MSBuild tooling API, which will enable the external tools to read (and edit if necessary) the project class file. Visual Studio will just be one such tool.

A slightly changed version of the project class file according to this architecture-

``` C#

r "nuget: Microsoft.CSharp/1.0.1"

using Microsoft.Build;
using Microsoft.CSharp;

public class MyConsoleApp : CSharp.ConsoleApplication
{
public MyConsoleApp()
{
TargetFramework = TargetFrameworks.Netstandard16;
FileIncludePattern = @"***.cs";
ProjectInformation = new Information{
ApplicationName = "My Console App",
VersionName = "1.0",
Author = "Mr. Awesome"
};
BuildTypes[BuildType.Debug] = new BuildType{ LogEnabled = true, LogLevel = Log.Debug };
BuildTypes[BuildType.Release] = new BuildType{ LogEnabled = false, IsNative = true };

    Dependencies.AddRange(new List<Dependency>{
        new ProjectDependency{ Name = "ExternalProject", Path = "../ExternalProject" },
        new AssemblyDependency{ Name = "ExternalLib", Path = "../ExternalLib.dll" },
        new NugetDependency{ Name = "Microsoft.NETCore.App", Version = "1.0.0" },
        new NugetDependency{ Name = "Microsoft.NET.SDK", Version = "1.0.0" },
        new NugetDependency{     // For Myget
            Name = "Microsoft.NET.Tools-pre", Version = "1.0.0",
            SourcePath="https://www.myget.org/F/Microsoft.NET.Tools-pre"
        }
        //new NPMDependency{}
    });
}

[MSBuildTask]
public async Task<Artifact> CleanBuild()
{
    if (BuildDirectory.Exists())
    {
        await Clean();
    }
    return Build();
}

}
```
Any thoughts?

[jnm2]

Add to recursion prevention that you can't invoke a delegate. As a corollary, you also can't use LINQ.

I think that a better idea is to time out after a second (by default) if the code hasn't finished running.

[danrien]

Why is it up to the build system to prevent halting? Developers should be
allowed to break their build this way if they want to IMO. It should also
be up to them to not break it, if that's what they want.

On Thu, Dec 8, 2016 at 10:16 AM, Joseph Musser [e-mail]
wrote:

I think that a better idea is to time out after a second (by default) if
the code hasn't finished running.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/msbuild/issues/1289#issuecomment-265781187,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA2j38HFr0XUok6CVDbmQ2ldVo8qYWbyks5rGC11gaJpZM4Kl-Za
.

[gulshan]

The minimal project file according to https://github.com/Microsoft/msbuild/issues/1289#issuecomment-265740595 -
```C#

r "nuget: Microsoft.Build.CSharp/1.1.0"

using Microsoft.Build;

public class MyConsoleApp : CSharp.ConsoleApplication
{
public MyConsoleApp()
{
TargetFramework = TargetFrameworks.Netstandard16;
FileIncludePattern = @"***.cs";
}
}
```

[laurencee]

@gulshan I'd think that file include pattern would be a good safe default value (considering you declared it as a CSharp.ConsoleApplication) so you probably shouldn't even need to specify it.

[Mike-E-angelo]

Do I hear a ONE LINE PROJECT??? :)

[gulshan]

@laurencee @Mike-EEE Going all default in minimal project? Sounds good to me. 😄
```C#

r "nuget: Microsoft.Build.CSharp/1.1.0"

public class MyConsoleApp : Microsoft.Build.CSharp.ConsoleApplication {}
```

[galich]

As long as minimalistic build tool compiles and runs C# app that performs actual build - I'm ready to grow 3rd hand to say 👍 !

@gulshan version is a good one and can be built on top of minimalistic build tool. As well as many others, people tend to have different opinions and ideas are changing over time. Look at ORM, DI or ASP.NET - some prefer attributed syntax, some prefer fluent, some load models from config files. Keep the door open for alternative ideas.

[gulshan]

@galich As this is just .net code, one can make a plugin to inherit or extend a project type and provide all sorts of things. All the doors are wide open!

[gulshan]

Some more description of upcoming SDKs is provided in #1493 . An programmatically implementable .net interface ISdkResolver will be used for integration with other tools like Visual Studio, VS Code, Monodevelop, plain CLI. Good to know.

[rainersigwald]

Team triage: We don't plan language overhauls in the foreseeable future.