Aspnetcore: Discussion: unifying to fewer GitHub repos

[natemcmaster]

This is a discussion item for https://github.com/aspnet/Announcements/issues/320. Please see the announcement for details.

Status

Repo | Code moved | Issues moved | Repo archived
---------------------------------------------|------------|--------------|--------------
AADIntegration | ✓ | ✓ | ✓
Antiforgery | ✓ | ✓ | ✓
AuthSamples | ✓ | ✓ | ✓
AzureIntegration | ✓ | ✓ | ✓
BasicMiddleware | ✓ | ✓ | ✓
Caching | ✓ | ✓ | ✓
Common | ✓ | ✓ | ✓
Configuration | ✓ | ✓ | ✓
CORS | ✓ | ✓ | ✓
DataProtection | ✓ | ✓ | ✓
DependencyInjection | ✓ | ✓ | ✓
Diagnostics | ✓ | ✓ | ✓
DotNetTools | ✓ | ✓ | ✓
EventNotification | ✓ | ✓ | ✓
FileSystem | ✓ | ✓ | ✓
HtmlAbstractions | ✓ | ✓ | ✓
HttpAbstractions | ✓ | ✓ | ✓
HttpClientFactory | ✓ | ✓ | ✓
HttpSysServer | ✓ | ✓ | ✓
Hosting | ✓ | ✓ | ✓
Identity | ✓ | ✓ | ✓
IISIntegration | ✓ | ✓ | ✓
JavaScriptServices | ✓ | ✓ | ✓
JsonPatch | ✓ | ✓ | ✓
KestrelHttpServer | ✓ | ✓ | ✓
Localization | ✓ | ✓ | ✓
Logging | ✓ | ✓ | ✓
MetaPackages | ✓ | ✓ | ✓
Microsoft.Data.Sqlite | ✓ | ✓ | ✓
MusicStore | ✓ | ✓ | ✓
Mvc | ✓ | ✓ | ✓
MvcPrecompilation | ✓ | ✓ | ✓
Options | ✓ | ✓ | ✓
Proxy | ✓ | ✓ | ✓
Razor | ✓ | ✓ | ✓
ResponseCaching | ✓ | ✓ | ✓
Routing | ✓ | ✓ | ✓
Security | ✓ | ✓ | ✓
ServerTests | ✓ | ✓ | ✓
Session | ✓ | ✓ | ✓
SignalR | ✓ | ✓ | ✓
StaticFiles | ✓ | ✓ | ✓
Templating | ✓ | ✓ | ✓
Testing | ✓ | ✓ | ✓
Universe | ✓ | ✓ | ✓
WebHooks | ✓ | ✓ | ✓
WebSockets | ✓ | ✓ | ✓

[poke]

What will this mean for the solution files? Will there only be a single solution with all projects in that repo?

I assume this will make it a lot easier to contribute since we will no longer have to cross reference projects manually to make things compile but at the same time it will also make things a lot more confusing considering the high number of components in the project.

I do have some concerns about the Extensions projects though since these always felt very separate and individual to me, so merging unrelated things into a single repository does feel a bit weird. Especially considering that usually Git practices often encourage smaller repositories over large ones.

That being said, I applaud your approach to combining the repositories by merging the existing histories. I do consider keeping the original Git objects intact very important, so this is a very good move!

[Tragetaschen]

Since the existing repos all follow a very, very similar structure, are you going to extract/replay the git history for subfolders like src into the merged projects?

[Tragetaschen]

Is there a way to "Subscribe" to parts of the merged projects? For example, I'm subscribed to SignalR and Kestrel, but not to MVC or Razor

[Tratcher]

@poke We'll start by merging repos as subfolders with their existing structure inside, no changes to sln or other project files. Long term we don't plan on merging slns, VS can barely open them as is, but we will plan to unify some of the other build infrastructure.

[natemcmaster]

What will this mean for the solution files? Will there only be a single solution with all projects in that repo?

As Chris said, there won't be one big .sln file for all projects in larger repos. Despite our efforts, VS can't handle 700+ .csproj files yet. So, instead, we'll have multiple solutions. Each represents a "family" or workload of projects within the repo, such as Kestrel.sln, Mvc.sln, ChrisRossDoesHttp2Work.sln. Some of our larger repos, such as EF Core, already do this.

I do have some concerns about the Extensions projects though since these always felt very separate and individual to me

For what it's worth, this isn't changing the nature of the product. I view this as an update to our code-splitting strategy which reflects the reality of how the product ships. We ship all of those extensions packages on the same cadence using the same rules for patching, servicing policies, versioning, etc. Consumers should not be affected by reorganizing source code.

Especially considering that usually Git practices often encourage smaller repositories over large ones.

That's what started us down the road to where we are now: 53 git repos, some of which only produce 1 assembly. As the number of repos has grown over time, it has become increasingly expensive to manage them all. IMO the ideal number of git repos is somewhere ≥ 1 and ≤ # of devs working on a team.

Since the repos all follow a very, very similar structure, are you going to extract/replay the git history for subfolders like src into the merged projects?

Not exactly. We're going to be doing some reorganization of folders and then merging unrelated histories into the same tree. I'm playing with fun git things right now like octopus and subtree merges to find one that makes git log and git blame usable, if possible.

Is there a way to "Subscribe" to parts of the merged projects?

This is a GitHub notification limitation that regrettably has no good workarounds yet as far as I know. We'll use labels on issue trackers to group issues based on product area, like SignalR and Kestrel. Personally, I've stopped using email notifications and use octobox instead. I made a PR to this project a while ago to support filtering on labels.

[poke]

We'll start by merging repos as subfolders with their existing structure inside

Wait, does this mean that the repository will basically look like this then?

/
  Kestrel/
    src/
    test/
    Kestrel.sln
  Mvc/
    src/
    test/
    Mvc.sln
  …

Or will all the existing directories basically be merged, so that e.g. a current /src/ Microsoft.AspNetCore.Mvc.Core/ will still be at exact that same path in the new repo?

This has a huge impact on compatibility of the existing Git objects and basically decides on whether you will have to rewrite all changes (creating new objects for pretty much everything, breaking compatibility), or whether the existing changes are just being merged (retaining the full history without any changes).

I would very much like if e.g. a commit 956441aa68ce7516bce69fda616a0fe060c83882 that currently exists in Mvc, would still exist as an identical commit 956441aa68ce7516bce69fda616a0fe060c83882 in the new Home repo.

---

We ship all of those extensions packages on the same cadence using the same rules for patching, servicing policies, versioning, etc. Consumers should not be affected by reorganizing source code.

It does have an effect on everything related to issues though. Compared to the Home repo (which does have united issues for multiple repos for quite a while now), the extension repos feel very clean and focused.

But I totally do understand your motivations, especially considering how often you had cross-repo changes in the past that needed to be managed and timed properly.

[Tratcher]

Yes, it will start with folders like Kestrel/src, etc.. We are looking at how to preserve the commit history despite the observed git move. Apparently the information is all there and the main challenge is that individual tools like git blame do not traverse it fully.

[poke]

Hmm 😕 I’m not too happy with that then.

But you could achieve this nicely by adding each individual repo as a remote and merging that in directly (so that the contents are at the root), and then sorting them into subfolders in an additional commit. That way you should be able to keep the full individual histories untouched while also having log/blame continue to work because you are only doing a file move afterwards.

[Tratcher]

There's quite a few top level files that would conflict if we did that. The reverse might work though, changing the structure in the old repo and then merge it.

[TheSamsterZA]

What about, say, the Docs repositories (including the ones specifically for things like Entity Framework Core and Visual Studio Web)?

I assume a repo like Blazor would be moved under aspnet/AspLabs?

[Eilon]

@TheSamsterZA this plan is focused around the products that ship as part of .NET Core, so this will have no effect on Docs, Blazor, or other projects.

[aharpervc]

This is a GitHub notification limitation that regrettably has no good workarounds yet as far as I know. We'll use labels on issue trackers to group issues based on product area, like SignalR and Kestrel.

As someone who enjoys observing progress from the outside (not a contributor), this _super sucks_. I love watching the performance optimization PRs and discussion going on with Kestrel, and now that's not going to be possible due to being drowned out with a bunch of noise and other things I'm not interested in.

[khellang]

@aharpervc We've been through this before; https://github.com/aspnet/Home/issues/2736 😕

[poke]

@natemcmaster I just saw how you handled the merging of aspnet/WebHooks and aspnet/AspLabs. If that’s how you are going to do it for all repositories, then I’m very happy with that solution! Good job! 😄

[natemcmaster]

Thanks! Yes, that is going to be our approach.

[Tratcher]

It looks like Github file blame is working pretty well, but file history is busted.

[natemcmaster]

GitHub's file history view is busted even when you move files within the same repo. There is not much we can do about that.

https://github.com/aspnet/Proxy/commits/404fcf500bce5fe6a75b2e6b20b40923b0c112a9/src/Microsoft.AspNetCore.Proxy/ProxyMiddleware.cs

[pranavkm]

History seems to be ok if you view it through not-GitHub:

[khellang]

If only MS could pull some strings over at GitHub... 😜

[poke]

I have a quick question: The merge is still in progress and it will probably take a bit more time until it is done. But every now and then, I still see issues getting moved from aspnet/AspNetCore to more specific repositories, although the announced practice was actually to use the Home repo for all issues. And this makes even more sense now considering that the other repos will be locked soon.

What’s the exact plan there? When are issues still created in specific repos?

[Tratcher]

@poke the code migration is starting with low traffic repos and will take several weeks. Issues for high traffic repos like MVC are still being moved to their specific repos until we get ready to merge the code for those specific projects.

[Sebazzz]

@khellang Well, issue moving to another repository is now beta so at least there's that.

[Eilon]

Unfortunately there's no REST API for moving issues 😦 But we'll try to figure something out to make it less painful for everyone.

[dotnetchris]

Yay for project health and sustainability efforts

[Meir017]

@natemcmaster would it be possible to edit this to include a todo list of the migration progress?

This is a discussion item for aspnet/Announcements#320. Please see the announcement for details.

Extensions

• [ ] Mvc
  ...

  AspNetCore

• [ ] Configuration
  ...

[natemcmaster]

@Meir017 good suggestion. See above for the latest status on this.

[Librazy]

aspnet/Tooling is removed from final set of repos of aspnet/Announcements#320 so there are 4 remains but it still says "plan to condense about 53 repos into 5"

[natemcmaster]

@Librazy clever reading. Stay tuned :)

[Librazy]

Since aspnet/Announcements is definitely won't be merged so there are 5? Or says, 4 code repos as aspnet/Tooling is also not a code repo.

[poke]

“Condensing N repos into M” does not necessarily mean that there will be only M repos left in the end but that those N will be superseeded by M repos. Repositories outside of those N would be left as they are.

The announcement also says “about” so that's already an estimation, so no need to correct that 😝

[ghost]

Is Razor Server Components missing from the list? I know @SteveSandersonMS has renamed and moved the Blazor Components part into the AspNetCore repo.

[grahamehorner]

@adrian109 yes I believe it is, @natemcmaster @SteveSandersonMS are you aware of the build issues that look to be path length related ?

[natemcmaster]

Is Razor Server Components missing from the list?

@SteveSandersonMS - can you elaborate more on the plans for aspnet/Blazor? My understanding is incompletely, but I don't think aspnet/Blazor is going away.

are you aware of the build issues that look to be path length related ?

No. Can you open a separate issue for this? This appears to be a different topic.

[Eilon]

aspnet/Blazor is staying for the near future for the Web Assembly stuff.

[Eilon]

Razor Components are part of MVC, and moving to aspnet/AspNetCore.

[natemcmaster]

We have finished moving source code. The last thing left to do on this is close/transfer issues and archive unused repos.

[Matthew-Bonner]

Sorry, but I'm unclear of how it is more compelling to have unrelated commits, issues and discussions all under one repository. Could you elaborate a little more on how you came to this decision and whether you are open to any influence in the change or not? I personally think this is a bad idea, it is no different to downloading something like SilverStripe, which is a mess, but if that is the direction you want to move in I'm sure we will be able to work with it.

[poke]

@Matthew-Bonner See above:

That's what started us down the road to where we are now: 53 git repos, some of which only produce 1 assembly. As the number of repos has grown over time, it has become increasingly expensive to manage them all. IMO the ideal number of git repos is somewhere ≥ 1 and ≤ # of devs working on a team.

Basically, in ASP.NET Core everything always releases at once, so having separations is making the release process difficult. Even more so in the future, when you can only consume ASP.NET Core as a single unit, it makes sense to have everything united.

While this makes consuming the repository (and especially issues) harder, I understand the motive behind this.

whether you are open to any influence in the change or not

Considering that the move with all its hard work is now mostly done, I don’t think that decision would be taken back now.

[Matthew-Bonner]

Releases and the code are two separate concerns, why does one have to impact the other?

Why are we changing the process, instead of fixing the build tools?

Why has this only landed on the community plate now, after it is too late for us to have our say on the matter?

[Matthew-Bonner]

Also, what happens to those who use one repo, and not everything else, do you have stats on how many projects will be affected by this change?

[poke]

Why has this only landed on the community plate now, after it is too late for us to have our say on the matter?

Original announcement, before anything changed, was aspnet/Announcements#320 on October 8th.

what happens to those who use one repo, and not everything else

What do you mean, use one repo? ASP.NET Core packages cannot be used (anymore) individually.

[Matthew-Bonner]

Visibility of this has just appeared in a closely monitored inbox, maybe I need to subscribe to more alerts then I guess.

Your last statement isn't true, what stops someone downloading a repo and using it in their project, the answer is nothing, people do and are doing this.

[Matthew-Bonner]

The following is a list of repositories that are tracked on github in projects we work on:
Hosting
Mvc
Razor
FileSystem
Logging
Options
Session

In our build system, which works, we pull in only the code we need in order to build a project, the time it takes to pull down our code, restore nuget packages, and then build everything is already slow, without having to download even more. These surface pro's overheat and shut down when they have to do too much work, I suspect more overheating as a result. I'm on to my 3rd surface pro, 5th dock, and 2nd power supply, all at Microsoft's expense. I'm not sure it is cost effective to make the surface pro have to work any harder :)

[Eilon]

You can still clone the AspNetCore repo (this repo) and build just sub-folders that interest you. But, as others have noted, the key here is that ASP.NET Core is essentially one product, that starting in 3.0, will ship in just 1 package really, so building it in 1 repo makes more sense for that. As with nearly anything, it's a trade-off, and the engineering team gave this many months of thought and felt it was the best balance (including considerations of contributors).

[Matthew-Bonner]

I think the point I was getting at was how much more time will be needed to download everything, it isn't just the surface pro's overheating that we have a problem with, it is a sluggish VPN and network speeds. This is only one of the points I've made.

The other points I can live with, as the ability to track changes to specific areas of .NET Core can be resolved by viewing a list of changes per folder, and we can resolve the problem of conflated issues by using effective tagging if that is your plan to do so, I know this already happens in other .NET Core repositories, so it isn't the end of the world.

I don't know the processes inside out, so I suspect there is good reason behind this change, but I'm still not sure this is the right thing to do though, as it goes against the grain of what the industry standard is.

@poke to help me understand better, it might help to explain what you mean by

having separations is making the release process difficult

[Eilon]

@Matthew-Bonner the things you point out are definitely some of the downsides of having a lot of projects in one repo, so I won't try to convince anyone otherwise 😄 It's very much a problem for anyone, including the ASP.NET team, working on the project. But it makes many other things easier, such as making sweeping changes or large refactoring among many inter-related projects.

[Eilon]

OK pretty much all issues should be where they belong now, almost all of which now have an appropriate area-* label for easier searching/filtering. We're still doing a few cleanups here and there, but it's almost entirely done.

[aspnet-hello]

We periodically close 'discussion' issues that have not been updated in a long period of time.

We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.