Dnn.platform: RFC: Repository Structure & Developer Experience

[mitchelsellers]

Please review the problem and suggested solution and provide your comments to help drive this feature implementation into a future platform release.

Description of problem

Currently, the developer experience for DNN Platform is complicated and requires multiple steps to implement changes or to get a local environment running. The current situation has been identified and communicated by many members of the community with the following key areas of concern.

• An inability to fork the solution and select 'F5' and run the application locally, without extra steps
• An inability to make simple changes to the platform without submitting multiple pull-requests to different repositories and the processing of those pull requests being important for proper build validation
• Confusion when looking for issues, or history on an issue that requires changes to multiple repositories
• Additional burden when managing dependencies between common components. (React.Common, PersonaBar, etc.)

There are other key considerations, but the concern is that the complexity of the project and management is reducing the effectiveness or ability for even experienced developers to be able to contribute.

Proposed Solution # 1 : Additional Build Support

One proposed solution is to continue to add to the CAKE scripts to provide additional build support, up-to and including, an ability to create additional directories outside of the working directory, downloading external dependencies, and making other local changes to allow a developer experience that would work.

Current raised concerns with this environment include.

• An inability to execute these scripts on corporate environments that might have higher security requirements
• A lack of resolution of the confusion portion regarding simple changes and multiple repositories (Eg. addition of a setting)

Current raised benefits with this environments

• Continues to keep the UI & API separate from each other
• Closer to supporting a future without a Persona Bar being needed

Proposed Solution # 2: Condensing Projects

Another proposed solution is to condense the project structure to have the Persona bar contained within the main DNN.Platform solution. The Persona Bar is implemented in a provider model, however, it is the single distributed provider, this would change the repository to have all components

Current raised concerns with this environment include.

• The size of the repository will go up
• Build times will go up (Possibly limited in impact due to VS Change tracking)
• Location of changes could be harder

Current raised benefits with this environment

• Single repository to manage issues, contributions, and pull requests. Meaning a change to add a setting could be reviewed once, and vaildated fully with a single review cycle
• Developer experience simplified with a single repository to download

** Discussion Needed

An internal group discussion has been leading towards Solution 1 above, however, that work is not yet complete and there have been concerns that even if we get there that it isn't good enough. A review & final direction is needed.

[ohine]

We are very close to having solution # 1 finished. All the cake targets exist currently and we just need a cake task runner to be configured so it will get a fully working development environment configured locally. I've been working on the documentation so this can be part of the contributing guide however the release management tasks for v9.2.2 and v9.3.0 took much more effort then estimated and I haven't had a chance to complete the initial drafts for review by the group.

As for solution # 2, I believe we shouldn't just copy items into one repo for the ability to press f5. We can achieve that today and leave the current separation. Now if we want to continue what was started and add React.Common into the AdminExperience project. That would be a good logical step forward.

What we really need is constancy and good documentation to help new developers get started. Downloading a 2gb repo of 50+ loosely related projects crammed into a single solution, knowing where to go to make a simple (or complex) change is a lot to take in at once. Following the process we implemented with CDF is the logically way forward, what's missing is the getting started documentation.

With that said, If we are going to condense, we shouldn't just condense the AdminExperience and Platform. We should condense everything for consistency. There isn't a need to have 29 projects in the AdminExperience, we should combine those as well. We should also combine CK and CDF and everything that is currently the platform. This way the process is identical across the board.

This will also require JRE to be installed for any platform development since the PB currently still requires it for it's build tasks.

I'd like to see a proposed solution # 3 added to continue the separation and define what is our core platform, and what is just an addon and could be shipped separately. We could easily release updates to the HTML module on it's own, or just release a new build of the AdminExperience which does not require a full platform release. This would greatly simplify improving individual areas of the product without requiring a full upgrade cycle for end users. Right now because the majority of the addons are part of the plaform repo, it's very difficult to tell what's standalone vs what requires the platform to be rebuilt. Every version is incremented regardless of if a change was done to that component.

[david-poindexter]

This is a very interesting and complex challenge.

Having empathy for the desired contributors is very important as we weigh the pros and cons of each proposed solution. Even though things have gotten better, it is still very difficult to get a combined Platform/AdminExperience environment up and running without a bit of a "hack job". We have Issues posted in Platform that are related to AdminExperience. And I frequently have conversations with people in the community that still don't understand what is where and why. Therefore, the unknown makes them fearful to even attempt making contributions beyond very simple ones. If people in our ecosystem are having difficulty, then how can we expect to gain new contributors?

Just some thoughts worth considering in the context of the conversation here.

[donker]

I'd love to see how #1 works. But given that the admin UI "reflects" the core it feels weird that any change needs multiple issues/PRs and needs to have coordination to make sure all parts get updated at the same time for fear of having a broken build.

The "hit F5 and see it work" is unfortunately something many .net developers are expecting. I'm not saying everything needs to be dumbed down to the lowest common denominator, but we need to ensure that you can easily build/run/change/rebuild/run our common distribution.

As Oliver alludes to, there is more than just folding stuff into a single repo. The current project could do with a little cleaning in terms of what goes where. If you look at similar projects the core and UI are well distinguished through the folder structure. Obviously it is also important in this regard that developers can see what is fixed and what is extension.

[hismightiness]

I think we need to take a step back and consider our higher-level goals.

Is our highest level goal to make the project follow some sort of best practices, standards, or integrate some cool new thing? If yes, this conversation is already on the right track, and I'm definitely leaning towards option #2 - but it's not that simple.

I think Oliver's suggestion of an option 3 might be a better solution (if I correctly understood the point he was making). Actually... A combination of 1 and 3.

The DNN solution is massive and it's only going to get bigger. The decision to centralize everything into a single solution was made when there were far fewer projects, dependencies, code, and other complications. It's grown so much, that even opening the solution is a mind-blowing experience for even newbies that are .NET veterans. Heck, even the "core modules" used to be part of the primary solution at one point, didn't they? They needed to be split out for good reason.

I believe that everything that is a non-essential extension should be split out into it's own repo. This would make issues easier to find and manage. People with domain expertise could focus on and maintain those respective repos. Extensions could potentially have independent release schedules, offering resolutions to issues that affect the ecosystem in a much more timely fashion. Moreover, newbies could easily open say, the URL provider and make updates. Build. Then test in one or many environments simply by installing it. Oh, and don't forget that there would now be a better and easier to understand collection of best practice examples for each of these extension points.

If our highest level goal is for the ecosystem to grow and for the platform to grow the number of contributors, we absolutely need to consider the lowest common denominator. We need to empower new people to be able to make an update, get some confidence, learn something new, then provide an even larger and more complicated update. If we can make it as simple as possible for people to help contribute code, then we'd be in a much better position to achieve growth goals in the community.

That's my $0.02. :)

[hismightiness]

Side Note: If we do this correctly, it shouldn't require a series of very long videos to get someone started on helping contribute code. Ideally, a single video of less than 10 minutes should be all that's needed. Imagine being able to say something like, "No matter which repo you want to help with, you open it and work with it the same way. At this end of this short video, you'll know everything you need to know in order to create a pull request in any repo in mere minutes."

[ohine]

As related to issue management. We could easily also just use a real issue management solution outside of github. We are have already reached the limitations and require 3rd party software to delegate management responsibilities to a larger group of individuals.

[ohine]

I'd love to see how #1 works. But given that the admin UI "reflects" the core it feels weird that any change needs multiple issues/PRs and needs to have coordination to make sure all parts get updated at the same time for fear of having a broken build.

That's not completely correct. You can do quite a lot without requiring multiple pull requests. However, with more complex changes for new features which require a change to the data structure and APIs, then yes multiple pull requests are required.

The demo on how this should work is how CDF is configured today. What's currently missing is automating the deployment of nuget packages. This was something I'd scheduled to work on this past weekend but with this discussion coming up, and the target all of a sudden changing yet again in the middle of active development I had to put it on hold so it wasn't time wasted which would be thrown away.

[mitchelsellers]

I'd love to see how #1 works. But given that the admin UI "reflects" the core it feels weird that any change needs multiple issues/PRs and needs to have coordination to make sure all parts get updated at the same time for fear of having a broken build.

That's not completely correct. You can do quite a lot without requiring multiple pull requests. However, with more complex changes for new features which require a change to the data structure and APIs, then yes multiple pull requests are required.

The demo on how this should work is how CDF is configured today. What's currently missing is automating the deployment of nuget packages. This was something I'd scheduled to work on this past weekend but with this discussion coming up, and the target all of a sudden changing yet again in the middle of active development I had to put it on hold so it wasn't time wasted which would be thrown away.

I think hat this is the crux of the issue. Almost any change to the PB, is going to require two items as it is updating the database or adding new settings. (Granted, this is an over simplification as yes UI layout fixes, or otherwise do not require multiples.)

The key for me is that CDF = Provider, and PB > Provider.

The reasoning is in the behavior.

Providers = A item implemented as a provider can complete it task with minimal input and dependencies on the items that it is implemented in. (For example a caching provider can do whatever it needs to do internally, if it has custom configuration it can, and should, manage that internally to the configuration of the provider. Another example would be an authentication provider, you can implement it, add fields, and change it without changing the format or structure of the provider model it was implemented in.)

Personal Bar = Although it was implemented as a provider, and you can swap it out for a third-party element, it is more rigid in nature, as the settings/options/features actually require the shape of API’s to change at the platform.

It is for this reason, why I can understand how some of this works. Now, if we can fix PB to be self contained, that could be a workaround as well

[ohine]

Almost any change to the PB, is going to require two items as it is updating the database or adding new settings.

I disagree. The majority of pull requests that we've processed have been either just a change to PB or just a change to Platform. Very very few have required two.

[mitchelsellers]

Just remember we have processed st least 5 major feature adds that are 1/2 implemented.

[ohine]

Just remember we have processed st least 5 major feature adds that are 1/2 implemented.

This is a separate issue, those were merged due to the age of the pull request. Also, there was zero documentation at the time they were created which made submitting a change to the persona bar nearly impossible. While our current documentation isn't ideal, at least it's started the process. We need to continue to create better documentation and spend the time actually improving the overall developer experience.

For example, react.common currently requires over 3gb of space and nearly a million files on disk just to clone + build, and a good solid 90 minutes to build. This isn't anywhere close to an ideal experience. The time should be spent on cleaning up what we have, not just moving around the location of a very poor developer experience to solve one very small issue which can easily be solved with just creating good technical documentation. The same good technical documentation that will still be required to explain what's what under a new massive single project solution which is what this RFC is primarily suggesting. It's a completely wasted effort in my opinion.

[kurtwilbies]

I suggest using Material-UI. Think in the long term. You can use react-frame-component.

[hismightiness]

The DAL and UI should be developed independently. Sure, in some cases, updates will be required in both for new and complicated features. But any updates to the DAL should include a required Web API endpoint so that features such as the persona bar can take advantage of it. This means, even a minor update to a SProc should include the requisite Web API endpoint.

[mtrutledge]

I think with the structure of the separate projects it makes testing anything you make above the level of Platform very difficult. I think Solution #2 where we merge Admin Experience and the dependent components into the Platform project. We can always publish various packages to various locations as needed, but having everything in one are would make contributing and development easier. I can see the pros and cons on each side, but if our goal is to make development easier and contributing easier then we should consider merging the rest of the projects.

[sleupold]

Despite the structure of the source code repositories, we should move all API into DNN Platform library. There is no benefit of PB user management accessing the database directly. Same applies for export and import. Multiple data provider files are making it difficult to optimize data access to core elements. However, I am having no objections PB maintaining its own tables for its content and permission.

[OllyHodgson]

As a relative newbie to the platform I'm conflicted on this. On one hand, it'd be really nice to be able to clone, change and build. Any steps we can take to make that eaiser will help. In other words, don't stop work on Solution 1 just because "even if we get there that it isn't good enough". It will be better than it was.

On the other hand, we have to be able to admit DNN is a really big and complex product. Complex products can be quite complex to work on. A certain amount of the learning curve will _never_ go away unless we massively strip features out from the product. I imagine there isn't anyone here who really knows the entirety of DNN Platform (and that isn't a bad thing). I'm still on the journey. Let me tell you it's been one hell of a lot eaiser to understand than SharePoint. It could be easier but to be honest, for me the biggest gap right now is documentation - it's all so fragmented and/or unofficial but invaluable. I know people are working on that, though.

To my mind, it doesn't really matter how many components / repos / etc we have:

• As a developer, it's more important that there are tools and instructions which can make it eaiser for me to work on the thing.
• As someone raising issues - it doesn't really matter where they raise them now that we can move them between repos - as long as we're willing/able to put time in to triage them.
• As for the "big corporate" worries, I work there at the moment :smiley: It's usually possible to get these things to work, if devs there are willing to work with the relevant teams to sort it out. Things like yarn, npm and nuget are a fact of developer life these days, so even _really big_ corporate are willing to use them (even it it means hosting their own repositories). Where that can't happen, we probably need documentation to outline the manual steps required (we probably need it anyway).

In terms of what other projects do:

• Wordpress appears to have their core code and issue tracking on their own infrastructure, but their new editor interface (Gutenberg) is being developed on Github. As far as I can see, they're doing issue management for it there, too. That model seems familiar, eh?
• The majority of Firefox's code is handled on their own Hg infrastructure, but some parts are on Github. The codebase is split up into components, because it's a complex product. Most of their issue tracking is done on Bugzilla but some is on Github (and sometimes both).
• _All_ of the code and issue management for Moailla's next generation engine, Servo, appears to be on Github. That too has been split into components, each in their own repo. I'm sure this sometimes means they need multiple PRs for a single change but they seem to manage OK.

All in all I _think_ I'm with @hismightiness on this.

</stream-of-consciousness>

@ohine

This will also require JRE to be installed for any platform development since the PB currently still requires it for it's build tasks.

Raise an issue, let's look at fixing it :smile:

[david-poindexter]

Dnn.React.Common was intended to be a stand-alone library of reusable, mostly ReactJS, components. There is a public registry that can be configured with one's local npm to install these components by packagename. Merging that repository with any of the other DNN repositories makes no sense to me.

Currently, Dnn.AdminExperience utilizes those components for its various extensions. No problem. It can continue to do so. Building one or the other is not dependent on the repositories being merged. Therefore, I have no issue with Dnn.React.Common staying stand-alone.

However, the same is not true in the context of comparing Dnn.Platform and Dnn.AdminExperience. These currently have A LOT of overlap and interdependencies. Therefore, I would be a fan of merging these two repositories OR documenting a very clear and concise process for making contributions that require changes to both. Right now it is way too convoluted and there is no clear-cut and defined way for making such contributions and managing/processing them. Major contributions are being held up for way too long due to these complexities and broken process. And other desired major contributions are not even getting off the ground due to these complexities and unknowns.

[ohine]

@nvisionative The documentation is currently being worked on. I've shared it with the CI and approvers team, the process is ongoing active development.

If you'd like to speed up the process, feel free to jump in and contribute. There is a ton of stabilization and cleanup we've been doing which has to be done BEFORE we can get to the complex long term changes. Everyone wants to run before we learn how to walk. As a volunteer effort, time is limited.

I will also say, you are making the case to merge one and ignore the other outstanding issues. There are also interdependencies on react.common and admin experience. Same with CDF and Platform. Less with CK and platform but they still exist. If we are going to merge things, we shouldn't do 10% of it. We should really plan it out and fix the problem instead of just throwing it over the fence for next month.

[david-poindexter]

@nvisionative The documentation is currently being worked on. I've shared it with the CI and approvers team, the process is ongoing active development.

Cool - I had no idea.

If you'd like to speed up the process, feel free to jump in and contribute. There is a ton of stabilization and cleanup we've been doing which has to be done BEFORE we can get to the complex long term changes. Everyone wants to run before we learn how to walk. As a volunteer effort, time is limited.

Sorry, I am not sure exactly what you are talking about here. I have been contributing as a volunteer. If you need/want something specific, please just let me know. I am not trying to _"run before we learn how to walk"_ - I am simply contributing my thoughts to the conversation. They can be taken into consideration or not.

I will also say, you are making the case to merge one and ignore the other outstanding issues. There are also interdependencies on react.common and admin experience. Same with CDF and Platform. Less with CK and platform but they still exist. If we are going to merge things, we shouldn't do 10% of it. We should really plan it out and fix the problem instead of just throwing it over the fence for next month.

I am not making a case to _"ignore the other outstanding issues"_. I have no clue how you got that from my comments. I am happy to learn things I don't know or understand. Again, I am simply sharing my thoughts.

[mitchelsellers]

Based on conversations that have happened outside of this RFC discussion, it appears, that for the immediate term, to be CLEAR "Right Now" we need to continue with a few items that are in progress.

We also need to work on a method to be better communicative of the work that is being done, as that is adding to the confusion.

At this point in time, the following initiatives & actions are either "In Progress" or are getting ready to be slated for work.

• Restructuring of the build process or project structure for React.Common to use a centralized node_packages directory to reduce the size & time necessary for building this library
• Restructuring of the build process or project structure for AdminExperience to reduce the size & time needed to build
• Restructuring of the Admin Experience build process to remove the Java dependency
• Additions to the Pull Request process to create NuGet packages for PB, React.Common, and DNN.Platform at each Pull Request, this will allow builds during development to consume these assets which would then allow testing of downstream changes. (For example the PB would consume a DNN.Platform NuGet package from a PR that added new API's for a feature change to the PB)
• Creation of additional Cake scripts to help developer experience
  
  
  
  • A script to prevent the deletion of .cs files on DNN.Platform to allow running in the folder structure
  
  
  • A script to properly locate/include other assets as needed to test PB, or other changes
  
  

These elements are the "cleanup" tasks discussed above. And sadly, have not been documented as issues. I believe that this captures the essence of all of the items that have been discussed in other channels. This work, with the exception of the Cake scripts, would need to be done in almost any structure discussion.

At this point, I would vote that we take the following actions.

1. For the developers working on fixes for the above items, quickly create an Issue to note the work and who is doing it for clarity and transparency. The same standard we are requesting from the greater community.
2. For the items listed above and any I might have missed that do not yet have a developer assigned, create the issue with a "Help Wanted" tag. This will provide to the community a listing of the items that need help, specifically around improving the developer experience.
3. Once all of the above solutions are implemented, we can re-visit a larger restructuring or condensation of the projects

[valadas]

This will also require JRE to be installed for any platform development since the PB currently still requires it for it's build tasks.

I guess java usage will be more of a challenge in big enterprise than node/npm nuget, etc.

[mitchelsellers]

To bring this issue around full-circle. Work is ongoing with regards to repository structure, build environment, and otherwise.

Steps will continue to be taken to improve the experience, however, major changes that are to be considered must be submitted with a Pull Request that would address the ENTIRE solution, including any changes necessary to the build environment within Azure DevOps. Additionally, any of these changes should be coordinated with the build support team (Oliver, Daniel A, Myself) to ensure that no other steps are impacted.

Additionally, the current processes will be documented in GitHub for clarity as soon as they are stabilized post changes that are being made as part of 9.4.0 to restructure the Admin Experience REPO and move React.Common into that repository.