C-toxcore: OVERHEAD development process.

[GrayHatter]

TL;DR, toxcore should be writing more code for our major issues/features faster.

With respect to major end goals only In talking with core, and client, developers/admins. I think the pace of development on toxcore should be much faster than it currently is. I'd like to propose a new format. We have enough developers, who care about the project as a whole more than individual features, that if were were to focus all of this time/work, into a unified issue/feature Toxcore would be moving at a runners pace, instead of a snails pace.

The problem

With our current weekly release cycle, the majority of the time spent by code developers is on the low hanging fruit that'll easily be ready within the release cycle window. This is causing the chilling effect of projects better suited to >1 developer writing code go neglected. (These would be the important ones.)

The Solution

1. Toxcore should stop shipping versions on a consistent release cycle. Instead, toxcore should be shipping features. These two aren't mutually exclusive, there's no reason patches and fixes shouldn't be merged when ready, but the overhead of drafting weekly releases as a priority should be reduced to, as needed, with the primary focus on getting an important feature or a bugfix ready as quickly as possible.
2. Toxcore needs to pick it's next feature/issue of a defined magnitude/scope, and all of the attention of the main developers should be on writing code for that feature and/or section. This means that [developer] needs to be in charge of selecting an outstanding issue of a reasonable scope with a definitive solution, and easily completable tasks (>= number of developers), then assign tasks to any developer. (Note: The review requirements toxcore uses shouldn't change, this means we need to ensure that while any number of developers may work on a file or function, we need to have enough to make sure that the code is reviewed by someone who hasn't directly touched the code to be reviewed.) The intent of keeping focus on a feature being, that the vast majority of the work done by primary contributors should be on THIS feature/issue, aiming for completing it correctly in rapid cycle.
3. Where applicable, these features should maintain the important goal of a provable (meaning well tested) system. Every logic or functional change we make should include a non-fragile test that proves the final changes match the goal. [Optionally, the goal should not be; "avoid break existing implementation". But instead should be "Complete, and test the implementation that offers the best solution"]
4. [Optional] We need to implement a staging branch for these features on the main toktok/c-toxcore repo and encourage anyone who want's to help with the functions to push quality commits the branch so that iteration can happen quickly, with support for other developers.

None of these suggestions are set in stone, the above are just breadth first suggestion that I hope will get toxcore working together on the important features. All comments, suggestions, and criticisms welcome.

[GrayHatter]

@nurupo @tux3 @cleverca22 @dvor @piling I wanted to assign you as well, but github says no issue will ever need more than 10 people... they don't know how much I like to spam people :D

[Zer0-One]

I agree, and I think that toxcore needs some kind of Project Manager-type
figure to help implement this. A PM would not be in charge of reviewing
code, but would handle all of the overhead/process stuff that you're concerned
about. I'm willing to handle that (if you want me to), and I think I have a
reasonable outline for the job:

• Outstanding issues (including feature requests) need to be prioritized.
• Based on issue priority, tasks need to be delegated to the appropriate devs.
• Work commences; when development lags, the PM needs to prod people to figure
  out what's blocking progress, and to react accordingly by re-assigning
  tasks, collecting information, delegating new tasks, etc.
• When the work on a task is complete, the PM should assign a "final review"
  according to whatever review policy we agree upon.
  
  
  
  • A "final review" should probably be an in-depth review, with every
    
    line checked for style, logic errors, etc.
  
  

The bottom-line is to remove the cognitive effort involved in management tasks
from developers who should be spending it on writing code instead.

I'm sure there are probably some more things that would improve this outline.
Any comments, suggestions, submissions, deletions, omissions, death threats, etc.?

[iphydf]

I would suggest we implement this workflow on a separate repository, e.g. uTox/c-toxcore. It is well suited for feature development, which is currently still very hard to do with confidence on TokTok/c-toxcore.

The current workflow was designed to facilitate stabilisation and improvement of the code base, which I have always considered a prerequisite to feature development. However, as I suggested in my initial project plan, there should be a feature development fork (at the time I designated grayhatter/toxcore) in which there is more freedom for experimentation. That is useful for trying new ideas and integrating different ideas into a single branch while still allowing the TokTok/c-toxcore (at the time /toxcore) to keep its slow pace.

TL;DR: Slow development pace and many small iterative releases on TokTok/c-toxcore is by design (note that the concrete steps of the original plan are outdated, but the process still seems appropriate - we're sort of in phase 2).

[alexeysvrv]

I agree for the most part. I would like to see more fix various important bugs, adding functionality. It is bad that the problems accumulate and resolve issues minimum priority. It is necessary to assign each issue a priority - high, medium, low. You need to deal with a complex problem that to her not to return. Offered once a month about to release a release with significant bug fixes and enhancements, the development lead in the dev branch.

[Ansa89]

Instead of using another repo, why not using another branch like "TokTok/c-toxcore:next"?
That way "master" would be for fixing bugs and "next" for adding features; then we could adopt a linux_kernel-like development cycle:

• inside "next" will be merged PRs labelled for next milestone
• run our quality checks on "next"
• merge "next" inside "master" when we are ready
• at any time we can release a "bugfix only" version merging high priority PRs inside "master"

[iphydf]

It's effectively quite similar. Reasons for not adding branches in TokTok/c-toxcore:

• I would like grayhatter to manage experimental feature development completely autonomously. I need not have a hand in that.
• It will not just be one branch. Even with the noble goal of "everybody works on a single feature", you will end up having multiple features being worked on at the same time, because serialisation like that simply doesn't work in a small geographically diverse open source project.
• A repository is an organisational unit. You can have different processes (e.g. a different review workflow) per repo. It's not obvious to me whether a rapid feature development branch should have the same workflows as a stable "slowly clean up and gather knowledge" branch.

There are a few other reasons, but these are the main ones.

[sudden6]

I'm in favor of @iphydf's idea of making a separate repository for rapid feature development with less strict rules. I think a branch won't be enough to setup all the different tools we use to different policies. This repo then could have various branches for everything or maintain the feature code in PRs.

As for the features, I know there's code floating around for the new groupchat's and some other stuff, but I can't really find a place with a list of what's WIP and where specs or ideas exist. One can gain that knowledge only by listening on the IRC or actively looking for specific stuff. I think a central place where ideas/WIP stuff can be discussed outside the main repo will help, because I'm certainly not the only one always following the IRC closely.

[GrayHatter]

Anyone want to propose a solution for when the protocols between the two become incompatible?

[jin-eld]

Has anyone considered using scrum? Not sure if this can be done properly in a community open source project, but I had very positive experiences with the process as such.

As for the weekly release cycle, I agree with @GrayHatter, I know the "release early, release often" idea, but "too often" can actually slow down development.

And that's where scrum comes in again - it goes in "sprints" where the developers commit to certain work/features which can be finished within a sprint. So in the end you ideally get some finished work blocks instead of "lots of random tasks all over the place", you can then still decide if you release at the end of a sprint or if your releases consist of several sprints and so on. Just throwing in an idea...

[iphydf]

I think weekly sprints can keep us focussed and give us very concrete goals we can easily plan for. Given that it is a spare time project, it's acceptable that we skip a release sometimes if we haven't had time to produce anything interesting that week.

What I did so far was assigning issues to milestones as a form of sprints. Creating a release has low overhead (a few person-minutes per week) but allows earlier identification of issues. It sets clear points in time, and if we find an issue with a particular release, we can mark exactly that release as bad and don't need to mark a large range of commits as bad.

The change I've made now is that instead of defining sprints ahead of time, I've only defined the current one and the plan is to sit together and decide on what we'll do for the next sprint after building the release for the current one. That would happen tomorrow.

Making very long sprints is not a good thing to do, in my opinion, because it's harder to focus on larger tasks. I think our velocity is higher if we split up a task into manageable pieces that can be done in a week's time.

[cebe]

You should play with https://github.com/TokTok/c-toxcore/projects a bit, maybe that is good to organize a workflow.

[SkyzohKey]

@cebe In progress. :)