Magit: Implement competitive collaboration tools

Created on 25 Jan 2017 · 13Comments · Source: magit/magit

Features that facilitate collaboration are one of Magit's weak spots. That is somewhat embarrassing, considering that this is were many other Git tools shine.

The primary reason for this is that I at some point decided to delay implementing such features because I already had enough on my hands and had to draw the line somewhere. It was easy to draw the line here because Magit didn't offer much here to begin with, and because Magit is primarily an interface to Git and Github et al. are not Git. I also put a lot of hope into some of the packages that set out to fill the gap.

But now the time has come for fully integrated collaboration tools. Here's a quick run-down on what I intend to do.

Support multiple back-ends/work-flows

The following two work-flows are currently in widespread use. I won't go into the pros and cons of either. Whatever your preference is, you may be forced to use the other approach to contribute to a particular project.

Web-based and centralized solutions, primarily "git forges" such as Github, Gitlab, and Bitbucket.
Work-flows centered around a mailing-list.

A third alternative does exist, but is not in wide-spread use.

Distributed code review systems.

The idea behind tools like git-appraise is to store code reviews, and possibly issues as well, inside the repository as git objects. I find that idea very appealing, but I think that for most projects this approach is not feasible, mainly because it requires contributors to get familiar with yet another new work-flow and a new tool before even being able to contribute.

What will Magit support?

So why am I mentioning those three approaches here? And which is Magit going to support? All three!

The title of the preceding section was "Support multiple back-ends/work-flows" and you might have gotten the impression that I was unsure which term was more appropriate and so just used both. But the reason I did it is that I am under the impression that, in this context, most users believe that a certain back-end dictates a certain work-flow, and that's an assumption that I would like Magit to challenge.

The back-ends

As a back-end, a git forge store information about pull-requests and provide a mechanism for transferring them using an api. The pull-request branch and its commits are commonly stored and transferred using git.
A mailing-list is not much more than a transfer mechanism. Everything beyond that are conventions, which may differ between projects. Which means that it is not possible to e.g. simply request a list of pull-requests; instead they have to be somehow identified and then extracted from maildir (or another email format). That may fail and also requires some configuration.
Like forges a distributed code review system (DCRS) stores and transfers pull-request branches using git. Unlike forges it does the same for additional information about pull-requests, such as conversations about them and metadata such as labels.

The work-flows

When using a mailing-lists or a DCRS one basically does everything locally.

When using a DCRS the work-flow is, to some extend, defined by that tool. The drawback is that the tool is required. Unlike for forges there is no simple alternative interface that covers at least the most basic tasks. This unfortunately makes DCRS unsuitable for most projects.
When a projects uses a mailing-list not only for talking about issues but also for submitting patches, then the basic interface essentially consists of the git commands am, imap-send, send-mail, and request-pull.

Those commands are quite appropriate, and I think the primary issue many users are having with them, are that they require some initial setup and that, unless one uses them on a daily basis, using them may very well involve consulting the man pages.
Finally forges. They provide very well defined and simple to use work-flows. But they also enforce them. If you want to do something more complicated or something that just isn't supported, then you have to do so locally. The manual steps required to do so can be quite involved.

Back-end and transfer mechanism planned for Magit

Like with a DCRS each repository will contain all the required data. However there will be multiple ways for it to get there, depending on the back-end.
- The first transfer mechanism to be implemented will be to import it from a git forge by using its API. Conceptually this will be the same as git fetch and the result is that local git objects are updated, but git isn't used to do it (except for the fetching the actual pull-request branches).
- The second transfer mechanism will be to actually git fetch from the canonical repository. Later it should also be possible to do without a canonical repository, at which point we will have implemented a full DCRS.
- For project that use a mailing list, it will be possible to manually create a pull-request branch for a particular series of patches found in a mail thread, or to import all pull-request threads at once. So the "fetch" step will be implemented by importing from a local maildir (or other email format) into git.
This will do a bit more than git branch ...; git am .... We also need to save some additional information that will teach e.g. a subsequent magit-push-current-to-pushremote to "push" by sending a new version to the mailing-list.

To take part in the conversation about an issue or a pull-request you will continue to have to use your email client. But basic information about issues and pull-requests will still be imported into git. This will e.g. allow displaying a list of open pull-requests in Magit and you will be able to initiate the process of replying on a pr/issue from Magit. But you will use your email program to actually write and send your reply.
After making local changes, they have to be "pushed". When using the DCRS approach this is literally done by pushing, when using a forge, then some of the information is made accessible to others by using the forge's api.

However, even though this is redundant in most cases data should also be literally pushed when not using the DCRS approach, for the following reasons:
- For projects that use a mailing-list, this is the only way to share metadata, such as issue labels, with others. This data will only be available to users of Magit, not all users of the mailing list (and at least initially it will only be displayed inside Magit but not your email program), but this might still be useful.
- When using a forge, then that back-end might not support storing all the data we have to store. For example, I think it is highly desirable when doing code review to keep old versions of the pull-request branch as well as the associated conversations easily accessible. Unfortunately at least Github doesn't really do that.
- As a backup mechanism.

This leaves a lot of details unanswered. I will study existing DCRS tools to learn from them. However I do not plan to use one of them. I think that would not be appropriate because I don't add a pure DCRS to Magit. Instead Magit will contain a versioned code review system. Eventually one of the supported back-ends will be a DCRS, another will be a versioned but centralized code review system, and the one that will be implemented first and end up being used the most will be the one that uses a forge as back-end.

Wrapping e.g. git-appraise would only make that harder, since it only tries to be a pure DCRS. However learning from it is very important, and we should try to make it compatible by using the same ref names and such.

The (mostly) shared interface

So far I have said little about the user interface that Magit will provide. I do have mentioned that I would like the interface to be mostly the same regardless of back-end, and that mailing-lists pose some challenges in this regard.

Instead of, or at least before, giving too many details here (which may change), I would like to emphasize one important point: newly added features will be tightly integrated into existing functionality, to the point of them becoming not much more than special cases of features that already exist.

As an example, consider pull-request branches. Magit already supports setting up both the push-remote and upstream when you checkout a branch created by someone else. But only if that branch existed in the origin, e.g. when it was created by someone with commit access.

In the future it will do the same for pull requests. You will be able to see logs of unpushed and unpulled commits right in the status buffer, or push after you made some changes using p p. It will work exactly as for other branches, even though pushing, for example, might actually involve sending emails.
Other features are so obvious that you can just expect them to be implemented. Of course there will be lists of open issues, and the will be implemented using "section inserters" which will allow putting them directly into the status buffer or some dedicated buffer, which ever you prefer. Standard Magit stuff.

If you ever noticed that in a list of submodules you can expand an individual submodule to list the "unpulled" commits in that repository, then you are right to assume that you will be able to do the same for pull-request branches.

And if you ever pressed RET on a list of submodules and were taken to a tabulated-list-mode buffer with additional details, then yes, the same will happen for the list of pull-requests.

Originally this post consisted of the introduction at the top and the following list. I am keeping this list for the time being because most things haven't made it into the above description yet.

Support both web-service based, as well as mailing-list based work-flows.
- Focus on web-services, or more specifically Github first.
- Which is why I am mostly talking about Github below - the competitors are implied.
- Feature parity is a long time goal.
Store all data locally, most of it using Git.
- Data is only synced periodically with the "remote", just like git fetch.
- Pull requests are stored as branches.
- All branch commands are automatically available for pr's. No effort, major win.
- Set the pr's target as upstream.
- Set the pr's source as push target (using branch.pushRemote).
  - This allows the upstream to push to the repository of the contributor.
  - Provided the latter has not disabled this Github feature.
  - This is very useful because it allows upstream to easily fix a trivial error in an otherwise perfect pr, without having to either wait for the contributor to do it, ignore the issue and merge anyway, push the amended commits to the target and close the pr with a comment "it says it was closed, but actually it was merged (with minor modifications)".
  - Because of that the name of the branch is fixed.
  - If it is bad, then that's just what it is. You can still "change" it at the time of merge.
  - Name conflicts are worse, master being the most likely one. Conflicts will likely cause some lose of functionality, more specifically there won't be a branch.pushRemote`
  - This requires adding a remote. But we don't have to fetch all the other irrelevant branches too if we setup remote.fetch accordingly.
- Automatically add issue number to merge commits.
- Like Github does, but give some control over formatting.
- Since we already use the contributor's branch name, we have to store the number somewhere else, e.g. in branch.<name>.pr.
- Preserve previous pr versions.
- Probably store past versions outside refs/heads/, refs/pr/<pr>/<version> maybe?
- The Github UI does not make previous versions of a pr easily accessible :-(
- Unfortunately this also appears to apply to the API :-[
  - Hopefully I am overlooking something (pulls,
    
    issues, timeline).
- If not, then we have to use the pull_request webhook and will have to suffer.
  - We have to provide a "fix github for magit" web-service.
  - and might want to make it available to others too.
  - and make it available for self-hosting too, of course.
  - Repo owners will have to enable the hook explicitly.
  - We cannot assume they do.
  - The collected information has to be written back to Github.
  - Can we use refs outside of refs/heads/ without Github zapping them?
Import Github comments as Git notes.
Create, view, and edit issues and pr.
- To create a new issue/pr, edit or add a comment, use a mode similar to git-commit-mode, the major mode would be markdown-mode obviously.
- Present an existing issue/pr similar to what Emacs mail clients do, one window listing the individual comments, another showing the selected comment.
- The list window will likely use tabulated-list-mode.
- The list window should allow switching between comments, commits, pr-versions.
- Maybe also provide a list that is more faithful to what Github considers an event and the order in which they occurred.
- I am not convinced of this approach yet, but it is a good initial approach.
List issues and prs
- Simple pr/issue list sections.
- Detailed pr/issue list buffers, likely using tabulated-list-mode.
Some other features:
- Visit pr/issue in browser.
- Automatically setup the bug-reference package. #2949
- Insert an issue numbers with completion.
- While completing, show the corresponding issue summaries.
- Optionally, automatically delete pr after merge. #2922
- Fork current repository on github and add that as a remote.
- List your repositories on Github.
- Show the local clones, if any.
- Default to repo at point when cloning from here.
- Teach clone and other remote related commands about user's Github repositories.
- And those of his/her organizations too.
- name or user/name can be used to specify remote instead of url.
- Optionally always add both the upstream as well as the personal fork as remotes.
  - Even though just one was specified explicitly. Or...
  - After cloning upstream, offer to add already existing personal fork.
  - After cloning personal fork, offer to also add the upstream.
  - And fix name of the personal remote.
  - Or always default to use something more suitable than origin for a fork.
- Adding a remote that doesn't exist yet on Github, optionally creates it there.

If something essential is missing from the above list, then fear not. It isn't complete. I do intend to make it possible to do everything in Emacs that can be done in the browser. Just more comfortably and integrated with the tool you already use to create and store the changes ;-) But yes, it will take a while until we get there.

Also I won't implementing as many features as quickly as possible but concentrate on getting the core infrastructure right. A solid, easy to configure, authentication method will come first. Some of the features mentioned above are already implemented by some existing package. Checkout this list. I would also like to encourage the authors of these packages to continue their work. Some competition won't hurt. Of course you are also welcome to "rebase" your package onto Magit, once I am done with the ground work. Either as part of Magit or an extension. Or not at all, that is perfectly fine too.

However there is one thing that I hope we can all agree on after some discussion: how we store credentials, see #2973.

feature request progress

Source

tarsius

👍28 ❤17 🎉12

Most helpful comment

I'm really excited abut this one! Being able to do code review in emacs will be a massive improvement to my workflow.

shonfeder on 2 Sep 2017

👍7

All 13 comments

You do not mention Gerrit above, not sure whether that is a gitforge or a DCRS or a mix of both.
Some projects use patchwork (http://jk.ozlabs.org/projects/patchwork/) as a complement to mailing lists.

It would be good if the design is aware of these tools so it is possible to write backends for them.

snogge on 24 Feb 2017

Some other relevant tools and discussions:

Some only marginally related tools and discussions:

GitTorrent (hn)

Some ideas that have come up along the way:

Consider implementing a "put me on the right branch for this issue" command. That command would either checkout an existing local branch, create and checkout a local branch from a remote branch, or create and checkout a new local branch. See https://github.com/magit/magit/pull/3247#issuecomment-342934247.

tarsius on 26 Feb 2017

After reading more about these tools, you might be wondering, why not just adopt one of them? Well, I might, but what I am currently envisioning is both less and more portable.

It would be limited to Magit users, since it isn't just a wrapper around some tool that can also be used by itself.

At the same time nobody would actually have to use it, non-Emacs users could continue to use the webinterface provided by Github/Bitbucket/... and would never have to know that you are using something else. (Except if a project goes with the fully distributed back-end.)

A major benefit of that approach would be that you could use it with any project, even when project actually uses some web- or mail-based approach. org-sync appears to follow that approach too, so I might adapt that.

tarsius on 26 Feb 2017

👍3

I'm really excited abut this one! Being able to do code review in emacs will be a massive improvement to my workflow.

shonfeder on 2 Sep 2017

👍7

I once tried to hack a pull request commenting interface onto https://github.com/sigma/magit-gh-pulls, so I'd love to be able to do code review within Emacs! It'd be much faster with big PRs which would be awesome.

ibrahima on 26 Sep 2017

this might not be the right issue for this suggestion, but here goes.

motivation:
https://twitter.com/devonzuegel/status/946490569374625792

proposal:
M-x magit-url-for-current-location constructs a URL to the current repo/branch/file/line in the back-end web service, typically Github.

By default it saves the URL to the kill ring and places it in the system copy buffer. With a C-u argument, it calls browse-url upon the URL as well.

mengwong on 28 Dec 2017

For that feature, https://github.com/rmuslimov/browse-at-remote works really well in my experience.

Wilfred on 28 Dec 2017

👍3

FYI, I've been using https://github.com/sshaw/git-link, which basically does the same.

To1ne on 29 Dec 2017

Also on that list is: https://github.com/osener/github-browse-file, nice to know there are several!

dgtized on 31 Mar 2018

Found an interesting article today via reddit that may provide some about of inspiration: https://blog.janestreet.com/putting-the-i-back-in-ide-towards-a-github-explorer/

vermiculus on 31 Mar 2018

👍1

And this is my response on that reddit thread: https://www.reddit.com/r/emacs/comments/88b51s/putting_the_i_back_in_ide_jane_sts_emacsbased/dwkwcg9/.

tarsius on 31 Mar 2018

❤1

And since I am in the mood for creating quick drafts I have also created one with basic browse-url support: https://github.com/magit/magit/pull/3408.

tarsius on 31 Mar 2018

👍1

I feel like for completeness someone should mention here your forge package, which is at least a start along these lines...