Magit: Teach Emacs' backup, auto-save, and undo features to use Git

[tarsius]

Magit already provides a backup mechanism, but that is in addition to the backup mechanisms offered by Emacs. Also it is only used inside Git repositories, only for files that are tracked by Git, and only works after creating the first commit.

But this implementation can be used as the basis for the feature I am suggesting here. The basic idea here is to use Git repositories as a "back-end" for Emacs' backup and auto-save features. (And maybe later even for the undo system, but I know to little about that at this point to know whether that is feasible.) Emacs' would continue to be the one who decides when to backup and auto-save, but it would optionally do so by calling a "Magit" function.

This has to be completely optional; the Emacs maintainers would never agree to depend on Git for backups. And rightfully so. And Magit isn't part of Emacs, so depending on Magit isn't even possible.

I will probably implement this feature separately from Magit, so that it can be added to Emacs before Magit is. That's the plan anyway; split out existing parts of Magit into reusable parts, and create new parts as separate libraries/packages from the start. These libraries/packages can then be used by Emacs and third-party packages, without having to depend on all of Magit (see #2956).

But in an initial step I will depend on low-level Magit functions. I also don't intend to get the blessing of the Emacs developers up-front. Instead I will implement a proof-of-concepts and if the idea is eventually rejected, then I am fine with that too. Should that happen, then things will remain inside Magit and the feature will be hooked into Emacs using advises and re-definitions, or maybe a compromise will be for Emacs to provide hooks.

In an initial phase I will probably copy the relevant libraries into a separate repository and patch them there, or create a branch in a fork of the Emacs repository.

---

The current wip-save implementation in Magit basically works as fallows:

(Please also read the documentation: Wip Modes.)

1. There are low-level functions for saving all tracked files as found in the working tree, or the staging area, to a dedicated wip/backup ref (e.g. refs/wip/wtree/refs/heads/master and refs/wip/index/refs/heads/master when the current branch is master).

   • Such refs are "hidden" by default because they are not branches or tags. But by using --all they can very easily be shown in Magit or Git logs.

   • These functions can either save all tracked files or only certain files passed as arguments. The also take a message as argument, which is saved in the reflog of the wip ref.

2. The various magit-wip* modes use these functions to create backups at certain well defined times.

   • magit-wip-after-save-mode creates a backup when saving a file-visiting buffer; this is somewhat similar to Emacs' auto-save features. But unlike Emacs auto-save it only saves when the user saves explicitly.

   It wouldn't be to hard to implement an additional mode though which periodically auto-saves unsaved changes. Such a mode would be an early poc, but the idea is to not add an additional mode and to instead hook into Emacs' existing auto-save feature.

   • magit-wip-before-change-mode is similar to Emacs' backup feature, but it is not equivalent. Emacs only creates a backup when first opening a file, while Magit does before every Magit command that may make a change. However it doesn't do so for non-Magit commands and if the first command that makes a change to the unsaved work is such a non-Magit command, then Magit would not have had a change backed up the initial version.

   So Emacs "initial backup" feature is still useful (it should just use Magit as the backend).

   • magit-wip-after-apply-mode saves after a Magit command did stage/unstage/discard/reverse/apply a change. This is particularly useful to recover after messing up a painstakingly grafted index. Emacs doesn't, and doesn't need to, have anything similar (though VC probably should).

   • Currently these modes are not enabled by default due to performance concerns. I am actually not sure how likely it is that they will cause noticeable delays, and just played it save. (I expect that the after-save variant never lead to delays, but am not so sure about the before/after-apply variants.) I do have all these modes turned on and never notice any delays, but this will have to be carefully benchmarked and if necessary optimized before being used as a back-end of Emacs' default backup and auto-save features.

3. Magit provides little specialized functionality to view wip refs and none to recover lost data from those refs. But to some extend that's the beauty - these things are not actually necessary. You get a better interface to inspect and recover old versions for free: Git and all kinds of Git interfaces, including but not limited to Magit.

   That being said, it cannot hurt to provide some additional tools.

   • magit-wip-log-current and magit-wip-log show the log for a certain branch as well as its wip refs.

   • Recovering a particular version can be done by doing a reset (x) to a commit on a wip ref. You can also only apply individual hunks. The important thing no notice is that this works exactly the same as in regular, non-disaster-recovery work with Git/Magit.

   • And really that's why I think using Git as backend is the way to go. From the perspective of the code initiating the backup there isn't much of a difference, but when recovering lost data you can use the same interfaces you already use on daily basis to do productive work. Packages such as backup-walker become unnecessary and we get a better interface without even having to implement it (it already exists).

   • Restoring an old index while keeping the working tree intact would be much harder though. For that we could really use some additional tools.

---

I would like to use Git as the backup/auto-save backend for all files.

• Not just tracked files - that's a somewhat artificial limitation of the current wip-mode implementation, which shouldn't be to hard to overcome.

• More importantly though I also want to do so for files that are not located inside a Git repository. For such files a special "global backup Git repository" could be used. This does however lead to some complications that have to be carefully dealt with:

  • Permissions. There probably have to be separate global repositories using different permissions.

  • Sensitive information like passwords could end up being preserved in history for a long time. Emacs' current auto-save mechanism has the same issue, but there it is at least fairly easy to purge such information. With a git-based backend we need some command to remove and garbage collect certain revisions known to contain sensitive data. Or at least this risk has to be communicated to users.

[tarsius]

Also see #2984.

[mlb-]

An emacs plugin that I use for more Git-like management of my undo states is undo-tree: http://melpa.org/#/undo-tree
Perhaps this will provide a useful reference for this issue.

[tarsius]

Regarding backup.

I have added a new function

- Function: magit-wip-commit-initial-backup

  Adding this function to ~before-save-hook~ causes the current version
  of the file to be committed to the worktree wip ref before the
  modifications in the buffer are saved.  It backs up the same version
  of the file as ~backup-buffer~ would but, instead of using a backup
  file as ~backup-buffer~ would, it uses the same worktree wip ref as
  used by the various Magit Wip modes.  Like ~backup-buffer~, it only
  does this once; unless you kill the buffer and visit the file again
  only one backup will be created per Emacs session.

  This function ignores the variables that affect ~backup-buffer~ and
  can be used along-side that function, which is recommended because
  this function only backs up files that are tracked in a Git
  repository.

I don't (any longer?) intend to replace the backup functionality provided by Emacs. It serves a very specific purpose; backing up the version of the file as it was before the user started modifying it in the current Emacs session.

The VC package by default disables this backup for files that are under version-control, which I think is probably a mistake (luckily this can be undone using (setq vc-make-backup-files t)). Just because a file is under version-control that does not imply that every change to it is always immediately checked in.

When using the Magit Wip modes, then it is, but only if the changes was made using Emacs.

After adding

(add-hook 'before-save-hook 'magit-wip-commit-initial-backup)

and ensuring that magit-wip.el is autoloaded, Magit now covers what make-backup does. But only for tracked files -- so there is one thing left to be done: make it work for any file.

But I would recommend to keep using Emacs backup mechanism, even once Magit does this for all files, and even for tracked files (which requires (setq vc-make-backup-files t)).

[tarsius]

Regarding auto-save.

It is not possible to teach the auto-save mechanism to rely on Magit without modifying do-auto-save, which is implemented in C.

I also don't think there is much to be gained from doing so.

Emacs periodically saves all files that you are visiting; this is called
“auto-saving”. Auto-saving prevents you from losing more than a limited
amount of work if the system crashes. By default, auto-saves happen
every 300 keystrokes, or after around 30 seconds of idle time. *Note
Auto Save: (emacs)Auto Save, for information on auto-save for users.
Here we describe the functions used to implement auto-saving and the
variables that control them.

So this is intended to protect users from losing work if something outside their control goes wrong. The backup mechanism (see above) serves another purpose: it ensures it is possible to recover the lost data when the user does not realize that there are "uncommitted" changes and then goes on to accidentally destroy them by making other changes.

The latter is very similar to what the magit wip modes do (except it only covers one specific scenario, while the wip modes should cover them all). So it made sense to teach the wip modes about that one case that they did not cover but which backup does cover.

It would make less sense to do so for auto-save. Auto-saving happens at unpredictable times - it has to be very fast else it would be very annoying. I don't think committing is fast enough, though it could probably be made faster.

I have decided to not teach auto-save to use git commits. I have not decided whether to teach the undo mechanism to create git commits though. If I end up doing that, then it would be redundant to teach auto-save to do the same.

I am still considering to implement one thing related to auto-save: a redefinition of recover-file, that shows a diff showing the difference between the actual file and the auto-save file. Currently one has to restore blindly and that could actually result in the lose of uncommitted changes.

[tarsius]

To summarize:

1. I taught the wip modes to do the same thing as backup.
2. Magit cannot and should not hook into auto-save.
3. It would be nice if recover-file presented a diff to the user but that is beyond the scope of Magit. I plan to eventually implement that but would probably use diff-mode not Magit diffs to do so.
4. Undo is to complex to hook into. undo-tree works nicely for me.
5. Instead of the original plan Magit's wip support should be improved in its own right. It should be faster, more reliable and easier to use, and there should be better tools to inspect the history. I have made progress on that front, but tooling in particular still has to be improved.