Notebook: Deleting a notebook deletes the checkpoint?

[dsblank]

Just had a student accidentally delete her notebook, and was surprised to find that the system (jupyterhub 3.2) removed the file stored in ~/.ipynb_checkpoints/

Is that by design? Should leave the checkpoints alone, I would think.

(It would be fine with me if every time it made a checkpoint, it would make a copy of the file at that point, date-stamped. Otherwise, "checkpoint" is the wrong word to use. UI for restoring from list could come later.)

[Carreau]

Yes it is on purpose. The API/UI does support multiple checkpoints if the content manager implement one.
The default content manager does by design support only 1 checkpoint, as we do not want to reimplement a VCS, and we do not want to require git, mercurial or a DB to store notebooks/

[dsblank]

Yes it is on purpose.

Please consider this issue to be a request to not delete the checkpoint when deleting the file. I now have to involve the sysadmins whenever a student deletes a notebook by mistake, and it takes some time for them to restore from tape, or wherever.

The API/UI does support multiple checkpoints if the content manager implement one. The default content manager does by design support only 1 checkpoint, as we do not want to reimplement a VCS, and we do not want to require git, mercurial or a DB to store notebooks/

One could have a multiple checkpoint scheme without much complexity and without having dependencies on anything else. Just keep copies of the last N saves of the notebook in the checkpoint directory. N == 0 would be no checkpoints; N is None would be no limit (eg, don't delete checkpoints).

[ssanderson]

@Carreau @dsblank I've proposed elsewhere that FileContentsManager could support multiple checkpoints easily with a slight change to how checkpoints are stored.

Right now, the checkpoint for foo/bar/buzz.ipynb is stored at foo/bar/.ipynb_checkpoints/buzz-checkpoint.ipynb. Supporting multiple checkpoints could be accomplished easily by changing the storage-schema so that we instead stored cps for each notebook in a subdirectory of .ipynb_checkpoints. This would mean that foo/bar/buzz.ipynb's checkpoints would be stored under foo/bar/.ipynb_checkpoints/buzz/<uuid>.ipynb.

This could be implemented either as a behavior change to the existing FileCheckpointsManager, or as a new MultipleCheckpointsManager or somesuch.

[ellisonbg]

Would it make sense to delete the checkpoint upon notebook deletion, but
save a copy under a separate name somewhere? Accidental notebook deletion
is a pretty painful thing to deal with.

On Tue, Sep 8, 2015 at 8:57 AM, Scott Sanderson [email protected]
wrote:

@Carreau https://github.com/Carreau @dsblank
https://github.com/dsblank I've proposed elsewhere that
FileContentsManager could support multiple checkpoints easily with a
slight change to how checkpoints are stored.

Right now, the checkpoint for foo/bar/buzz.ipynb is stored at
foo/bar/.ipynb_checkpoints/buzz-checkpoint.ipynb. Supporting multiple
checkpoints could be accomplished easily by changing the storage-schema so
that we instead stored cps for each notebook in a subdirectory of
.ipynb_checkpoints. This would mean that foo/bar/buzz.ipynb's checkpoints
would be stored under foo/bar/.ipynb_checkpoints/buzz/.ipynb.

This could be implemented either as a behavior change to the existing
FileCheckpointsManager, or as a new MultipleCheckpointsManager or
somesuch.

—
Reply to this email directly or view it on GitHub
https://github.com/jupyter/notebook/issues/405#issuecomment-138608544.

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

[dsblank]

@ellisonbg Yes, that would be better than nothing! You could have a .ipynb_trash folder for example where deleted things of all kinds could go. You can make it an option that wasn't on by default, if you were concerned that some wouldn't want it.

In general, Jupyter can do much more to make it so that we don't accidentally lose files. Even emacs doesn't delete the tilde file :)

[dsblank]

@ssanderson I think it will save many of us many headaches if users can have a richer checkpoint system. I encourage multiple checkpoints, and to not delete them when deleting a notebook. Thanks!

[minrk]

I'm still -10 on supporting multiple checkpoints in the default manager. Then we are just reinventing VCS badly. Alternative checkpoint implementations can wrap git or database systems to provide it, but I strongly oppose implementing mini-git on the filesystem.

I don't have a good answer for accidental deletion. Failing to delete the checkpoints on deletion of notebooks means that we are guaranteeing that we clutter the filesystem with hidden files with no intention of or mechanism for deleting them, which is no good. On the other hand, we could add to the contents manager our own version of trash, which is reimplementing yet another wheel. If we do this, I would propose that we do our best to actually call system APIs to put things in the system trash, rather than implementing our own trash folder.

[dsblank]

@minrk I think it already implements a "checkpoint system" badly: there is exactly one checkpoint, which goes away when you delete the file. What kind of a checkpoint system behaves like that? I am not suggesting any other git-like functionality.

Perhaps you can rename the option to "Backups", and keep the last N backups, timestamped, in a round-robin queue.

Perhaps you can clear the "trash" when single-user server shuts down. Or clear out files that older than M days on startup.

I suspect that if you poll users, they would vote for "cluttering the file system with backup files" rather not having a single backup file.

But if you can hook into each system's trash API, go for it! Thanks!

[minrk]

@dsblank we never had any intention of implementing a complete checkpoint system. All we want is the most basic protection from autosave. The form this took is ensuring that 'manual' saves are always preserved, and can never be clobbered by autosave. That is the complete scope of what we intend to provide with the default system.

[dsblank]

@minrk You succeeded in that. However, there is a a more basic requirement that this doesn't cover: recovery from accidental deletes. In my opinion, this should be part of the default system. How about a tilde file, like emacs, that is replaced with last save, and is not deleted when you delete the file? Could be a default setting on whether to delete tilde file or not when you delete a file. The editor should also leave a backup/tilde file, too. This would be independent of the so-called checkpoints.

[minrk]

I think dealing with accidental deletes is a sensible problem to work on, I just don't think it has much to do with checkpoints. Replacing forceful delete with move-to-trash (however that's defined) would seem to be a decent solution.

[ellisonbg]

I agree with @minrk that the handling of deletes is separate from the
multiple checkpoint questions. Just focusing on the accidental delete
question here - I agree it is a problem and that we should come up with a
strategy for tackling it. I like Min's idea of trying to move it to trash,
either the OS level one or a special jupyter trash location.

On Mon, Sep 14, 2015 at 7:34 AM, Min RK [email protected] wrote:

I think dealing with accidental deletes is a sensible problem to work on,
I just don't think it has much to do with checkpoints. Replacing forceful
delete with move-to-trash (however that's defined) would seem to be a
decent solution.

—
Reply to this email directly or view it on GitHub
https://github.com/jupyter/notebook/issues/405#issuecomment-140100165.

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
[email protected] and [email protected]

[ghost]

Any solution to recovering an accidentally deleted notebook?

[minrk]

If you've just deleted the notebook, e.g. via rm, its last checkpoint may be in .ipynb_checkpoints/ in the same directory as the notebook. If you've gone through the confirmation dialogs of the notebook UI and confirmed that the notebook should be deleted, its checkpoint will be removed, as well.

[blas-ko]

Isn't there a way to recover a Jupyter notebook deletes by the browser? I'm kind of desperate about it ...

I want to recover the Construccion Tabla Enigh.ipynb , but I can't find it anywhere...

[takluyver]

There are 'undelete' tools which scan 'empty' space on your hard drive for files, because deleting a file doesn't normally wipe the data. You may have some luck with that, but it only works if the space hasn't been used for another file.

[evcu]

that is not user-friendly, accidentally delete my notebooks, instead of stoping them. And now trying to recover and there is no undo

[vivekananthrp]

Hi, I recovered my .ipynb file from .cache please find the procedure here
http://stackoverflow.com/questions/38819322/how-to-recover-deleted-ipython-notebooks/44044643#44044643
-Vivek

[nudomarinero]

I have just been bitten by this "non-bug". The disaster came from a combination of factors:

• The use of a trash bin icon (generally associated to send files to the trash bin). This made me not to think deeply about the "delete" action.
• The selection of notebooks via the same checkboxes to "shutdown" and "delete" where they are kept marked after issuing an action. For example, after a notebook is shut down its checkbox is still marked.
• Scrolling in the home page that can make some notebooks go out of view still marked.

A notebook marked for shutdown that is out of view can be easily deleted without even noticing.

Perhaps this is technically not a bug but certainly contributes to a bad user experience. I encourage the developers to seriously look for a solution to this and to a way of recovering deleted notebooks (at least during the session or even better for some additional time after).

[takluyver]

1968 implemented a safer delete method. I think the main hold up is that we don't currently have a good way to get things back from the platform's trash system. This is especially important on servers where the user can't just pull up a GUI file manager to examine the trash.

[takluyver]

What do we want to do with this for 5.1?

• We could have send-to-platform-trash as an opt-in feature, but that may not help much - people are only likely to turn it on when they've already accidentally deleted something.
• We could turn on send-to-platform-trash by default, but there's no GUI way to see the trash or get files back. Maybe that's better than nothing. There are CLI trash interfaces, although none of them appear to be installed on Ubuntu by default.
  
  
  
  • On the other hand, if deleting doesn't free up disk space or remove sensitive information, and there's no obvious way to empty the trash, people may well be legitimately annoyed.
  
  
  • Send2trash, the Python package, has a couple of bugs that I don't much fancy having to explain to users, such as this one.
  
  
• I think it's definitely too late to implement our own trash mechanism for 5.1, and I'd like to avoid doing so at all.

(Or, of course, we can punt this for another release)

[minrk]

I think using send2trash by default is worth a try for 5.1. Even less-than-ideal bugs like https://github.com/hsoft/send2trash/issues/9 save users from data loss, even if the restore process isn't as convenient as one might like.

And if it fails, the failure mode is a failure to delete files, which, while annoying, is certainly better than losing data. Folks for whom it fails can disable the trash mechanism via config while we identify the bugs we need to catch/work around.

[takluyver]

Bumped to 5.2 along with #1968 because of issues with send2trash that still need fixing.

[renschler]

lost my notebook with 2 days worth of work. should have been using git :(
I agree with comments above, the UX is very bad here and I'm sure many encounter this issue
This will be a great improvement! Thanks for the work

[invisibleroads]

send2trash fails in a Fedora 26 container and I'm not sure why. Deleting files worked in the previous release but is broken now. For now, our workaround is disable the trash functionality.

c.FileContentsManager.delete_to_trash = False

Here is the traceback:

File "$VIRTUAL_ENV/lib/python3.6/site-packages/notebook/services/contents/handlers.py", line 235, in delete yield gen.maybe_future(cm.delete(path)) 
File "$VIRTUAL_ENV/lib/python3.6/site-packages/notebook/services/contents/manager.py", line 275, in delete self.delete_file(path) 
File "$VIRTUAL_ENV/lib/python3.6/site-packages/notebook/services/contents/filemanager.py", line 507, in delete_file send2trash(os_path) 
n
File "$VIRTUAL_ENV/lib/python3.6/site-packages/send2trash/plat_other.py", line 185, in send2trash dest_trash = find_ext_volume_trash(topdir) 
File "$VIRTUAL_ENV/lib/python3.6/site-packages/send2trash/plat_other.py", line 145, in find_ext_volume_trash trash_dir = find_ext_volume_fallback_trash(volume_root) 
File "$VIRTUAL_ENV/lib/python3.6/site-packages/send2trash/plat_other.py", line 139, in find_ext_volume_fallback_trash check_create(trash_dir) 
File "$VIRTUAL_ENV/lib/python3.6/site-packages/send2trash/plat_other.py", line 84, in check_create os.makedirs(dir, 0o700) 
File "$VIRTUAL_ENV/lib64/python3.6/os.py", line 220, in makedirs mkdir(name, mode) NotADirectoryError: [Errno 20] Not a directory: b'/home/user/Experiments/Logs/Untitled.ipynb/.Trash-1000'

[takluyver]

@invisibleroads can you open a new issue about that? At a glance, it looks like it's getting the trash location wrong.

[takluyver]

I've opened #3130 to discuss that problem.

Closing this issue since we now delete to trash by default, so there's a way to recover from accidental deletion.