Cherrytree: C++ Bug: Permanent Data Loss In File Attachments

[metal450]

I've had multiple instances (on Linux) where I'm working in an attachment, & edits get permanently lost. I've been trying to figure out how to reliably reproduce it, but haven't been able. However, I am able to anticipate when it's about to happen:

Typically when editing an attachment, if you save the attachment, CT will notice & show that the ctb has unsaved changes (note: I have autosave off). In some cases though, I will edit the opened attachment, save, but CT doesn't show any modifications. At this point, anything else done in the attachment will be lost. Even if I explicitly save in CT (i.e. even though it didn't show it had unsaved changes), it loses whatever changes took place in the attachment.

This has actually caused me some considerable data loss, until I realized I need to explicitly watch for it. When it does, I re-save the opened attachment elsewhere, then delete the attachment in CT and re-attach the correct/recently-modified version.

[txe]

In pygtk, instead of changing widgets, they are recreated, so the text buffer is flagged to be modified automatically. In C++, widgets just change their properties and the text buffer doesn't know that it was modified.

[metal450]

I'm not really sure how to interpret this? I didn't test the pygtk version, just in C++, it's not just a matter of the UI not showing that it was modified - even if I explicitly save, changes to the attachment are lost (when this occurs - which isn't always).

[txe]

Never mind, it was my theory, and I took a look, the theory was actually wrong. Maybe it is caused by another reason.

When an attachment is edited, we check every 0.5 second the modification time of the file. Maybe the time is not updated, so we don't see if the file is changed.
So, what kind of files do you edit and by what application?

[metal450]

Docx file, using WPS Office: https://linux.wps.com/

Again though, it's not predictable. I might be editing a file all day many times, & it successfully does save the changes. Then all of a sudden, it stops seeing the changes - and will not see them or that file any more, no matter how many more times I change it (unless I quit & re-open it). I think it may have to do with when I'm going back & forth editing the file & CT directly, but I just haven't been able to consistently reproduce it.

[metal450]

Once it gets into that state where it's no longer picking up changes, is there something I can do to help debug it?

For example, at this moment: I have an attachment open, I can see that the file CT "extracted" it to is /tmp/.9I57O0/8-1-32390-filename.docx, and when I save it, the modification date of that file has changed. But CT never notices.

I hadn't observed this issue earlier because I was working primarily on Windows, but now that I switched back to Linux this has become the most serious issue - as I have lost data now on multiple occasions :(

[txe]

I figured it out. When we save an attachment, we use a unique id in a file name and save this id in a widget.
When we need to put a changed file back into a widget, we find that widget by that id.
The issue here, after undo/redo, all widgets are re-created in a text buffer, and ids are lost.

@giuspen , I think of changing the current system of editing an attachment. It will be not easy to restore ids after undo/redo. Also, if user edit a lot of attachment, we will check a lot of temp files again and again.

Maybe during editing an attachment, to show a modal dialog and user press a button there (e.g. finish or update changes) after finishing editing a file? It will make the process of replacing data more robust.

[metal450]

My 2c: Showing a modal dialog makes things much less flexible & I'd _really_ hope there's a better way. I frequently edit multiple attachments at the same time, or have an attachment open & go back to work in the CT UI, referring to other notes, etc. If I'm understanding correctly, the modal dialog approach would effectively mean my entire repository of notes would be inaccessible until I close out the file I'm currently editing.

[giuspen]

@txe I will try and have a look tomorrow or one of the next days, frankly I don't remember how that works so cannot advise I need to spend some time and re-visit. @metal450 thanks for testing and reporting this issue and not giving up, obviously data loss issues are top priority for us.

[txe]

Just to say what issues I see currently:

• undo or redo recreates widgets thus unique id related to an attachment file is lost
• user can undo after an attachment being auto updated (without knowing about it) and thus lose the changes, Logically, an attachment should be equal to an attachment file.
• user can undo/redo a fresh created attachment, so it is hard to keep an unique id related to an attachment file.

[giuspen]

I haven't had chance to look at the code yet but I think we could track opened temporary files attachments in a map to node ID + attachment file name, and then force the user to use non duplicated attachment names in the same node

[giuspen]

I have started to look into the code and I changed my mind in forcing a different name for embedded files in the same node.
I'm thinking instead about adding a unique embedded file id/counter local to a node, this information hidden from the user and not saved to database (just populated at database read time/new added embedded file).
Maybe this unique identifier could come handy in future for other embedded widgets local to a node, so I could put it in the base class CtAnchoredWidget.

[giuspen]

Still thinking loud, if you cut and paste an embedded widget from a node to another node then it could be a problem to track the existing opened file... I should go for ids for the tree if not for the whole app

[giuspen]

If we support the copy/cut and paste it gets unnecessarily too complicated, furthermore I think that if we undo/redo we want to keep tracking the opened file but we don't want to drive the opened file content since that is driven from the external application.
So as a first step I will proceed with this unique identifier for an anchored widget which is also known in the undo/redo state machine, then the second step is to use that ID to track the opened file status.

[txe]

Hi @giuspen, it is a nice fix, though it probably don't address

user can undo after an attachment being auto updated (without knowing about it) and thus lose the changes, Logically, an attachment should be equal to an attachment file.

I think, just dropping mod_time in CtEmbFileOpened after undo/redo auto-updates the attachment.

[giuspen]

Hi @txe I thought to let the external application drive the embedded file if still opened, but I'm not sure it is the right thing to do... If you believe the behaviour you are suggesting is the right one please feel free to add your proposal