Simplenote-electron: Offline changes from desktop can be overwritten on next sync.

[ghost]

Steps to reproduce

1. Desktop Client 1.0.8, Windows 10. Working as expected for a period of months.
2. Experienced network issue for approx. 2 days, continued to use desktop client exclusively during the window.
3. Then, when using iOS client after this window, noticed that existing notes had not been updated, new notes had not been synced.
4. When desktop regained connection, notes that had been opened on iOS w/ new update time pushed to client, overwriting changes.
5. Desktop history wasn't present in version history. New notes created on desktop remained there and didn't sync to other clients.

Note, didn't use other clients (web, mac) in the interim - just the W7 desktop and iOS client.

What I expected

Desktop changes would have taken priority over older notes and published as the current version.

What happened instead

Desktop changes were destroyed across ~6 notes by older content from the first mobile client to connect. Notes were not recoverable in the note history.

OS version

iOS - current release as of 18.07.2017
Windows 7 - 1.0.8

[dmsnell]

Thanks for finding this and getting some steps in! We'll try and see if we can replicate.

I'm confused some on step 4

When desktop regained connection, notes that had been opened on iOS w/ new update time pushed to client, overwriting changes.

Were the notes new on the desktop and unsynchronized in iOS? What exactly was overwritten? New notes on iOS should nominally be different entities altogether than new notes on the desktop.

[ghost]

Thanks for the reply, to clarify:

• The notes that were overwritten were existing notes that were on both Desktop and iOS.
• The offline desktop client had new content, that had not been synced.
• The online mobile client (iOS) had the old versions of the notes, that I opened to check.
• When the desktop client came online, the new desktop content was removed, and the old content from the mobile client was there.

New notes created on the desktop didn't appear to sync into the iOS client at the time, but were still there.

[dmsnell]

This seems related to an issue we had in the past. We might need some time to dig in but we take sync bugs seriously. Thanks again for the report and debugging!

[ghost]

All good! Good luck hunting it down. Put the suggestion forward via twitter / support that it might be worth having a diff interface for this stuff?

Breaks the simplicity mantra, but aside from if there are sync issues it would also be useful where multiple clients have been offline for a while and the notes have diverged quite a bit.

[gerv]

I've had something like this happen very recently. I was on a plane, and used the desktop app. Later, I found all content I'd added to the desktop app had been overwritten with an earlier version. My other client is the Android app, which is presumably where the bad data came from. As you can understand, losing content like this is most frustrating.

It may be relevant that I updated the clock on the computer running the desktop app at some point during the flight, and that the Android clock wasn't updated until I landed and connected to a mobile network. But then, I didn't have mobile data so it wouldn't have synced until I got to wifi, which was some time later. So it may not be a clock skew issue. And I'd hope your timestamping was immune to that sort of thing!

[gerv]

It may well be that notes I opened on Android during that period got wiped on the desktop, and notes I didn't open didn't. Looking at things now, a day or two later, there is one note which is still out of sync. The updates I did on the plane are still there on the desktop, but they are not there on Android. This is a note I almost certainly didn't use the Android app to look at during the period of disconnection. Both devices have had good connectivity for some time, so I'd expect the difference to be resolved by now, one way or the other.

Is there some way I can determine the "sync status" of this particular note at each end, which might help you debug this problem?

[ghost]

@gerv, sounds similar to the sequence I managed to create.

It's an interesting case, I'm assuming it's supposed to roll the versions from different sources into the note history in the order of their timestamps. If not, it might not be the worst approach so at a minimum it retains the full documents from each source if there's a conflict or the sync process breaks down.

Diff of changes from different sources would be my preference but that can be intimidating for non-technical users, at least from what I've seen form people's first exposure to the process.

[dmsnell]

Thanks all for commenting here and helping to provide context for the issue. We certainly don't rely on any timestamps when updating notes. One of the design decisions at play here is having a single central writer so we have been able to leave out all the complexity of trying to rearrange edits: the result is that edits are first-come-first-serve on the server.

@jleandroperez may have some insight from issues in the past on this.

There is one case I'm ware of where changes will be lost but they _should_ be saved in the note's revision when things are working properly. Let's say we have a Mac app Steve and an Android app Larry. When we start on our flight Steve and Larry have the same data. We make edits on Steve offline to some notes. When we arrive at our destination we open the notes on Larry and make conflicting changes and those are immediately sync'd to the server. When we connect Steve back up to the internet and it attempts to send its changes to the server, the server says, "I'm sorry Steve, but the note you edited has been changed by Larry since you made your edits and I can't automatically merge them. You will need to tell me what to do because I can't find a way to make both edits fit together." Then Steve is supposed to decide if it wants to completely wipe out the changes Larry made or forget its own changes and accept Larry's changes. I haven't reproduced this yet but I'm guessing that we're dropping the changes on Steve to accept Larry's changes, if you will.

Now this scenario should only happen if we make conflicting changes. For example, if at the beginning we had a note whose contents were "Pet" and Steve changed it to "Cat" while Larry changed it to "Dog." For most changes they won't conflict and we wouldn't be considering this.

If it can Simplenote will merge non-conflicting changes so this won't be a problem.

Diff of changes from different sources would be my preference but that can be intimidating

This is certainly true. It may be possible to do for small changes but presenting the kind of diff that developers are used to might be very confusing. I can hardly resolve merge conflicts on source code I deal with and I've been doing it for years 🙃

---

I don't think it's likely we'll be able to get this resolved immediately but it is one of our top priorities. Unfortunately we're in the middle of a sprint to get v1.1.0 out the door but after that I think we should have more resources available to focus on this. In the meantime, thanks again for all your wonderful help!

[gerv]

dmsnell: I confess I'm somewhat confused about how the History is supposed to work, particularly on the Linux app, because I always find it only has a day or two's history in it, even for notes which are older than that. And in this case, as far as I could see when I checked, it didn't have the lost changes anywhere in the history.

A first-come, first-serve, throw-things-away system is not really "sync" in any meaningful sense, and it's a bit disappointing to hear that this is the algorithm! At the very least, why not create a new note called "Note Title (conflicted version)" or similar, which has the version which would otherwise have been thrown away, and let people resolve the changes manually, or just delete that note? Just throwing away stuff seems like a very bad solution. I agree the usable solution is not to have people manually resolving line-by-line diffs. I guess you could also make it appear in the history properly, but only if the history has some level of actual persistence.

But going back to this problem, I'm not sure I made actual _edits_ on Android. So perhaps the problem is that it's triggering the "edited" flag when it shouldn't?

[dmsnell]

because I always find it only has a day or two's history in it, even for notes which are older than that

That doesn't sound right. It should have history going back to the very first edit of the note (within reason - after a thousand or so changes the very first one might disappear). Another thing we'll look into.

A first-come, first-serve, throw-things-away system is not really "sync" in any meaningful sense, and it's a bit disappointing to hear that this is the algorithm! At the very least, why not create a new note called "Note Title (conflicted version)" or similar, which has the version which would otherwise have been thrown away, and let people resolve the changes manually, or just delete that note? Just throwing away stuff seems like a very bad solution. I agree the usable solution is not to have people manually resolving line-by-line diffs. I guess you could also make it appear in the history properly, but only if the history has some level of actual persistence.

We don't throw things away: the apps are nominally designed to keep the "thrown away" versions in history. If we have a bug in that behavior that's a different issue. Simplenote is designed intentionally with the goal of never losing any data. The algorithm is there to prevent a host of related bugs other systems have when trying to determine ordering of edits.

But going back to this problem, I'm not sure I made actual edits on Android.

That's helpful to know, thanks!

So perhaps the problem is that it's triggering the "edited" flag when it shouldn't?

Well we don't actually have any kind of "edited flag" but instead always rely on the actual data: do we have changes registered for a given note…

[ghost]

Nothing additional to report, but thanks @dmsnell for the awesome writeups and insight into the sync scheme.

[gerv]

That doesn't sound right. It should have history going back to the very first edit of the note (within reason - after a thousand or so changes the very first one might disappear). Another thing we'll look into.

Looking at this, my two most heavily edited notes (Work Jotter and Personal Jotter) are the ones which seem only to have a day's history. And they are edited both from Android and Desktop. (Although given this bug I've taken to making a temporary note on Android for new stuff and then copying it over when I get back to my desktop...) Other notes have history stretching back months. So something is happening that truncates the history on those notes... A bit of fiddling hasn't let me to some steps to reproduce yet.

[dmsnell]

As an update for everyone here who has noticed history issues, I discovered an artificial limit in the Simplenote library for node environments. It was only loading up to the thirty-most-recent edits when more might exist on the server. I have a patch open to resolve this and we're hoping to get the fix into v1.1.0. That fix would open up the full history of revisions.

[GeoJunkie]

Just had a ticket (#696030-zen) with a user reporting this with tags.

[rohitagarwal257]

Any update? I recently lost my note while syncing between the desktop client and the web browser. Is there anyway in which the note can be recovered? I've looking in the trash folder as well as the update history, but nothing works.

[rachelsquirrel]

I've got a report in #841926-zd with steps for what happened.

Day 1. Changes were made to a note in the Windows app in offline mode.
Day 2. More changes were made to the same note in the iPhone app.
Day 3. Opened Windows app in offline mode and the changes from the first day were visible. Put the app online and the changes were lost.

[rachelsquirrel]

I've got another report of this with some slightly different information.

1. Changes are made to a note in offline mode.
2. The app goes online but the changes are not synced with the server.
3. The note is edited again on the Linux app and only then does the note sync with the server.

Changes in offline mode are not synced with the server until the note is modified when it's online again.

891721-zen

[tarikmh]

This happens to me constantly! I'm on the brink of dropping Simplenote because of all the info and hard work I've lost. Please fix this ASAP.

[cecoates]

User report in 989491-zen, sounds similar in the other direction (older desktop notes overwrote newer Android notes).

1. Changes are made to a note in the Android app, but in offline mode.
2. Windows app is opened, while the Android app is still offline.
3. Android app is opened after phone WiFi is turned back on, sometimes accidentally, but no changes are made in Android at this time.

4. The older Windows notes will now sync to the Android app, overwriting the newer changes in the Android notes.

   Android (4.4.2) Simplenote Android app (ver. 1.5.6) and Windows 7 PC app (ver 1.1.3).

[jenlynnemc]

User report in 1023014-zen. Similar to issues reported above.

1. Changes are made in the Windows App Offline
2. Simplenote and OneDrive were synced prior to being offline.
3. When the user went online, it downloaded the cloud version instead of uploading changes
4. Web app and Windows app showing same synced info.

[planetahuevo]

Oh! lost a lot of work from yesterday because of this... :(
Is there anyway to get the data back?
Anything at all we can do?

[planetahuevo]

Ok,
After some tests, the only solution I have found is not to use simplenote on multiple devices (or not use it at all) :(.
This is worse than not having sync at all, because loosing one day or one week of work is too much risk.
this should be top priority on any sync app.
I recommend to anyone reading this to change to a text editor with Dropbox.
Dropbox will never replace content and will create conflicting copies if something has a conflict and you can always go and review old versions, so you have the control all the time.

[ghost]

@planetahuevo I don't normally shill but this seems to be impacting a lot of people, I had a few more instances of content loss around Aug-Sep last year, then switched to Standard Notes (standardnotes.org). It uses a strategy where it keeps conflicting files instead of attempting to merge, no issues so far.

[planetahuevo]

Thank you @zedgoat
I will check that too. I found ResophNotes, which is a windows Client that sync with simplenote but also allow to save on the folder (dropbox sync) so I am trying with that, I hope dropbox copy will save the day when the sync problem happen again.
I am considering if sync with simplenote it is needed at all or not at the moment.

Also, I was able to reproduce the issue 100% so I can explain it here to the devs to fix, if they want to.
It seems that simplenote takes the lates update as the one to use, and never merges changes with conflicting copies.
I have 2 simplenote apps, laptop and desktop.
The desktop has always internet connection, but the laptop sometimes it does not.
I start the note on the desktop. Fine and sync ok.
Open the laptop and get the synced note, fine too.
Keep working on the laptop but eventually internet is not available.
I do not have internet, go home and the note is not completely synced.
I go to the desktop, open simplenote, and start editing the note again.
Then, after some new typed text, I realise I do not have the latest changes. But the simplenote has already sync the changes at that time.
I go to the laptop, open the app thinking that simplenote will be smart enough to sync and merge the note, but it does not.
Simplenote open the laptop, check that the date of this note is older than the on in the server and remove the laptop one (loosing all the work done and not able to recover because it never reached the server) and replace with the latest one from the desktop.
In the way, it also lose the historic changes from both notes, so you are doomed.

This is basic stuff and I cannot understand how the simplenote server is not smarter than this.

[MorgothSauron]

I did not have problem with Simplenote yet, but reading this makes me think I should look for an alternative before a problem occurs. I'll have a look at Standard Notes for sure.

[anaemnesis]

User report in 1161737-zen. Although potentially in reverse.

Changes were made in web app
Changes reverted after user opened the Android app
Both the cloud and offline app showed the older version
User was able to restore the updated version from the note history popup

[mgalbritton]

User report in 1213729-zen.

Changes were made on a mobile device not connected to wifi.
User opened up app on desktop and and changes were not synched.
Then when opening up the phone, while connected to wifi, changes from computer overwrote to mobile, causing a loss of data.

[sjaved87]

Another user report in 1220599-zen

The user was online while making the changes and here are some details about the application:

Some details: Windows 7, 64x. Simplenote version 1.1.3. Don't use markdown. Don't use it on any other platforms. Never shared or published. Dark theme. Sort by last modified. Display condensed.

[dubyajay]

A user report in Zendesk: #1243253:

Which app do you need help with?: Windows
How can we help?: I made a lot of notes in the Windows app while I was offline (on a plane); then when I connected, they simply disappeared (just now).
I've checked the Trash and 'previous versions' and can't find the notes.
Is there any way they can be restored?
Thanks

[erricgunawan]

Possible related report from 1252299-zen

Sometimes when i make a note on the MAC offline it doesnt quite sync to the phone (when I eventually go back online).

User asked if there is a manual sync process.

I suggested making a small edit to the note when they get back online to "force" the sync process, as @rachelsquirrel suggestion here.

Waiting for the user if the workaround works for them.

[sfbalg]

I just ran into this problem, too. Tried your suggestion, did not work. Anyone else?

[nv-quan]

I also ran into this problem today. I think this is quite serious for a multiplatform note taking app. Hope this will be fixed soon.

[mgalbritton]

Another report of this issue in #1383309-zen.

Scenario of what happened:
User was editing date with Win+iOS apps at work
Came at home, run Linux client and found:

• missing lines
• found old lines which they removed in Win/iOS clients few hours ago
• broken lines (damaged 2 lines concatenated to one line)
  They opened iOS client and this client synchronized this corrupted data from Linux

[mgalbritton]

Another report in #1392039

User uses Simplenote across different platforms and the syncing ends up overriding info.

[justinkterry]

This constantly happens to me using the Linux and Android apps, is any progress being made towards fixing this?

[franklin373]

Just begin use SimpleNote, most new version, only add 4 notes, I really like its sync speed and Markdown. I use multiple devices and browser. But second day after I use SimpleNote, bring back one laptop to company, find laptop windows app looks all good, but new modify content not save to cloud, for I use browser and iPhone do not show new change. No way out, so I close windows app and lauch it again, well, all new change lost. It is too bad, I give up to use. I really surprise why app cannot save to cloud but no warning appear.

[mirka]

This issue (offline changes won't get sent to the server at all if you quit the app before going back online), was fixed by #1098 and has been released in v1.3.4 🎉

Please post a new issue with details if you still see syncing issues in the latest version. Thanks!

[lagerratrobe]

Yesterday, I worked on a Mac while in the office and saved notes from the day in the Web interface. Today, I'm working from home on Windows. Initially, I saw the notes from yesterday and get a message to update to 1.9.x. I do so and now neither the local or Web App shows my work from yesterday.

Fortunately, the history does show my stuff from yesterday and I can "revert" to that version, but still, what a weird thing, to have the old version get saved as today's.

[hansbkk]

Just lost a bunch of data. Android only, but multiple devices.

How about a manual sync option, or at least display last sync timestamp?

[dmsnell]

@hansbkk please contact our support at [email protected] so we can see if we can help you find your data.

unfortunately if there were a manual sync option it wouldn't help much because the app is constantly syncing. we have been working on finding ways to better indicate the connection status though, which can show things like "the last time the app sent its own changes" or "the last time we heard back from the server" but can't guarantee "we have received all of the remote changes"

[alfem]

I have also lost lots of text. And the worst of all: I use simplenote for reminders, and obviously I can not remember what lines of text have I lost.

Syncthing between Android and Web app does not work. I loved Simplenote, but it is time to move on.

[MorgothSauron]

@alfem You should check Standard Notes I switched to it after using Simplenote for few months and loosing too many notes.

[alfem]

@alfem You should check Standard Notes I switched to it after using Simplenote for few months and loosing too many notes.

I have just moved to it. The automatic import was really nice (although it did not import my tags)