Wp-calypso: Post Editor: Autosaves are no longer saving a revision

[jblz]

I'm not 100% sure, but my gut says that something changed relatively recently in the way we handle revisions for post (and page) drafts when the editor performs and "autosave."

@designsimply & I both seem to recall that, until recently, the Calypso editor deviated from core WordPress behavior in that "Autosaves" to drafts resulted in revisions being saved (they're explicitly blocked from doing so in core).

When I look back through moderately old posts and pages, the revisions seem to corroborate our gut feelings. There's also this from the WPCOM support docs:

Each autosaved copy of the post is saved, and you can browse and restore prior versions by clicking on Revisions.

This is maybe related to the change in #19261: passing the autosave param to the API call results in the DOING_AUTOSAVE constant getting set. /cc @roccotripaldi

It defintely seems like the correct semantics to set this constant for these calls -- we are, after all, doing an autosave. This becomes a bit problematic, however, in the case of co-editing a draft (see steps to reproduce below for why).

My question here is, should we patch core somehow to allow us to save revisions for autosaves as before?
By not doing so, could subsequent changes by other users could result in an inconsistent content state and / or data loss?

Steps to reproduce

1. Starting at URL: https://wordpress.com/post
2. Choose a test site with multiple user account on it
3. Draft a post & make multiple changes
4. Witness the "autosave" feature kick in after periods of inactivity (the Save button changes to Saved)
5. Make some more changes and click Save before the autosave feature kicks in (do this a few times for good measure)
6. Open the draft for editing as the another test user
7. Make some changes & watch the autosave kick in
8. Return to editing as the original user

What I expected

• In item # 7 above, I expected to see my changes persisted because the UI told me they were Saved
• In item # 8 above, I expected to see the changes from the second user somewhere because they told me they were saved (because the UI told them they were Saved)

What happened instead

• The draft was not updated.
• No revisions were created.
• The contents of the Autosave were not available to the original author at all.
• The other user assumed their changes were saved in a way I'd be able to see them & I was not.

Context / Source

dogfooding #anxiety #painpoint.

[lancewillett]

@mcsf @mtias In the context of Gutenberg, will there be built-in revisions? Or will it tie into core functionality related to Calypso via an API?

One concrete example we need to tackle is real-time editing, where the client maintains state and pushes-pulls changes from the server via an API. Based on certain conditions, we'd love to be able to intelligently push client-side auto-saves to the server — which core WP doesn't currently support.

Knowing what's coming for Gutenberg will help Delta / Samus prioritize work on post revisions in Calypso. Thanks!

[lsinger]

An additional data point:

I edited a draft started by a colleague, but my changes didn't show up for me as a revision in the history. They appeared "saved." They showed up in preview. They show up now that the original author has published it.

And now that it's published, they even show up in the revision log / history — under the original author's name, and timestamped to when he published it.

[mtias]

@lancewillett we are not touching the revisions interface directly. There are issues with the API and how it handles revisions, as right now it is creating a lot. Ideally core knows what should be a revision and what should not be, and the client doesn't have to handle that.

[aduth]

The behavior was recently changed server-side defining the autosave constant, which is checked as a condition for aborting the creation of a new revision:

https://github.com/WordPress/wordpress-develop/blob/c52f6ba99510afa345637ca1734176bd3e88250b/src/wp-includes/revision.php#L113-L114

r165508-wpcom

[aduth]

Also noting that, as described, this is "working as intended" so far as how core has traditionally treated autosaves. The previous Calypso behavior of autosave was effectively the same as a "full" save, at least for drafts, revision and all. For WordPress.com, it's been fine enough, since it can be occasionally handy to persist frequent revisions, and we're more okay with the data requirement hit. Ideally there's some middle-ground where the server is smart enough to combine multiple changesets made within a short timeframe, or more efficiently store diffs between two states of post.

See also: https://github.com/WordPress/gutenberg/issues/3656

[kraftbj]

I'm having trouble finding it, but excessive revisions was a known issue/problem from the Jetpack side when using Calypso. When using Calypso, you could easily rack up hundreds of revisions for a single post, each one having only a few words changed, but with no easy way to filter them out.

One idea I had was add meta data to the auto-saved revisions that were saved so they would be automatically dropped after X number or drop any older than Y when the post was saved, so we could still keep the latest few autosaves in the end of a diaster, but not kill a db.

[jblz]

excessive revisions was a known issue/problem from the Jetpack side

Definitely -- the number of draft autosaves we were seeing in revisions were borderline problematic on non-jetpack sites as well (in terms of UX)

One idea I had was add meta data to the auto-saved revisions that were saved so they would be automatically dropped after X number or drop any older than Y when the post was saved, so we could still keep the latest few autosaves in the end of a diaster

Clever. Although, old revisions are cycled out as the limit is hit, so this might be kind of moot.

Something else we've talked about for a quick workaround is to trigger a call to create a revision on a significantly longer debounce timeout so we don't accrue so many revisions. This might be a happy medium -- at least on draft content where the Save / Saved indication is so prominent

[kraftbj]

Clever. Although, old revisions are cycled out as the limit is hit, so this might be kind of moot.

True. Core doesn't impose a limit, so Jetpack sites potentially could have a metric ton of revisions. WP.com looks to cap it at 25 for non blue/VIP flagged sites.

[rossanafmenezes]

Hi there! Is this being addressed? I have been getting some users on LC complaining about their changes not being saved even though it shows as saved and also not showing on their history so they can't go back to previous versions. I have just done a test myself to one the users (interstrates
) and while at my end the change was saved, the last revision showing was from 18 hours ago.

CHAT 1282963-hc
Ticket 845066-zen

[lancewillett]

trigger a call to create a revision on a significantly longer debounce timeout so we don't accrue so many revisions

This is the best solution, in my opinion — we want to avoid sending each character change as a revision. There are more elegant ways to solve it, but I think we should focus on other work.

[nosolosw]

This issue may be the cause for 1027324-zen - user reported their content was lost while writing and he can't access the history.

[dllh]

I think 1978240-chat may be another case of this, but I'm not certain.

[davipontesblog]

I have a user who swears they have worked on a post “all night” but there are no revisions in the post in calypso (post is saved as a draft currently). The post is currently blank. Do we perhaps not save revisions on drafts? I didn’t think this was the case, and there are no mentions we don’t here. Is this related to this issue?

1170022-zen

3719281-hc

[allykmyta]

I have a user who was working on a post (which is saved as a draft) and the next day found that half their content had disappeared with no revisions available. User saved the post not only by the autosave feature but confirmed that they have also clicked "Save" as well throughout the building of their content. I'm thinking this is related to this issue but want to double check.

1176908-zen

[pauljacobson]

We have what appears to be a related report in 1235098-zen. Here is the initial issue:

I have a specific page following is the its link. This page doesnt open for me and when I click it, it keeps loading but never shows. I had the same prob before so i created a new page with the same content. Looks like the problem continues.

The user and HE troubleshooted by checking alternate browsers, clearing browser cache, and checking browser extensions. These didn't appear to resolve the issue. The user's subsequent testing seems to point to an autosave issue.

Steps to reproduce

The user did some testing for us in order to reproduce the issue:

1. Open a page (after logging into Wordpress)
2. Click edit button at the bottom of the page.
3. Make some changes to the page.
4. Don’t save it, leave the page for say 15 min.
5. Hit Update button at the top right corner, the draft expires because of inactivity, so it will say ‘cannot save’ so close the edit or close the browser.
6. Now, try opening the page, and it will say page not found.
7. When You (on ur end) open the same page, you do it as an external user and not logged into Wordpress user, so it works. For me as long as I am logged in I can’t open so I can’t edit.

There is a video in the ticket (I haven't included the video here because it may contain personal information).

What I expected

The autosave feature should preserve edits to a Post or Page, leaving the published version accessible through it's link, while retaining saved updates for further review and publication through the Revision History.

What happened instead

Leaving the Page open in the editor for 15 minutes, then returning to the editor to save the Page, results in the Page becoming inaccessible using it's URL.

Browser / OS version

Google Chrome and Internet Explorer on Windows. The user mentioned that when they tested the specific page link in Safari on their Mac and iPhone, the link worked.

[eurello]

I think this might have happened in #1279237-zen. User had a very long post that they worked on one day, but we have no revisions saved from that day, only revisions from several days before and several days after.

[KokkieH]

Several more reports that appear to be about the same issue in p2EDhh-tH-p2

[jblz]

Pursuing a server-side solution in:

• D16545-code
• https://core.trac.wordpress.org/ticket/44786

[kristarella]

Throwing in my 2c that I've tested this in relation to #1385513-zen and found it is definitely the case that autosaves are not saved as a revision, and I think this might be contributing to more cases of lost text overall because a user may not manually save ever, then something random goes wrong and they lose the post in the browser and can't get it back.

[JoshuaGoode]

I have two potential tickets related:

• 1442402-zen this user added text and images to a post over the course of a couple hours. You could see images added via the activity log; however, all content was missing later and no revision history was available. The user canceled their plan.

• 144250-zen this user utilized Gutenberg for the post in question so it may not be related; however, a similar situation where all their content went missing and there was not a revision history. I witnessed the content before it vanished and can verify they had a lengthy block of text.

[pauljacobson]

I have another instance of this in 1518731-zen where a user wrote an initial post. They then continued working on their post, and closed their laptop after editing their post further. When they returned, the updates didn't reflect in the draft.

I just did an hour of work on a post, assumed that the autosave feature which is displayed on my draft post page was working, closed my laptop to take a break, and came back to find none of my work was saved. How do I recover the work I did?

There is no revision history link available in the post editor. I also don't see any updates in the Activity Log (the user has a WordPress.com Premium plan).

We canvassed connectivity issues with the user but they said the following:

I think the issue goes deeper than just a connectivity issue. I was working on a draft for over an hour and there's no way that I was disconnected from the internet for an hour. The current draft that shows
up on my site is one that was saved before my last work session (where I lost my work).

The user's browser and laptop: Google Chrome Version 70.0.3538.77 (Official Build) (64-bit). I have a 2010 macBook Pro running High Sierra 10.13.6

[kristarella]

Had another chat where this was a problem for a user 3935536-hc

[kristarella]

Another one 6043116-hc

They tried to add a caption to an image and the whole editor went blank, previous hours of work not present in revisions.

[iamgabrielma]

Adding 1518731-zen

@jblz @lancewillett we're starting to see more users experiencing this problem. Any specific team/person that we should loop into the conversation to push this further and/or raise awareness?

[lancewillett]

Thanks for the updates and ping on this. We're tied up a bit in other work, and other teams familiar with the editor are actively engaged on the Gutenberg migration to WordPress.com. We'll check back soon, but cannot promise a quick fix right now.

[sashastone]

Another instance: 1559738-zen

[KokkieH]

Another report via Twitter. Also worth noting is that they specifically mentioned using the new editor.

[katiebethbrown]

Another here: 1607752-zen

[separatelyunited]

Another report here: 6549816-hc

In this case, the user's post shows it was last updated 3 hours ago with 1 revision from 5 hours ago, but the revision only shows the post title being added.

Looking in the Activity log, you can see that there were images added to this post during that time frame, but there are no traces in the revision history.

Follow up to the user in 1672102-zen

[iamgabrielma]

One more: 9143124-hc

It seems that since the change it's not auto-saving after i type everything. And... even after i hit save, i just restarted my computer and it didn't save the changes i had made. I also cant find the history

[JoshuaGoode]

Another report on 8527753-hc

Classic Editor on Chrome. User notes that they had their editor window open, never closed, for around 2 hours.

Activity log shows images being added to posts and that post was published 3 hours ago. However, only one entry in the revision history. 800+ words and embedded images were reportedly lost.

[mattsherman]

@creativecoder Did you uncover any new insights into what is going on when you started looking into this?

[iamgabrielma]

One more: p9F6qB-2fB-p2

[iamgabrielma]

1727131-zen

I have been working all day on a page. Suddenly,
the page went blank. No writing. No pictures. No feature image. Nothing.

[jessestu]

6140885-hc | Editor: classic

Another report triggered by the image caption issue that @kristarella mentions above. From the user:

I just neared completion of a blog post about my 2019 goals. The button to add a caption to a photo seemed bubby, and when I clicked on the caption button, it deleted all the written content for my post. Then, when I went into draft history, it only had versions saved from Jan 2, 2019..although I had saved edits from today, yesterday and last week...but they aren't visible?!

I've verified nothing is in the revision draft. Image uploads to the post are noted in the Activity history.

This is a trust-breaking bug given that this user specifically clicked Save Draft multiple times over the weeks of working on the post and trusted our software to save it.

[kristarella]

9957781-hc | Editor: block

User lost latest revision, despite it definitely saying it was saved. Spent a loooong time trying to find a way to retrieve from Chrome cache, but couldn't do it.

[iamgabrielma]

1779354-zen

Revisions I saved are not showing in the history.

Asked for more details and will update the comment.

[ajaykj]

From: 1779354-zen (reported by @iamgabrielma) above

I use the original editor and was using auto-saves, but I made sure to click save through the process as well.
I finished editing around 10:20 EST last night, but I do not remember if I left it on an auto-save or clicked save. Regardless, I would think there should be some record of saved progress from last night.

When I logged on this evening, the post was as I had left it. I made a few adjustments to an image, saved, continued to tinker and then the entire post completely disappeared. I had thought since I had saved I would be able to reload or get back to the save I had started on - the one from 10:20 EST on 2/4 - but that was not the case.

[ajaykj]

10320521-hc | Editor: Classic

Using Chrome on Windows 10. The post was empty when published. The user edited the post later and added images to it. This wiped the contents of the post. Nothing in the post revisions.

[kavyagokul]

4808532-hc
Using classic editor, auto-saves are not saved as revisions, and content disappeared after adding images.

[kavyagokul]

10638730-hc
Using block editor, classic block. Saved content disappeared after uploading media.

[katiebethbrown]

10737580-hc

they saw "saved" before publishing and saw the content on the preview and once they hit "publish" all content disappeared.

Using Safari.

I can see images added and the post published in the Activity Log, but the post is blank with no revision history.

Screenshot of Activity Log is in the notes.

I do see that they were able to publish the post and then there is a "Post Draft Modified" activity 3 minutes later, so I'm not sure how they modified a draft version after the post was published but that might have some significance.

[kcswong]

10931575-hc
Using classic editor, they published but nothing appears in revision or on published post. You can see an instance of 6 images uploaded to the post in Activity.

[jessestu]

4792678-hc using block editor (I believe in the iFrame).

I spent all afternoon writing an article and wordpress saved NOTHING! Is there any way to try to recover this?

Not with this bug except to not use our product. 😢 I'm refunding them some money and directing them to use WP-Admin.

[galakhyati]

The user's post (https://wordpress.com/post/karenzzeit.blog/5033) content went all missing, when they added the images to the post. There is no History link on the post.

I was able to replicate on the post, but not on a new post.
The revision is showing on Activity page, but not on the post.

Chat: #10747710-hc

[TeniCola]

Another report in HC-11163478 / ZD-1874124, a user who lost revision data from their drafted post when they tried to add a caption to their existing image. It also happened again shortly afterwards during said chat, and the user was using the Classic editor.

Though this issue suggests that this seems to happen with the Block editor, I advised the user to try using the new editor for the time being to help avoid this, but they're still wary about losing more progress.

[samiff]

Another in 2201309-hc

Calypso editor being used. Can see where the user attached image to post in Activity log, but the draft post is completely empty with no revision history.

[joweber123]

Another instance of this in 11630549-hc

User saying they purposefully saved posts but those revisions are not showing up in history. Using classic editor.

[joweber123]

Another instance of this in 11633045-hc

User had many hours of editing where they were actively pressing the save button but nothing is appearing in the revisions history.

[synora]

Another report in 9872166-hc.

User the classic editor and relied on auto save to save content. It wasn't saved and the latest revision history was from 12 hours ago. They're reporting they've worked on the draft since then.

I did note in Activity Log that they uploaded images to the draft post. They're not sure if the content went missing after the images were added.

For now, I suggested manually pressing Save in the draft post and checking the revision history to ensure we've saved their content. I performed a demo of this in chat (using a draft post on their site) with the user to confirm this workaround was solid.

[synora]

10306103-hc

Relied on auto save and then clicked the Save button to save a draft page, only to find it didn't actually save. The content wasn't within their Revision History either.

They added the missing content in chat, click Save, then viewed it with the Preview button, but it disappeared when they went to publish the page. I was unable to replicate this. Anything I saved and published from the history appeared on the site.

I also suggested placing the content on a fresh draft page to see what happens there, but they had to leave.

[damiannep]

Another report in #1942143-zen.

[lancewillett]

Wow, this is a long report now. Please keep adding customer insights and data so we can see the patterns and know if all the above user reports are related to the same underlying cause.

The original theory of this issue is that if we create revisions on a certain frequency of autosave events we might help avoid lost content during publishing sessions.

What are the next steps to investigate and close this one?

One area to explore could be autosave failures... are we capturing the error state properly? Do we know why the autosave is failing?

If the errors aren't clear, we should at least send a message to the user about the autosave failing for context.

Then, we could look at creating revisions from autosaves based on a logical debounce rate — not every character stroke — but some logical frequency.

[roccotripaldi]

I wonder if recent reports are related to this:

https://core.trac.wordpress.org/attachment/ticket/46076/46076.diff

Although, that would not explain reports of errors while using the classic editor.

[donpark]

@lancewillett - Some options to consider:

1. Remove Auto-save feature - As is, I think it hurts more than helps and there is some value in post not changing unless updated. No exposure to failures.
2. Reduce Auto-save to client-only thing - Changes are persisted to IndexedDB and 'replayed on load' only if post wasn't modified from elsewhere. On explicit update, changes are flushed to DB. Auto-save won't be a visible activity any more so similar to removal. Minor exposure to client-side failures.
3. Save only changes - Changes are appended to a post meta using CONCAT and merged into post_content on explicit update. Append only update means DB update failure impacts only only the latest change.
4. Send only changes - This is just optimization on item 3 to reduce load/diff/append cycle to just append. Reduced exposure to communication failure.

[donpark]

Some more thoughts on the issue:

• Auto-saves are not revisions - Revisions are history of intentional updates where auto-saves are snapshots of an update in the making. Revisions are what users remember where I doubt users can even tell when auto-saves happen. What this means is, when user is shown a history of revisions mixed with auto-saves, they won't be able to tell which is what unless they're marked and even then they will have hard time remembering which auto-save had what changes.

• Making auto-save smarter has limited gain - Smarter auto-save is one that is dynamic and adaptive, using a combination of a) auto-save interval setting, b) amount of accumulated change, c) nature of changes, and d) rate of change to find the best time to auto-save. But all that would accomplish is reducing the torrent of revisions down a notch.

• Revision, as is, cannot support auto-save - Current implementation was not designed to support auto-save use-case so disabling it was the right thing to do although doing so silently was not. Also how we access revisions do not anticipate worst case scenarios made more common by auto-saves. For example, there as a site that returned 500 whenever Customizer was opened. It turns out the post containing site's custom theme's CSS file had over a thousand revisions. While shocking, the number of revisions shouldn't be impacting Customizer yet it does.

[kriskarkoski]

1961030-zen

So, I was typing down some texts for the latest draft. I actually almost finished it. However, all in the sudden, the text was gone. Although I could still see the activities of me uploading some pictures to the draft at the 'activity section', I was no longer able to see those pictures in my draft nor the text. I couldn't find the revision bottom as well.

[kristarella]

@donpark Some thoughts on your thoughts

Auto-saves are not revisions

This might be true, but it was not always the case AFAIK. It also results in data loss if the current version gets lost. There is nothing to go back to.

Remove Auto-save feature

OR, stop auto-save from disabling the save button. One problem currently is that auto-save disables the save button, so if an autosave has happened recently, there is no way to save a draft.
Screenshot after autosave: https://cld.wthms.co/p2epJu

Revision, as is, cannot support auto-save - Current implementation was not designed to support auto-save use-case so disabling it was the right thing to do although doing so silently was not. Also how we access revisions do not anticipate worst case scenarios made more common by auto-saves. For example, there as a site that returned 500 whenever Customizer was opened. It turns out the post containing site's custom theme's CSS file had over a thousand revisions. While shocking, the number of revisions shouldn't be impacting Customizer yet it does.

It seems to me like keeping this many revisions is extreme anyway. Is there a reason we don't limit to the last 25 or something like that?

[donpark]

Remove Auto-save feature

OR, stop auto-save from disabling the save button. One problem currently is that auto-save disables the save button, so if an autosave has happened recently, there is no way to save a draft.
Screenshot after autosave: https://cld.wthms.co/p2epJu

Yes, that makes sense. If Auto-save is removed or hidden then Save/Update button should never get disabled. I suspect it's being disabled now to prevent auto-save and manual-save running concurrently.

Revision, as is, cannot support auto-save - Current implementation was not designed to support auto-save use-case so disabling it was the right thing to do although doing so silently was not. Also how we access revisions do not anticipate worst case scenarios made more common by auto-saves. For example, there as a site that returned 500 whenever Customizer was opened. It turns out the post containing site's custom theme's CSS file had over a thousand revisions. While shocking, the number of revisions shouldn't be impacting Customizer yet it does.

It seems to me like keeping this many revisions is extreme anyway. Is there a reason we don't limit to the last 25 or something like that?

I think it may already be the case in some context since there is a blog option that caps maximum number of revisions. It should be capped on Atomic but isn't and don't know why not. Calypso magnifies the down-side of auto-save so if the decision was intentional, it should be revisited.

[damiannep]

It seems to me like keeping this many revisions is extreme anyway. Is there a reason we don't limit to the last 25 or something like that?

A cap makes sense to me, but as a user and an HE, I really hope it's not the last 25. I could see some beginner users saving after every change, and losing more important, significant changes. I think having some timestamped revisions on some frequency would also be good, in addition to a certain number of recent changes.

[kristarella]

A cap makes sense to me, but as a user and an HE, I really hope it's not the last 25. I could see some beginner users saving after every change, and losing more important, significant changes

Yeah, I picked the 25 number partly out of the air and partly from my memory of this doc:
https://jetpack.com/support/custom-css/

Which says only 25 CSS revisions should be saved, but in the issue @donpark referenced regarding the customizer there were over 1000 revisions.

Although according to https://wordpress.org/support/article/revisions/ the default for posts is all the revisions.

Maybe we could add a new kind of save that says something like IF current save is autosave && revisions == 0 THEN save as new revision and another for IF current save is autosave && last revision is n minutes ago THEN save as new revision.

[synora]

Another report 12158016-hc.

They've published another post after some suggestions in a previous chat, only for auto save not to capture the changes.

[creativecoder]

A summary to help regroup on this very frustrating issue:

1. Root cause: blank autosaves

It appears that something is triggering an autosave call that deletes all post content, creating a blank post.

User reports sometimes indicate this is related to publishing a draft post or adding/updating media, but I've also seen cases when this appears to happen without these factors. And I found at least one case where the autosave call that wiped out the data seemed to happen after the post was published.

In the server logs of API calls, the only pattern I can see is a series of autosaves for a particular draft post that gradually grow in size, and then an autosave that's drastically smaller, followed by a support request about lost content for the post (DM me if you want links to specific log entries). We don't have the request content/body, so it's hard to tell exactly what's being sent, but a reasonable guess would be the smaller request for an autosave has a blank post body and then the content is lost.

I believe this issue only happens with the classic Calypso post editor, as when viewing logs we see calls to /rest/v1.2/sites/{site}/posts/{post}?http_envelope=1&context=edit&autosave=true.

Ideally, we could fix the root problem, and remove the mysterious request that seems to delete the post content. But I haven't been able to replicate this issue, so it's hard to know how to fix it.

Digging into how autosave is tied to lifecycle actions in https://github.com/Automattic/wp-calypso/blob/master/client/post-editor/post-editor.jsx might be a next step.

2. Making this worse: lack of revisions

Autosaves of draft posts do not create revisions. I believe this behavior comes from how Core works.

I think having draft posts save a revision in the same way that published posts do (see wp_create_post_autosave) would go a long way to preventing this kind of data loss. Note that autosaves are limited to 1 special autosave revision per post, per user, so as not to create many revisions with very small changes.

For drafts, this would still leave users vulnerable if they never manually save and the autosave revision was lost for any reason. So the UI updates suggested above to not disable the Save button when autosaving a draft are important, as well.

[donpark]

1. Root cause: blank autosaves

It appears that something is triggering an autosave call that deletes all post content, creating a blank post.

This one looks like a bug that is in the same _neighborhood_ but distinct enough to be broken out as an issue of its own.

2. Making this worse: lack of revisions

Autosaves of draft posts do not create revisions. I believe this behavior comes from how Core works.

I think having draft posts save a revision in the same way that published posts do (see wp_create_post_autosave) would go a long way to preventing this kind of data loss. Note that autosaves are limited to 1 special autosave revision per post, per user, so as not to create many revisions with very small changes.

For drafts, this would still leave users vulnerable if they never manually save and the autosave revision was lost for any reason. So the UI updates suggested above to not disable the Save button when autosaving a draft are important, as well.

The function that determines how a post should be autosaved is autosave function in client/state/posts/actions.js.

For published posts, it calls editorAutosave() which eventually uses /sites/$site_id/posts/$post_id/autosave undocumented endpoint.

For drafts, it calls saveEdited() which results in either POST /sites/$site_id/posts/new or PUT /sites/$site_id/posts/$post_id depending on whether post ID exists or not. I think it's the /update endpoint that runs into DOING_AUTOSAVE blocker in wp_save_post_revision.

So, in theory, we could do the following:

1. If post has a post_id, use /sites/$site_id/posts/$post_id/autosave endpoint.
2. Otherwise, use POST /sites/$site_id/posts/new.

This assumes /sites/$site_id/posts/$post_id/autosave endpoint does not assume post is published.

[donpark]

@jsnajdr any thoughts on the proposed solution here?

[creativecoder]

This one looks like a bug that is in the same neighborhood but distinct enough to be broken out as an issue of its own.

Good point--blank autosaves seem to be the root cause of the support requests referenced here, but it's a separate issue from this one about autosaves and drafts. I'll create another issue for it.

So, in theory, we could do the following:...

As we chatted about in Slack, modifying and using the v1.1 autosaves endpoint for drafts seems like a good next thing to investigate in terms of trying to resolve this issue. A couple of caveats:

• We'll need to test this very thoroughly, as Core no longer works this way and there could be unexpected side effects.
• The v1.1 autosaves wpcom endpoint is used for the Classic Calypso editor. Gutenberg uses the Core v2 autosaves endpoint, which will continue to directly update the post when autosaving a draft. We'll either need to override this (which comes with some complications) or live with the incongruity of draft autosaves working differently for Classic and Gutenberg editors, which could be very confusing for our users.

[donpark]

We'll need to test this very thoroughly, as Core no longer works this way and there could be unexpected side effects.

Haven't gotten around to it yet but, yes.

The v1.1 autosaves wpcom endpoint is used for the Classic Calypso editor. Gutenberg uses the Core v2 autosaves endpoint, which will continue to directly update the post when autosaving a draft. We'll either need to override this (which comes with some complications) or live with the incongruity of draft autosaves working differently for Classic and Gutenberg editors, which could be very confusing for our users.

I think we'll have to cover both endpoints.

This snowball is growing. ⛄️

[noteeth]

1998636-zen

I had a chat that turned into the above ticket.
User lost a blog post called "Portfolio". Error message popped up (user doesn't remember message) and then it seems the editor crashed and the post they were working on became blank. The user had saved drafts but after the error no drafts were retrievable.

[druesome]

1993440-zen

User scheduled a post called "Why Facial Symmetry Is So Attractive" on their website https://thehumanevolutionblog.com/ on April 29, as can be seen in their Activity Log. For some odd reason, the post content got completely erased and left the user with just one revision that is a very early rough draft of the final version. Is there anything we can do for this user?

[donpark]

Is there anything we can do for this user?

AFAIK, no. Sorry.

[GeoJunkie]

Another one here: 2006710-zen

User was editing a post titled "Family (2019) |Movie Review|". They went to edit a caption on a photo and it blanked their post out, with no revisions available.

The combination of not storing autosave revisions and wiping the contents is troubling.

[kcswong]

2012206-zen

Same as the last comment, this user was trying to move an image by cut/paste to another area of the post which caused their content to disappear (all except the image they moved) and all revisions were wiped. Using Classic Editor.

[donpark]

Do these recent incidents involve multiple authors? If not, it'd be nice to know how to reproduce the problem.

[davipontesblog]

Another report on #2035462-zen. As reported by the user:

...it was called Why Letter Writing is Important and I was last editing it about an hour or so ago. It had been auto saving as usual but then the page changed to the homepage and my post hasn’t appeared in draft sections or trash sections and I would like to see if it’s possible to retrieve.

[liviopv]

Another report 2033198-zen

User was working on a post and seeing the post being autosaved. Power went out and nothing was actually saved in the revision.

[thebud15]

Reported in #12920112-hc with a follow up ticket at #2046971-zd

[ccwalburn]

Another report in 2338996-hc.

This user said they lost their content while editing a caption. They had seen WP autosaving multiple times as they were working, but when they went back to the post, there were no recent revisions.

The last saved changes should be on May 27th just before 12 pm, but the Activity log only shows changes on May 26th.

They said that this was the second time they had trouble while editing a caption. The first time, the screen went white while editing a caption, but they didn't lose the content.

[rachelwinspear]

Another report in 1005366-hc - Follow up ticket here: #2062945-zen

[rachelwinspear]

Another report: #13049627-hc - Follow up ticket: #2064875-zen

most of the times when I finish a blogpost and I save it as a draft of publish it, it fails to post or save. Only after I push 'preview', exit the preview and save/post it does work.

[eoinsav]

Another report: #9592817-hc

I worked on a blog post all morning, and accidentally closed the page. Thinking it would be easy to retrieve, I looked in drafts for it, but only see yesterday's work!
I saw that it said that it was autosaving but did not manually press save

[coder-karen]

Another report: #2067164-zen

User pressed save multiple times on a draft in the WordPress.com dashboard (Jetpack connected site), but all content was then lost when checking later and no revision history to be seen.

[kavyagokul]

Another report in 6202924-hc

[tbradsha]

Angry user here: #11053436-hc

[Gilmoursa]

Another report in: #10156412-hc

[lancewillett]

Let's please continue pushing this ahead to a fix.

Re-ping @jsnajdr on https://github.com/Automattic/wp-calypso/issues/20265#issuecomment-485904619

[jsnajdr]

I did some research about how autosave works in Core and WordPress.com and looked at the proposed solutions.

Autosave in WordPress Core:
if the post is already published (i.e., not a draft), then autosave stores one special post revision (with a special name) per author. If that autosave revision already exists, it's updated and the old content is overwritten. If it doesn't exist, a new revision is created from scratch.

If the post is a draft and it's being edited by the post author, the autosave just updates the draft post, almost indistinguishable from a regular save. No special autosave revisions are created.

The only difference from a regular save is that draft autosave doesn't create a new revision. The intent is to avoid spamming the revision history by autosaves, and to create revisions only on intentional saves initiated by the user.

This "no-revision-on-autosave" policy is implemented with the DOING_AUTOSAVE env variable. When doing a save, the new revision is created by the post_updated action hook calling wp_save_post_revision. That function returns early and does nothing if autosave is in progress.

The "no-revision-on-autosave" policy is likely the cause of the issue reported here. If a broken or empty draft content is autosaved, the old content is lost forever. I suspect that most cases of this are caused by some Calypso Classic Editor bug that throws a JS error, clears the content and then saves it anyway.

@azaozz created a Core patch that saves a revision in case when the new autosaved draft sufficiently differs from the previous one. Avoids one character diffs.

Autosaves on WordPress.com:
The behavior is currently the same as Core, only the implementation is different. It predates the Core endpoints, as these didn't exist at all until recently when they were needed to implement Gutenberg.

The autosave logic is in two WP.com REST endpoints:

• the /autosave endpoint that Calypso uses only for non-drafts. Will save autosave revision per user. It also has a code path for drafts that just updates the post, but it doesn't set the DOING_AUTOSAVE flag and therefore doesn't avoid saving a new post revision. That's a bug. But this code path is never used.
• the v1.2 post update endpoint that has an optional autosave query parameter. If that parameter is true, then the DOING_AUTOSAVE variable will be defined and new post revision won't be created. Calypso uses this endpoint with autosave=true to autosave drafts.

It would be desirable to fix the draft code path in /autosave, stop calling the update endpoint and always call the one and only /autosave endpoint from Calypso Classic Editor.

Solutions:
@donpark asked some time ago:

[Solution:]
if post has a post_id, use /sites/$site_id/posts/$post_id/autosave endpoint.
[...]
any thoughts on the proposed solution here?

That triggers the buggy code path that doesn't set the DOING_AUTOSAVE before updating the post, therefore creating a revision. That's of course a desired behavior in our case, but... 🙂

Doing it so bluntly will lead to too many unwanted draft revisions being created. See this Gutenberg bug where the reporter got 9 revisions just by typing a formatting text for a while. That's exactly where we would end up if we do this.

I like the more subtle approach proposed in @azaozz's Core patch: create the draft revision only when the autosaved content significantly differs from the saved version. There are two alternative implementations in that ticket: one just compares the length of post_content, the other does a text diffs of the two post_contents (using the Text_Diff class) and counts changed lines.

We could finish and merge that patch to WP.com (quickly) and then also to Core (probably more slowly). Thoughts?

[donpark]

We could finish and merge that patch to WP.com (quickly) and then also to Core (probably more slowly). Thoughts?

I like the new plan. My suggested solution was half-baked, based on half-assed understanding of how autosave works. Let's go with this.

[azaozz]

Core's https://core.trac.wordpress.org/ticket/43760 was one of the things that got delayed because all eyes were on the Gutenberg merge. Now that the block editor is merged and working well, it will be great to add this "safeguard" when doing autosaves, both to WP.com and to core.

Thinking the text diff approach will be more precise and quite slower. Since this will always run in the background/asynchronously speed is not that important, so perhaps it is the better choice.

However counting lines may be somehow problematic. It is possible to have all of post_content on one line which would mean lots of revisions with minimal changes. Thinking we should be counting something else, perhaps number of changed places + size of changes, or just added + deleted characters (will comment on the core ticket too).

I'll milestone the core ticket for WP 5.3. Let me know if I can help further.

[jsnajdr]

Thinking the text diff approach will be more precise and quite slower.

I don't know how much the slowness is caused by the Text_Diff implementation, which might do much more work and create a more complex data structure than we really need to make the save/no-save decision. Some streamlined version of a diff algorithm that just produces the count and sizes of the add/remove/edit ops could help here. Processing two strings, even long ones, should be superfast in 2019.

Would it make sense to implement the whole decision algorithm inside the wp_create_revision_for_autosave filter and make it overridable?

One more thing: we need to be careful about which autosave exactly we store permanently as a revision. Consider the following edit:

• old_autosave: a very long post carefully crafted over last few hours
• new_autosave: corrupted or empty string

We want to make the old_autosave into a permanent revision. The new_autosave significantly differs from the old one and triggers the permanent save, but it's not the actual thing that should be saved. It can be a bit counterintuitive 🙂

[azaozz]

Would it make sense to implement the whole decision algorithm inside the wp_create_revision_for_autosave.

Yes, thinking it can be in the function for now. Can't think of any other place such logic may be needed at the moment. If it is, we can easily abstract it a bit and move it to its own function/method.

One more thing: we need to be careful about which autosave exactly we store permanently as a revision.

Right, it is a bit counterintuitive. We always want the post's current/old post_content data, not the new data coming in the autosave when that new data will overwrite the post's post_content.

This actually brings another question: should we be looking at creating "interim" post revisions when the user hasn't saved a draft post for a while? I.e. do the same thing as if the user clicked on "Save Draft". This can perhaps be a strlen() and time based comparison to the last post revision, and will reduce the frequency of autosave revisions. (This is something for the client, not the server.)

[jsnajdr]

This actually brings another question: should we be looking at creating "interim" post revisions when the user hasn't saved a draft post for a while?

What would be the benefit of that? Do I understand correctly that the condition would then be both diff-based and time-based? I.e., "create a revision if the content changed significantly OR a long time elapsed since the last revision"?

[ccwalburn]

Another report in 2147723-zen.

[lancewillett]

Project update: @jsnajdr will continue working on this, with input from others, after his break July 1-7.

[noteeth]

Another report in 2155363-zen

No auto-saves to a post during 3 hours of active post creation with auto-save on.

[marcuswisecaesar]

Report here: 2170419-zen

No auto-saves to a post during several hours of work. History does show a save 6 hours ago, but that save is devoid of content.

[ajaykj]

13978631-hc

Content lost when adding an image. No post-revisions or history found on the post. The user was using Chrome.

[kcswong]

Content lost when adding an image as well. No history. Using Chrome.

14033108-hc

[anant1811]

Another report in 2175757-Zen. In this case, the user mentioned seeing it autosave but later when they went back to the post, they realised they have lost the changes.

Also, no post-revisions or history found on the post.

[anant1811]

Another one, 199847-hc
Post having issue: post/pensionarnysspensionar.com/21721/

This user also reported that they saw the post Autosave.. and later, the content was missing because they didn't manually save/update it.

[jsnajdr]

Fix for Simple sites was deployed to WP.com earlier today in D30389-code.

If an autosave is significantly different from a previous one that is going to be overwritten, we create a permanent post revision from that autosave. Normally, revisions are not created for autosaves. Now, if a user finds out she lost her post content, she can open the Post Revisions dialog and the lost content should be there as one of the last revisions.

More patches will be needed to deploy this fix for Atomic and Jetpack sites, too. I'll post updates as we proceed.

[thesacredpath]

User just provided a screencast of the issue.

https://trendings616160297.blog/screen-recording-2019-07-23-at-1-41-52-am-mov/

2199399-zen (Atomic site)

[eoinsav]

Another report of a page not auto-saving. They lost 40-minutes of work: 14316373-hc (WordPress.com Premium)

Update from Sheri:

User description of the problem:

i was just finishing up a new page
i hit 'publish;
and it said you are not allowed to edit this post
and now it seems to be gone?

Based on the user's description of the problem in this case, it sounds they opened a page for editing while logged in as one account then logged out of WordPress.com and back in as a different account (or maybe they just got logged out somehow?) and then tried to edit the draft they previously started. I am not sure how else to interpret their explanation because the 'you are not allowed to edit this post' message should only appear if the currently logged in user doesn't have sufficient rights to edit the page.

I also noticed that the a revision was saved for that user's page (blog_id 152521822, page_id 402, revision 403) when they created it on 29 Jul @ 18:29 and the revision showing the initial save had a title of "Auto Draft" and the content was completely empty.

[eoinsav]

Another report here: 14334065-hc (WordPress.com Premium plan simple site)

They saw it autosaving but came back to the post and their changes were no longer there. There are Post Revisions on the post but they are older, which I assume means the post was published and then changed to a draft while they made edits.

[designsimply]

@eoinsav can you please note the type of site when you add notes about new reports? And a quick user quote explaining the problem if you've got anything quotable/helpful!

[jsnajdr]

@eoinsav @designsimply There are several important details that are helpful when debugging autosave issues:

• is the site Simple, Atomic or Jetpack? We deployed an autosave enhancement/fix to Simple sites several days ago, but we're still wrangling the Atomic deployment
• which editor exactly was used? We have four combinations of Classic and Block editor in Calypso and wp-admin. They differ in several subtle details: whether they use auto-drafts or not, whether they ever lock the edited post or not...
• when the content was lost (that can happen for variety of reasons), is there a permanent revision (or several of them) that has been saved during the edit? That's what our fixes are about: make the content-losing bugs recoverable rather than attempting to fix all of them

[eduardozulian]

Another report:

• 3742331-hc
• Follow-up: 2254372-zen
• WordPress.com simple site
• Premium plan
• Classic Editor

User info:

1. Started a draft and added quite a lot of content
2. Made sure that the draft saved several times
3. Always save manually after an editing session.
4. Went to the draft today and all the content other than the title has gone.

Also:

• There were several hyperlinks in the text, but the user has used these links before.
• There were no photos added.
• The only thing different was that prior to creating that post, the user changed the theme to Clear News

[msilbers]

reported in #6127572-hc

WordPress.com simple site, Personal plan
They access their drafts from Calyspo, Block Editor
Safari version 12.1.1

They started a draft on 8/29 and worked on it for an hour or two, letting it auto-save the whole time. They say the auto-saving was visibly happening, so they never tried to manually save it. The post was all text only (paragraph blocks).

There are no revisions saved, not even the title of the post. It doesn't appear in the activity log and there's no record of a deleted post.

"I had written a post this last Thursday 8/29 titled Saturn Return, it said it autosaved but now it is nowhere to be found..not in drafts or trash, is there anyway to restore it - please help!" - user quote

[galakhyati]

Another user reported the issue: #2387674-zen

[designsimply]

@galakhyati can you add a bit more detail for the case you reported?

• Is the site Simple, Atomic or Jetpack?
• Which editor exactly was used?
• When the content was lost, was there a permanent revision saved?

[cameronvoell]

Have a user with missing revisions: 2411956-zen
Interestingly I've verified edits show up in the site activity log, but do not show up in the post history revisions.

• Jetpack site
• Classic editor was used
• Permanent revision saved: Yes, I think so... I read this comment (https://github.com/Automattic/wp-calypso/issues/20265#issuecomment-517405407) but not positive I understand what you mean. All post data was not lost. when the user returned to the post, it was just missing several of the most recent updates.

[kriskorn]

Happychat: 16340669-hc

_is the site Simple, Atomic or Jetpack?_

Simple.

_which editor exactly was used?_

Block editor in Calypso.

_when the content was lost (that can happen for variety of reasons), is there a permanent revision (or several of them) that has been saved during the edit?_

There is no sign of that post in the drafts or anything at all. No Activity log message. Totally vanished.

[ivan-ottinger]

Another report:

Ticket: 2470571-zen
WordPress.com Simple site (Free)
Block Editor, Calypso

User info:

I just wrote a test post on [site address], my wordpress.com site, and autosave never turned on. When I tried to navigate away from the page, it gave me the pop-up warning that I had unsaved changes

I also tried to test the autosave and it didn't work at all for me. Even after I clicked "Save Draft", the autosave didn't seem to trigger. 🤔

[eduardozulian]

• Chat: 17351599-hc
• WordPress.com AT site
• Classic Editor
• Edge 18.17763 on Windows 10.0.0

Extra info:

• That they were using the same computer on both edits
• They are using the Classic editor for a while now, so it wasn't an editor switch issue
• They didn't manually save the post but relied on the autosave tool
• No revisions saved

[kwight]

This thread contains references to all three editors (Calypso - the OP, Gutenberg, and Classic); I think we'll need to split these out to separate issues, since they handle revisions differently, and this issue is becoming a bit of a dumping ground for any revision issues.

[gwwar]

For folks that are assigned to this, please close out this issue in favor of a milestone or project that captures problems noted in this long thread. Note that this will likely require an audit before we can create actionable issues. Many of the reports here are based on content loss symptoms, which may involve many bugs/flows to resolve.

Once a project/milestone is created, we can help prioritize and get folks assigned to tasks.

See also p7jreA-1y8-p2 for tips on how to approach this or DM me if folks had questions.

[davemart-in]

The Calypso editor is in the process of being deprecated. I'm going to close out this issue.