Wp-calypso: Gutenberg: Compatibility issue in classic editor for some posts with Gutenberg blocks

[rachelmcr]

There is a compatibility issue between Gutenberg and classic wp-admin/Calypso for posts when:

• Created in the classic editor
• Gutenberg blocks are added without changing the classic block
• And the post is then updated in the classic editor again

h/t @KokkieH for the original report of this bug.

Steps to reproduce

1. Starting in the classic editor (in wp-admin or Calypso) create a new post.
2. Add some content to the post, and save or publish it.
3. Visit /wp-admin and selected "Activate Gutenberg."
4. Open the post in Gutenberg. (Note that the previously added content is in a classic block.)
5. Add a new Gutenberg block (e.g. a paragraph block) to the post and add content to it.
6. Save your changes to the post.
7. Deactivate Gutenberg on the site.
8. Open the post in the classic editor (in wp-admin or Calypso) again.

What I expected

I expected all of the post content to be there, including what I added in Gutenberg.

What happened instead

Only the post content added in the classic editor appeared. All of the content added in the Gutenberg block was gone from the editor in the visual or HTML view.

(In addition, in the classic wp-admin editor all of the content was collapsed into a single paragraph, regardless of the paragraph breaks that existed before.)

The front of the site showed all of the post content (including the content from Gutenberg) until I updated the post again in the classic editor, at which point all Gutenberg content was lost. I also did not see the missing content in the post revisions, so I couldn't recover it.

Browser / OS version

macOS 10.14.1 / Chrome 70.0.3538.77

[Copons]

I can't seem to reproduce this.

Steps I've followed:

1. Start with a site not using Gutenberg.
2. Create and publish a new post with some content (1 paragraph of text with a link in the middle).
3. Opt in to Gutenberg (which automatically redirects me to the wp-admin Calypsoified Gutenberg).
4. The post content loads as a classic block. Add 1 paragraph block and 1 image block after the classic block, and update the post.
5. Opt out of Gutenberg by clicking the "Switch to Classic Editor" in the More menu (which automatically redirects me to the Classic Calypso editor).
6. The post content loads as expected: the original paragraph with a link inside, followed by another paragraph and an image. If I switch to HTML view, I can see the <!-- wp --> comments inserted by Gutenberg.

[mmtr]

I can't seem to reproduce this.

Same here.

I published a post containing a paragraph and image and switched later to Gutenberg. After adding a new paragraph block in Gutenberg and publishing the post, I switched back to the Classic editor, where I can see both the previous content created in the Classic editor and the new paragraph added in Gutenberg.

[rachelmcr]

When I test this in a non-a8c account on a test site, I can still reproduce the issue.

Can you confirm that Gutenberg was deactivated on the site? The behavior you're describing is what I see when Gutenberg remains activated but I use the classic editor (in WP Admin or Calypso), but the bug occurs when Gutenberg is deactivated.

[Copons]

When I test this in a non-a8c account on a test site, I can still reproduce the issue.

Can you confirm that Gutenberg was deactivated on the site? The behavior you're describing is what I see when Gutenberg remains activated but I use the classic editor (in WP Admin or Calypso), but the bug occurs when Gutenberg is deactivated.

It might be a difference in what we mean with "deactivated".
I mean that Gutenberg (plugin) is installed and enabled, but the site's selected editor is classic.
(Basically by running dispatch( { type: 'EDITOR_TYPE_SET', siteId: state.ui.selectedSiteId, editor: 'classic' } )).

AFAIK, Gutenberg (plugin) is always installed and enabled now, isn't it? (cc other @Automattic/lannister that might have more accurate recollections of this)

[rachelmcr]

Yeah, hm, I probably should have specified in my repro steps above: I'm explicitly hitting the "Deactivate Gutenberg" button at /wp-admin on the site — does that do something more than the "Switch to Classic Editor" option? (It does set the editor preference to classic as well.)

[kwight]

I had no problem reproducing this with @rachelmcr 's original instructions, starting from wp-admin with the classic editor.

In my case, there was an autosave with the expected content, so I was able to roll back to that, but I'm not sure what causes the initial content when the editor loads to be lacking.

Deactivate Gutenberg on the site.

AFAIK, Gutenberg (plugin) is always installed and enabled now, isn't it? (cc other @Automattic/lannister that might have more accurate recollections of this)

Yeah, hm, I probably should have specified in my repro steps above: I'm explicitly hitting the "Deactivate Gutenberg" button at /wp-admin on the site — does that do something more than the "Switch to Classic Editor" option? (It does set the editor preference to classic as well.)

Yes, we have to be super clear about terminology with all of this 🙃 Gutenberg (the general code) is always "activated" now on dotcom; this means that the APIs are all loaded, regardless of what editor a user is seeing. This is what allows us to properly output Gutenberg blocks on the front-end, regardless of a user's current editor choice.

In turn, users now can select whether they want to use the classic or Gutenberg editor with the same dashboard admin banner we've had since the blog sticker opt-in was added at the start of September ("Activate Gutenberg"/"Deactivate Gutenberg"). We kept the terminology the same to avoid confusion, but maybe it only adds to it 🙃

All that said, this is a legit issue we need to dig through.

[kwight]

Noting that this might be related to wp-saving-post cookie checks; Gutenberg might not be handling this properly, but I'm not sure about anything yet.

[kwight]

@Copons @mmtr Are you able to reproduce starting in wp-admin? I didn't use Calypso at all to reproduce this.

I'm wondering if a few things are at play. From what I understand, core keeps tabs on post content in session storage, and updates it when the post itself is updated; I've noticed that this doesn't happen when updating a post in Gutenberg (the session storage for that post doesn't change). If we then switch to classic and load the same post, core seems to load what's in session storage (hence the missing Gutenberg content), and throws the not-helpful revisions error ("The backup of this post in your browser is different from the version below.) when comparing the two (the sessions storage version compared with the successfully-saved Gutenberg version).

This is just what I'm guessing from playing with it and observing cookie and storage behaviour; I haven't dug deep into plumbing yet to see if that makes any sense. (I'm also not clear yet how the wp-saving-post cookie handling is involved, if at all; it's possible Gutenberg isn't handling that properly on dotcom either.)

[kwight]

Curious if this started happening recently or has been with us a while, I tested some rollbacks, and was able to narrow it down to the recent merge of D20168-code (the commit before it doesn't have the issue of disappearing GutenContent, while the D20168-code commit does). Maybe there's a filter that needs to be added back in or removed? cc/ @brandonpayton

[mmtr]

@mmtr Are you able to reproduce starting in wp-admin? I didn't use Calypso at all to reproduce this.

Nope, I get the same result either starting in Calypso or in wp-admin.

I recorded some videos with the process I follow so you can double check I am not missing any step:

• Starting in Calypso
• Starting in wp-admin

[kwight]

After some (kinda) enlightening Xdebug work, I have another lead: it could be Markdown-related. @rachelmcr can you confirm that your test site _does_ have Markdown enabled, and @Copons @mmtr can you confirm that your test sites _do not_ have Markdown enabled? The setting can be found in Settings > Writing (wp-admin/options-writing.php):

[rachelmcr]

Interesting! Yes, my test site does have Markdown enabled.

[gwwar]

D18636-code may be related, unless @mkaz or @georgeh recall any other changes

[mmtr]

@mmtr can you confirm that your test sites do not have Markdown enabled?

Confirmed!

[brandonpayton]

I was not able to reproduce until I enabled Markdown. Now I am seeing the issue.

[brandonpayton]

In my testing, the latest revision still has both freeform and block content, but when Gutenberg is deactivated and the content is loaded in the classic editor, all post content including and following the first block delimiter is absent from the editor.

The good news is that there is still a revision with the missing content.

[brandonpayton]

It turns out we have core Gutenberg logic for disabling WP.com markdown when inserting a post:
https://github.com/WordPress/gutenberg/blob/8d9b4b380ea385b723c91889b4b4674504646073/lib/plugin-compat.php#L34

If I paste raw block content into the Classic editor and save, a revision containing that content is saved, but the editor continues to display the previous post content without blocks, even after reloading. I'm not sure why.

We may need to remove more markdown-related filters for block content. Time to dig a bit more.

[brandonpayton]

We have some internal patches proposed to address this:

1. D20947 - addresses issue of missing content.
2. D20852 - addresses issue of missing content and lost paragraphs.

There is also a Jetpack PR that will fix the case of lost paragraph. It is being weighed against D20852.
https://github.com/Automattic/jetpack/pull/10635

[brandonpayton]

D20947 landed on WP.com and addresses the issue of missing content.

The remaining issue when Markdown for posts is enabled and the user has not opted into Gutenberg:
Posts containing block content will have their <p> tags removed by Markdown processing but never added back via autop because we do not autop posts with block content.

[brandonpayton]

The good news is that once we fix the issue of markdown processing stripping paragraphs, users can re-save their post and the stripped paragraphs should be restored.

This is because of the way our markdown processing saves a copy of the original content and hands it to the editor as if it is the post_content. Once markdown processing is fixed, we can re-save content and have p tags preserved.

Note: Depending on the solution we choose, actually transforming markdown may not work for posts containing block content. If you have any input here, please let us know.

[brandonpayton]

Jetpack merged a simple fix for markdown processing of block content, and that fix should make its way back into WordPress.com.
https://github.com/Automattic/jetpack/pull/10635

[kwight]

Noting that JP 6.8 was released today, so this should be coming to WP.com soon.

[gwwar]

@dmsnell do you think we can put in a better fix now that we have the server parser work in? The workaround in Jetpack is still a bit of a hack. We can maybe follow up in another issue if that makes it easier.

[brandonpayton]

Note: The fix from Jetpack was landed on WordPress.com via D21362. We now avoid applying markdown processing to any post containing blocks.

[dmsnell]

@gwwar yes but maybe not quickly. as we now have the parsing functions available my approach is to run the full parser on all posts, even non-Gutenberg posts. we can apply the typical processing (autop, texturize, etc…) on only the non-block content.

I'm sure we'll need to start with an audit of some kind to evaluate how much work this will take and what places we'll need to update. just last week, for example, I was surprised by separate logic for multi-page documents.

well, that's maybe more global. when I spoke with @brandonpayton about this specific issue with markdown I think we can apply that same concept from within easy-markdown#transform or we could even fully hook into the Gutenberg rendering pipeline.

// not sure on the filter name - it's in Core but not in the plugin
$transformer = function( $html, $block ) {
    if ( null !== $block['blockName'] ) {
        return $html;
    }

    return markdown_transform( $html );
};
add_filter( 'rendered_block', $transformer );

$transformed = do_blocks( $text );

remove_filter( 'rendered_block', $transformer );

return $transformed;

[brandonpayton]

The issue of disappearing content is fixed, and if you edit a post with block content in the classic editor, paragraphs added via the Visual tab appear to be preserved well.

If you add lines in the Text tab of the classic editor, they are still run together, but fixing this will take the more targeted approach using the server-side block parser. We'll be creating another issue for that.

[brandonpayton]

well, that's maybe more global. when I spoke with @brandonpayton about this specific issue with markdown I think we can apply that same concept from within easy-markdown#transform or we could even fully hook into the Gutenberg rendering pipeline.

The plugin currently transforms markdown on wp_insert_post_data, saves the output to post_content, and saves the original content in post_content_filtered (a bit confusing : ). It would be great to continue transforming on save so we don't have to transform with each post render, but if we call do_blocks directly, we'll also be getting and saving dynamic block output which is not what we want. Is there a way we can invoke the parser directly and avoid this?

[dmsnell]

it gets complicated quickly. this is, of course, solved, if we move rendering to the editor in a way that doesn't require server-side rendering, but that doesn't help us hear.

we should be able to parse and reserialize but I'll have to do some digging - I don't know if the server has serialization logic built in.

$blocks = gutenberg_parse_blocks( $text );
foreach( $blocks as $block ) {
    if ( null !== $block['blockName'] ) {
        continue;
    }

    // we know "classic blocks" have one string item in their inner content array
    $block['innerContent'][0] = markdown_transform( $block['innerContent'][0] );
}
$next = gutenberg_serialize( $blocks );