Wp-calypso: Gutenberg: Instagram embeds broken on Atomic sites

[vindl]

Update: this issue has been resolved for Simple Sites and Jetpack but it's still affecting Atomic sites.

Instagram embeds are not previewed correctly in the editor, although they are rendered fine on the fronted after the post is published.

Steps to reproduce

1. Use a Jetpack/Atomic test site that has opted in to Gutenberg.
2. Navigate to editor in wp-admin.
3. Insert Instagram embed, either by copying the link directly into the paragraph or using the Instagram block.
4. This results in Sorry, we could not embed that content.

For a full audit of embed issues please check out https://github.com/Automattic/wp-calypso/issues/28126 - there might be other embeds that are broken due to the same root issue.

[vindl]

/cc @georgeh @notnownikki @Automattic/lannister

[notnownikki]

The problem looks like the scripts attribute of the response is a different data structure to core.

In core, it's an array of script URLs that we create script elements from to inject into the preview sandbox. The WPCOM response is an object, mapping script names to URLs, so when we try to call .map on that, we crash.

[gwwar]

Thanks @notnownikki! ~What if we update the api middleware for wpcom to use the native core endpoint? This probably explains why the editor preview works for us in the native Calypso integration~

Edit: misread this issue!

[gwwar]

As an aside, I think we can probably help verify embed support for upgrades by creating a few test posts that we can verify manually or with a visdiff. We'll get around to this later, unless @Automattic/flowpatrol would like to be ahead of us.

[JavonDavis]

(there might be others, I didn't perform the full audit)

I did a full audit on this manually here -> https://github.com/Automattic/wp-calypso/issues/28126.

[vindl]

Thanks @JavonDavis! Just a note that a couple of Gutenberg updates landed on WP.com since then - some previous issues might be fixed, and some new ones could've been introduced.

I'll change this one to mention one specific embed so it can be picked up for fixing. We can also start creating separate issues for the rest of the bugs mentioned in the audit.

[alisterscott]

As an aside, I think we can probably help verify embed support for upgrades by creating a few test posts that we can verify manually or with a visdiff. We'll get around to this later, unless @Automattic/flowpatrol would like to be ahead of us.

I've started on this here: https://gutembeds.wordpress.com/

I'm happy to add anyone to the site that would like to contribute

[JavonDavis]

I'm happy to add anyone to the site that would like to contribute

🙋‍♂️I already have URLs for each embed block so I'm happy to contribute

[alisterscott]

@JavonDavis sent you an invite - the URL to edit is https://wpcalypso.wordpress.com/gutenberg/page/gutembeds.wordpress.com/2 I'll take a break for a bit to avoid conflicts

[alisterscott]

Once we have full HTML block content for gutembeds.wordpress.com we could duplicate it on a Jetpack site also

[JavonDavis]

https://gutembeds.wordpress.com/ has all the embed blocks now and before I forget again @gwwar minor thing here but do you know if it's already in the backlog somewhere to change the text for the Polldaddy Embed block, should be Crowdsignal now right? If not I can go ahead and open an issue for that so it's tracked here.

[gwwar]

do you know if it's already in the backlog somewhere to change the text for the Polldaddy Embed block, should be Crowdsignal now right?

Thanks @JavonDavis It's a core issue: https://github.com/WordPress/gutenberg/issues/11517

[vindl]

I'm happy to add anyone to the site that would like to contribute.

Could you please invite me too? It will be useful for testing the updates to Calypso integration.

Thanks for creating this @JavonDavis and @alisterscott!

[alisterscott]

Could you please invite me too? It will be useful for testing the updates to Calypso integration.

Done

[alisterscott]

All the same embeds on a Jetpack site: https://gutenpackembeds.mystagingwebsite.com/ (I've added both @vindl and @JavonDavis to this new site also)

[lakellie]

Received a report of this error in Calypso from a user in #1718003-zen

[katiebethbrown]

Had another report from a user in #9531420-hc

[carladoria]

reported on 5909599-hc

[sjaved87]

another report here 9704371-hc

[msilbers]

this was reported in #9920682-hc but the user was happy to know that its just a messaging error and still works

[Aurorum]

Possible report here: https://en.forums.wordpress.com/topic/instagram-and-youtube-embeds-not-working/

[notnownikki]

This is also happening in wp-admin. The problem is that the render preview endpoint returns scripts in a different form to core - it's an object with script names as attributes and urls as values, rather than an array of values. The middleware used to convert this, but I think the render endpoint changed.

[notnownikki]

I've put a fix up for review for wpcom which fixes this. I'll update here once it's been deployed.

We should note, there's also a rendering issue for pinterest, but that's also present in gutenberg core.

[notnownikki]

Instagram fix has been deployed to wpcom.

[gugaalves]

another report at 9800033-hc, tested on my personal site and it was not working too.

[xpurichan]

@notnownikki I got a new report of the Instagram embed error today, and I also replicated the error on my test site today. It seems like the problem may still be present if the fix has already been deployed.

6611811-hc

[notnownikki]

@xpurichan can you tell me the URL you were trying to embed, and how you accessed the editor? I'll test on my personal site.

[xpurichan]

Yes definitely! I tested this URL on two simple sites:
https://www.instagram.com/p/BnOcZz1l_T9/

As expected the block still works despite the error (which was "Sorry, we could not embed that content."), and we can also embed the content without the error using the shortcode block and IG shortcode.

[notnownikki]

I think it's because Calypso is using the oembed proxy endpoint, and the WP5 release missed out code that let shortcode embeds work through that. There's a core ticket open to address that at https://core.trac.wordpress.org/ticket/45447

@aduth would you be able to suggest the best way forward here? I think that in wp-admin we load the gutenberg-wpcom middleware which intercepts the oembed proxy request and changes it to use the embeds/render endpoint instead. I'm not sure that the middleware is getting loaded in Calypso, but adding the classic embed code back into the oembed proxy endpoint might be a acceptable workaround until core get it too?

[aduth]

I'm not too familiar with the specific implementation in Calypso, but if it is using the WordPress REST API endpoints, maybe WordPress.com (or Jetpack, since it's where the the shortcodes are implemented) could add filtering much like what's left in the Gutenberg plugin to support this behavior.

but adding the classic embed code back into the oembed proxy endpoint might be a acceptable workaround until core get it too?

If the question is whether the WP.com API endpoint should incorporate the same behavior to try a URL as a classic embed, it seems reasonable. I'm not sure it's even necessary to be a temporary measure. To me, it sounds like reasonable feature parity to bring to the WP.com equivalent endpoint to core's oEmbed proxy endpoint.

[notnownikki]

If the question is whether the WP.com API endpoint should incorporate the same behavior to try a URL as a classic embed, it seems reasonable.

Sorry, I meant the other way around. The WP.com endpoint already tries a URL as a classic embed, but the core oembed proxy endpoint doesn't, and won't until https://core.trac.wordpress.org/ticket/45447 lands. This isn't a problem in wp-admin, as middleware directs the request to the WP.com endpoint, but for some reason that doesn't happen in Calypso, and we get the current core implementation of oembed/proxy which doesn't yet do the classic embeds.

@gwwar my first thought it we should have whatever middleware Calypso uses translate the embed proxy request in the same way as happens when using wp-admin, and once core has the classic embed support, we can switch back. How does that sound?

[dcoleonline]

Reported in 10562113-hc

[Robertght]

Another case #1819712-zen

[kwight]

Noting that this issue solves itself once we start iframing the block editor from wp-admin, hopefully within the next few weeks (I verified again today that embeds work great with this approach).

[Aurorum]

Related (or duplicate?): #31155

[arunsathiya]

@kwight I tested again, and based on your last comment, I am able to see the Instagram embeds just fine on Gutenframe. Hopefully this issue can be closed now, or are there something else to tackle before closing this one?

[vindl]

After switching to Gutenframe this issue is no longer reproducible so I'm going to close it.

[xpurichan]

I would like to share that this is still definitely an ongoing issue for AT and Jetpack sites. Should this issue be kept open for them, or a new one opened? Or are we just leaving it broken until those begin to use Gutenframe eventually?

[vindl]

I would like to share that this is still definitely an ongoing issue for AT and Jetpack sites. Should this issue be kept open for them, or a new one opened? Or are we just leaving it broken until those begin to use Gutenframe eventually?

Thanks for confirming @xpurichan. I'll update the description and labels in order to note that it now affects only Jetpack and Atomic sites.

I don't think switching these to Gutenframe will help, because this is probably not an integration issue but one that can be reproduced in wp-admin too.

[codebykat]

This is working fine for me on a Jetpack site inside Gutenframe (WordPress.com/post URL), although it does have a funny gray background:

As per @kwight it does seem to still fail on AT.

[kwight]

Updated the description to reflect the latest error:

[wynwin]

Also reported in 11123364-hc

[kwight]

Took another look at this today; noting a few things.

• The embed works as expected for both dotcom and Jetpack sites. Currently, Gutenberg on a Jetpack or Atomic site is only accessible through its own wp-admin, not Calypso (although hopefully the iframe approach being used for dotcom sites will extend to JP+AT sites in the coming months).
• It is only the wp-admin block editor preview for Atomic (both v1 and v2) that is broken. However, the front-end embed displays properly.
• The error in the editor for an Atomic site is a 404: {"code":"oembed_invalid_url","message":"Not Found","data":{"status":404}}
• The URL format for the fail (Atomic) and success (Jetpack) are the same: /wp-json/oembed/1.0/proxy?url=https%3A%2F%2Fwww.instagram.com%2Fp%2FBpA0QJvBJ78&_locale=user
• There are two instances in core of oembed_invalid_url, with the second likely the one that's triggered in this situation: https://github.com/WordPress/WordPress/search?q=oembed_invalid_url&unscoped_q=oembed_invalid_url
• It looks like it's here in get_data that the failure is happening: https://github.com/WordPress/WordPress/blob/master/wp-includes/class-oembed.php#L345. I'm not able to step through/log an Atomic site; @obenland please ping me privately if this is something you can help me with.

[samiff]

Another in 12137301-hc

[dcoleonline]

reported in 11859029-hc but told the customer that the actual site would load the embed just fine

[deancroyal]

Reported in 13231003-hc, also told customer that embed would load on site (which it did).

[chickenn00dle]

Another case in 2082956-zen. Confirmed the post appears fine in the front end.

[vindl]

I'm no longer able to reproduce this on Atomic sites. I've tried reproducing both from wp-admin and WordPress.com block editor (Gutenframe), and it works fine in both cases.

It also works regardless of whether I just copy the text from the clipboard or manually insert the Instagram block.

There are two recent updates that might have contributed to resolving this:

• Gutenberg update to v6.1.1 on Atomic
• All Atomic sites have been moved to v2 (as of last night).

If the issue is still present on some site please check the Gutenberg plugin version on it first. Some sites might be running an older plugin version or with no plugin at all (just Gutenberg that comes with Core). In those cases, please work with the Atomic team to get this updated to v6.1.1 or newer.