Not scraping youtube video properly when you use the video option in a post.
Create a new post, select the video option, paste a youtube URL into it.
Actual Result:
Just prints link and a download symbol
Expected Result:
Expect it to frame a picture from the video and make it a hotlink.
This worked earlier today, suspect some change Screwtube has made.
Friendica Version:
This is Friendica, version 2020.12-dev that is running at the web location https://friendica.eskimo.com. The database version is 1382/1382, the post update version is 1349/1349.
Friendica Source:
Github
PHP version:
PHP 7.4.13 (cli) (built: Nov 28 2020 06:24:59) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Zend OPcache v7.4.13, Copyright (c), by Zend Technologies
SQL version:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 331171
Server version: 10.5.8-MariaDB-1:10.5.8+maria~focal-log mariadb.org binary distribution
Thank you for the report, it's possible Youtube changed their embed policy.
Well it appears to be a friendica issue, or maybe a Firefox issue, but it turns that it's not lifting images from other sites with the link option either, but it shows it in the HTML in the preview so I'm not sure if it's friendica or firefox or what that is going wrong. Here are a couple of screen shots that hopefully will help, one is a website URL showing how the website displays in firefox, and the other is a preview of the link in friendica.


Are you using any kind of third-party content blockers? I'm getting this kind of result with PrivacyBadger since for a while it's been blocking all third-party domains by default.
I have some ad blockers but haven't changed those, also, friendica
displays the image of a post I made 12 hours ago but not a current one.
Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
Knowledgeable human assistance, not telephone trees or script readers.
See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.
On Thu, 17 Dec 2020, Hypolite Petovan wrote:
Date: Thu, 17 Dec 2020 09:32:21 -0800
From: Hypolite Petovan notifications@github.com
Reply-To: friendica/friendica
reply@reply.github.com
To: friendica/friendica friendica@noreply.github.com
Cc: Robert Dinse nanook@eskimo.com, Author author@noreply.github.com
Subject: Re: [friendica/friendica] Youtube no longer skimming video properly
just showing a link and download arrow. (#9666)Are you using any kind of third-party content blockers? I'm getting this kind of result with PrivacyBadger since for a while it's been blocking all third-party domains by default.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/friendica/friendica/issues/9666#issuecomment-747586265
Thank you for the added details.
More details, it still does scrape from Bitchute properly but not from Youtube and a number of websites. I tried deleting various
plugins, did not help.
Still more details, I tried from a different computer with no plugins installed, still the same. But both had recently upgraded Firefox so thought ok maybe firefox broke something, so I tried chrome, also does not work.
I can confirm that the youtube player doesn't show up anymore. I tried sharing a YT video via [video] and [youtube] tags. Both didn't work. Preview is empty. Post is empty.
https://libranet.de/display/0b6b25a8-125f-dbb0-8e5b-1fe073281343
Item source:
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:thr="http://purl.org/syndication/thread/1.0" xmlns:at="http://purl.org/atompub/tombstones/1.0" xmlns:media="http://purl.org/syndication/atommedia" xmlns:dfrn="http://purl.org/macgirvin/dfrn/1.0" xmlns:activity="http://activitystrea.ms/spec/1.0/" xmlns:georss="http://www.georss.org/georss" xmlns:poco="http://portablecontacts.net/spec/1.0" xmlns:ostatus="http://ostatus.org/schema/1.0" xmlns:statusnet="http://status.net/schema/api/1/">
<id>https://libranet.de/profile/alfred</id>
<title>Steffen K9 🐰</title>
<generator uri="https://friendi.ca" version="2020.12-rc-1383">Friendica</generator>
<link rel="license" href="http://creativecommons.org/licenses/by/3.0/"/>
<link rel="alternate" type="text/html" href="https://libranet.de/profile/alfred"/>
<dfrn:account_type>0</dfrn:account_type>
<updated>2020-12-17T19:26:41Z</updated>
<dfrn:owner>
<name dfrn:updated="2019-03-08T02:40:04Z">Steffen K9 🐰</name>
<uri dfrn:updated="2019-03-08T02:40:04Z">https://libranet.de/profile/alfred</uri>
<dfrn:handle dfrn:updated="2019-03-08T02:40:04Z">[email protected]</dfrn:handle>
<link rel="photo" type="image/jpeg" media:width="300" media:height="300" href="https://libranet.de/photo/21461315195de41929762c2323794558-4.jpg?ts=1575226148" dfrn:updated="2019-12-01T19:49:08Z"/>
<link rel="avatar" type="image/jpeg" media:width="300" media:height="300" href="https://libranet.de/photo/21461315195de41929762c2323794558-4.jpg?ts=1575226148" dfrn:updated="2019-12-01T19:49:08Z"/>
<dfrn:birthday>2021-01-25T23:00:00Z</dfrn:birthday>
<poco:displayName>Steffen K9 🐰</poco:displayName>
<poco:updated>2019-03-08T02:40:04Z</poco:updated>
<poco:note>Operator of LIBRANET.de, venera.social and misskey.de, #Fediverse enthusiast, #Coffee addict. I like #Music #Cycling #Relaxing #ScienceFiction #Linux #Nextcloud #Friendica and you! :-)</poco:note>
<poco:preferredUsername>alfred</poco:preferredUsername>
<poco:utcOffset>+01:00</poco:utcOffset>
<poco:tags>friendica</poco:tags>
<poco:tags>linux</poco:tags>
<poco:tags>fahrrad</poco:tags>
<poco:tags>kaffee</poco:tags>
<poco:tags>musik</poco:tags>
<poco:tags>vinyl</poco:tags>
<poco:tags>sci-fi</poco:tags>
<poco:ims>
<poco:type>xmpp</poco:type>
<poco:value>[email protected]</poco:value>
<poco:primary>true</poco:primary>
</poco:ims>
<poco:address>
<poco:formatted>Halle an der Saale, Germany</poco:formatted>
<poco:locality>Halle an der Saale</poco:locality>
<poco:country>Germany</poco:country>
</poco:address>
</dfrn:owner>
<entry>
<author>
<name>Steffen K9 🐰</name>
<uri>https://libranet.de/profile/alfred</uri>
<dfrn:handle>[email protected]</dfrn:handle>
<link rel="photo" type="image/jpeg" media:width="80" media:height="80" href="https://libranet.de/photo/21461315195de41929762c2323794558-4.jpg?ts=1575226148"/>
<link rel="avatar" type="image/jpeg" media:width="80" media:height="80" href="https://libranet.de/photo/21461315195de41929762c2323794558-4.jpg?ts=1575226148"/>
</author>
<dfrn:owner>
<name>Steffen K9 🐰</name>
<uri>https://libranet.de/profile/alfred</uri>
<dfrn:handle>[email protected]</dfrn:handle>
<link rel="photo" type="image/jpeg" media:width="80" media:height="80" href="https://libranet.de/photo/21461315195de41929762c2323794558-4.jpg?ts=1575226148"/>
<link rel="avatar" type="image/jpeg" media:width="80" media:height="80" href="https://libranet.de/photo/21461315195de41929762c2323794558-4.jpg?ts=1575226148"/>
</dfrn:owner>
<ostatus:conversation href="https://libranet.de/display/0b6b25a8-125f-dbb0-8e5b-1fe073281343" ref="https://libranet.de/display/0b6b25a8-125f-dbb0-8e5b-1fe073281343">https://libranet.de/display/0b6b25a8-125f-dbb0-8e5b-1fe073281343</ostatus:conversation>
<id>https://libranet.de/objects/0b6b25a8-125f-dbb0-8e5b-1fe073281343</id>
<title/>
<published>2020-12-17T19:25:02Z</published>
<updated>2020-12-17T19:25:02Z</updated>
<dfrn:env>W3lvdXR1YmVdaHR0cHM6Ly95b3V0dS5iZS9CTGZlMy1xS2dna1sveW91dHViZV0</dfrn:env>
<content type="text">[youtube]https://youtu.be/BLfe3-qKggk[/youtube]</content>
<link rel="alternate" type="text/html" href="https://libranet.de/display/0b6b25a8-125f-dbb0-8e5b-1fe073281343"/>
<dfrn:comment-allow>1</dfrn:comment-allow>
<dfrn:location>Halle</dfrn:location>
<statusnet:notice_info local_id="101906809" source="libranet.de"/>
<dfrn:diaspora_guid>0b6b25a8-125f-dbb0-8e5b-1fe073281343</dfrn:diaspora_guid>
<activity:verb>http://activitystrea.ms/schema/1.0/post</activity:verb>
<activity:object-type>http://activitystrea.ms/schema/1.0/note</activity:object-type>
</entry>
</feed>
The long YT URL doesn't work either.
[video]https://youtube.com/watch?v=BLfe3-qKggk[/video]
Edit: also switching 'embedding rich content' on and off doesn't help. Adding the YT domains to the exception field doesn't help either.
When I look at this post https://libranet.de/display/0b6b25a8-125f-dbb0-8e5b-1fe073281343 on other systems like Diaspora, Mastodon or Misskey the post looks right. URL and YT player are displayed correctly.
My uneducated guess: it's a bug in Friendica.
@AlfredSK You're referring to another known issue. This issue is about the link preview of a Youtube video/article, not about the embed itself (via the relevance BBCodes).
Oops. I apparently didn't understand what this issue is about. 🤔 I'm sorry.
Just curious is this dead? It is still not scraping youtube video for me.
Have you tried just pasting the YouTube link in the post compose field?
@MrPetovan just tried it but the embedded video doesn't show up.
While I can see the embedded video at https://libranet.de/display/0b6b25a8-125f-fb2e-3759-fa2761437272 there is only a youtube link when visiting the same post in my network stream
Hmm, interesting. Thanks for the added information, and welcome back to the Friendica GitHub pages!
I applied the recent updates and still video scraping not working for youtube.

There is a video shot so you can see how a youtube link currently looks. This is with the most current Friendica updates.
Interesting, I have no idea where the downward arrow comes from, is it a vier-specific display?
I tried changing to smoothy and still same down arrow displays.
Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
Knowledgeable human assistance, not telephone trees or script readers.
See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.
On Wed, 13 Jan 2021, Hypolite Petovan wrote:
Date: Wed, 13 Jan 2021 18:02:51 -0800
From: Hypolite Petovan notifications@github.com
Reply-To: friendica/friendica
reply@reply.github.com
To: friendica/friendica friendica@noreply.github.com
Cc: Robert Dinse nanook@eskimo.com, Author author@noreply.github.com
Subject: Re: [friendica/friendica] Youtube no longer skimming video properly
just showing a link and download arrow. (#9666)Interesting, I have no idea where the downward arrow comes from, is it a vier-specific display?
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/friendica/friendica/issues/9666#issuecomment-759875310
I have similar problems. New install Version Friendica 'Red Hot Poker' 2021.01 - 1384
[video]https://gorf.tube/videos/watch/3fc914e3-c584-45ab-8079-79017c692a82[/video]
The result, just showing a link to the video, no player or thumbnail.
When posting a link to Friendica node
https://friendica.lt/display/63edefc7-1360-030f-3a0b-4a4108624718
On Facebook preview looks OK

But the Actual post looks like that

@4nanook have you checked if that download button is added by some browser add-on?
@mikauskas umm. Thanks for including that domain in your comment.^^
@MrPetovan just tried it but the embedded video doesn't show up.
While I can see the embedded video at https://libranet.de/display/0b6b25a8-125f-fb2e-3759-fa2761437272 there is only a youtube link when visiting the same post in my network stream
On what version libranet.de works? Perhaps it is possible to check what changes since then could ruin new Friendica versions?
@MrPetovan @AlfredSK @annando
Version 2020.09-01 player works with code
[video]https://gorf.tube/videos/watch/ab4dcae4-5499-45f8-bd51-56d0d0da56e5[/video]
https://old.friendica.lt/display/04a6b8f1-2660-0725-98f2-3cf295167749
Whatever happened it was between Version 2020.09-01 and version 2021.01 - 1384
Problem first reported on 17 Dec 2020, but it is still not on Known Issues list https://friendi.ca/2021/01/04/friendica-2021-01-released/ Many nodes are still on older versions. Perhaps people should be warned?
Have downgraded server to 2020.09-01. Now "Show more dynamic" is not working properly.
And unable to follow a user:
`Internal Server Error
'https://domain.com/dfrn_request/kedas&addr=6d696b6175736b6173406c6169737661692e6c74 is not a relative path, please use System::externalRedirectTo
Any updates on fixing the video problem?

If you downgrade the core, you have to downgrade the addons as well. No update on this issue.
If you downgrade the core, you have to downgrade the addons as well. No update on this issue.
Both core and addons are from the same package and there is only "Showmore Dynamic" v 1.0 anyway. Have disabled the addon. Can live without it for a while. Hopefully will start working itself after an upgrade.
After downgrade previously created posts with videos and preview pictures to links are displayed nicely again.
Apparently, the user was still not confirmed. The false alarm about unable to follow.
Apparently, on Version 2020.09-01 follow button is missing, people cant follow each other.
On Version 2020.07-01 controls are missing below the share window.
Downgrading to version 2020.03 ends up with database errors.
I'm back to version 2021.01 and really need this issue to be fixed. How can I help to get it done?
This issue is expensive to look into because it depends on two nodes having different node settings. So reliably reproducing this issue requires an elaborate setup, and as such it will be less likely to be picked up, especially since the issue doesn't affect all nodes depending on their settings as well.
Low overall impact + high troubleshooting cost isn't a great recipe for any issue to be solved, and unfortunately there's little anyone can do about it.
What can I do to help? Is it known what specific settings affect this? If not, perhaps we could ask a site that has the current development version working to upload their settings and compare to mine?
This issue is expensive to look into because it depends on two nodes having different node settings. So reliably reproducing this issue requires an elaborate setup, and as such it will be less likely to be picked up, especially since the issue doesn't affect all nodes depending on their settings as well.
Low overall impact + high troubleshooting cost isn't a great recipe for any issue to be solved, and unfortunately there's little anyone can do about it.
It is not really expensive. Just bought a hosting plan for a year with a free domain for just $18 https://friendica.online/
Can make you admin and provide ssh access if you need it.
I'm sorry, I wasn't talking about money, I was talking about time since this project runs with volunteers.
I'm sorry, I wasn't talking about money, I was talking about time since this project runs with volunteers.
No bother, I needed another hosting account to stop messing with experimental software on production one anyway.
The issue is not so little as it might look on the surface. Unfortunately, it is not just two nodes. The same thing here:
https://friendica.feneas.org/display/76158462-8060-202e-a133-c46618058322

https://friendica.online/display/ffc05492-2760-215c-8831-b0d499617892

https://laisvai.lt/display/2e25ce5c-1860-200a-5dae-9e8541976832

https://squeet.me/display/962c3e10-1260-2161-d3d1-f66343131482

How many more servers I have to test for the problem to be taken seriously?
How big an impact you are waiting to become? The problem was not present in older versions, many servers are still on older versions and not many people noticed or bother enough to report. Those using Frio theme hadn't notice the problem because there is no video button to generate, you have to manually start entering and the server will suggest [video][/video]
It isn't a question of seriousness either, all issues are serious, but again, our time is limited and we have to pick our battles.
It isn't a question of seriousness either, all issues are serious, but again, our time is limited and we have to pick our battles.
Any ideas on what files to look at? I think just a comparison of code between V 2020.07-1 and 2021.01 can help what had ruined videos.
There has been some significant changes in the BBCode to HTML conversion function in src/Content/Text/BBCode.php for example.
You can quickly test your changes on your server by using the /babel page.
There has been some significant changes in the BBCode to HTML conversion function in
src/Content/Text/BBCode.phpfor example.You can quickly test your changes on your server by using the
/babelpage.
Thanks found it, line 1932:
// Backward compatibility, [iframe] support has been removed in version 2020.12

So, who removed iframe compatibility, why, and what they suggest to do instead?
It's me 😇.
Although I see a potential cause for the current behavior, we are replacing remaining [video] tags after the explicit video files replacement (based on the extension) by a link, which prevent the next [video] tag replacement to try OEmbed which would potentially enable the display of a player.
That sounds like an acceptable solution.
It's me .
Although I see a potential cause for the current behavior, we are replacing remaining
[video]tags after the explicit video files replacement (based on the extension) by a link, which prevent the next[video]tag replacement to try OEmbed which would potentially enable the display of a player.
Weird approach. So, you removed support of the working function (iframe is still standard way of embedding if embed button pressed while getting share link on Youtube) in favor on might be working OEmbed in some very far future? Wasn't possible to do these experiments on a separate fork and merge with the main one when/if OEmbed is proven to be working better? I hope it happened by accident, not the standard way of maintaining the project? I am just starting to build of new social network and really need to know how serious you guys are about this project. Mastodon and Diaspora working much better and I am seriously considering switching platforms while the community is not big.
Hi, this was an accident because I do not use the [video] tag myself and we don't have a complete suite of automated tests to catch regressions. We do heavily rely on users for feedback because it is a community-driven project, there's no central organization and few people are regularly working on it.
If you are uncomfortable with this, it wouldn't be a bad idea for you to switch to a more popular project with more maintainers and a donation incentive system.
@MrPetovan what do you do not use instead of the [video] tag?
I can help with testing in the future.
I just don't share videos on my personal Friendica account, so I wouldn't have been able to spot this regression through regular Friendica use.
Thank you for your offer, you already provided valuable information in this issue, just please be patient about the time to a fix.
@MrPetovan
Another symptom of the same problem - glitch with attachments.
v2021.01
https://friendica.online/display/ffc05492-1160-22f9-94c4-d7f632080230

v2020.09.1
https://fv91.friendica.online/display/bc07ef10-1360-22f9-0aab-685095211400

v2021.1
BBCode::convert (raw HTML)
<div class="oembed rich"><blockquote class="wp-embedded-content"><a href="https://aapsonline.org/can-covid-19-revolutionize-the-treatment-of-viral-diseases/">‘Can COVID-19 Revolutionize the Treatment of Viral Diseases?’</a></blockquote><script type="text/javascript"><!--//--><![CDATA[//><!-- /*! This file is auto-generated */ !function(c,d){"use strict";var e=!1,n=!1;if(d.querySelector)if(c.addEventListener)e=!0;if(c.wp=c.wp||{},!c.wp.receiveEmbedMessage)if(c.wp.receiveEmbedMessage=function(e){var t=e.data;if(t)if(t.secret||t.message||t.value)if(!/[^a-zA-Z0-9]/.test(t.secret)){for(var r,a,i,s=d.querySelectorAll('iframe[data-secret="'+t.secret+'"]'),n=d.querySelectorAll('blockquote[data-secret="'+t.secret+'"]'),o=0;o<n.length;o++)n[o].style.display="none";for(o=0;o<s.length;o++)if(r=s[o],e.source===r.contentWindow){if(r.removeAttribute("style"),"height"===t.message){if(1e3<(i=parseInt(t.value,10)))i=1e3;else if(~~i<200)i=200;r.height=i}if("link"===t.message)if(a=d.createElement("a"),i=d.createElement("a"),a.href=r.getAttribute("src"),i.href=t.value,i.host===a.host)if(d.activeElement===r)c.top.location.href=t.value}}},e)c.addEventListener("message",c.wp.receiveEmbedMessage,!1),d.addEventListener("DOMContentLoaded",t,!1),c.addEventListener("load",t,!1);function t(){if(!n){n=!0;for(var e,t,r=-1!==navigator.appVersion.indexOf("MSIE 10"),a=!!navigator.userAgent.match(/Trident.*rv:11\./),i=d.querySelectorAll("iframe.wp-embedded-content"),s=0;s<i.length;s++){if(!(e=i[s]).getAttribute("data-secret"))t=Math.random().toString(36).substr(2,10),e.src+="#?secret="+t,e.setAttribute("data-secret",t);if(r||a)(t=e.cloneNode(!0)).removeAttribute("security"),e.parentNode.replaceChild(t,e)}}}}(window,document);//--><!]]></script><iframe sandbox="allow-scripts" security="restricted" src="https://aapsonline.org/can-covid-19-revolutionize-the-treatment-of-viral-diseases/embed/" width="425" height="240" title="“‘Can COVID-19 Revolutionize the Treatment of Viral Diseases?’” — AAPS | Association of American Physicians and Surgeons" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" class="wp-embedded-content"></iframe></div>
v2020.9.1
BBCode::convert (raw HTML)
<div class="oembed rich"><blockquote class="wp-embedded-content"><a href="https://aapsonline.org/can-covid-19-revolutionize-the-treatment-of-viral-diseases/">‘Can COVID-19 Revolutionize the Treatment of Viral Diseases?’</a></blockquote><iframe width="425" height="240" title="“‘Can COVID-19 Revolutionize the Treatment of Viral Diseases?’” — AAPS | Association of American Physicians and Surgeons" class="wp-embedded-content"></iframe></div>
Hi, this was an accident because I do not use the
[video]tag myself and we don't have a complete suite of automated tests to catch regressions. We do heavily rely on users for feedback because it is a community-driven project, there's no central organization and few people are regularly working on it.If you are uncomfortable with this, it wouldn't be a bad idea for you to switch to a more popular project with more maintainers and a donation incentive system.
@MrPetovan so there is no Friendica donation system at all?
Nope, there's no constituted association, so it comes down to individual developers, and none of us are asking for donations that I know of. However, if you're on a node hosted by someone else, I'd suggest to find out whether they are accepting donations to cover hosting costs.
@MrPetovan Friendica is overdue to get a professional product. Facebook is misbehaving badly these days. At beginning of January Telegram had registered 25 million new users in 3 days. Many of them could choose Freindica if it was a more mature product sometime before because Telegram is not even a social network (users don't have a timeline). We both know that the hosting cost is nothing in comparison with developers, testers, and sysadmins labor costs. And covering hosting costs wouldn't make the system much better.
I don't disagree with you on principle, but there is a cliff between the currently volunteer-run project and a professional product. A financial but also skill cliff that we won't be able to cross anytime soon. So I tend not to dwell too much on it and keep working at my pace on it.
All we need is infinite time days.