Rss-bridge: Solved - Arabic encoding issue with facebook bridge + FB2bridge ?

[rmdes]

the best way to show what's going on is with a screenshot, currently my rss-bridge runs on my own server, i had no issue at all deploying it, i used this git repo to run in on a default LAMP stack. I'm very happy but...

when I scrapp a facebook page using either Facebook Bridge or FB2bridge I see this :

Does this comes from a missing dependency on my server ? where does it come from? of course any and all facebook pages return this kind of characters and I'm really not cool with it because since I subscribe to these page with my RSS reader, these arabic chars also get displayed on the feed reader.

any hint on how to solve this if it's limited to my instance is very much appreciated :)

edit :

si j'utilise un autre bridge pour obtenir un flux de cette même source, bien sûr ce problème est inexistant :

[teromene]

I think it comes from your server : https://bridge.suumitsu.eu/?action=display&bridge=FB2&u=medormag&format=Html doesn't have the issue.

[rmdes]

@teromene Damned...any idea on how to debug this ?

I'm not sure what can be missing since rss-bridge installed just fine without any missing requirement
so if it's my server (ubuntu 16 64bit) how can I start debugging this ?

It's really annoying to have these characters coming from nowhere...(or from somewhere in this case)

[teromene]

What encoding headers do you get when loading the page ?

[rmdes]

Hmm I'm not sure sure where or how to double check this...

[teromene]

If you are on Firefox, right-click, view page info -> general -> text encoding.

[rmdes]

Oww ok I thought this was a server side question, I'm using default encoding : UTF-8
edit: this is the output I see on Firefox, no matter the browser this issues comes when using any of the Facebook Bridge (hence me thinking this must something server-side?)

[teromene]

this should give you the encoding of the server.
Could you try to generate the JSON version of the bridge ?

[rmdes]

When I'm checking the encoding on the that after generating the html view for "medormag" facebook page it's still UTF-8

edit: but isn't that a normal encoding for a server ?
this really seem to be injected on the page, but why on the facebook bridges and not on any other bridges ?

[rmdes]

Should I change the RSS-Bridge server side to display using : iso-8859-1 instead of UTF-8 ?
and if so why am I being the only one with this ?

[rmdes]

I activated debug mode, but i'm not seeing anything particular : https://bridge.armada.digital/?action=display&bridge=Facebook&u=medormag&format=Html

[rmdes]

I changed the line 197 of the index.php page to <meta charset="iso-8859-1"> instead of default <meta charset="utf-8"> but the problem remains exactly the same. not sure what to explore more..

[teromene]

No, the default should be UTF-8.
Can you provide the output of the JSON version ?

[rmdes]

Json version of the same page : https://bridge.armada.digital/?action=display&bridge=Facebook&u=medormag&format=Json

edit: the json seems to return the arabic characters in some encoded form

"title": "\u202bM\u00e9dor - \u0627\u0644\u0635\u0641\u062d\u0629... | M\u00e9dor \u00b7 #fakenews d\u00e9barque en...", "author": "\u202bM\u00e9dor - \u0627\u0644\u0635\u0641\u062d\u0629 \u0627\u0644\u0631\u0626\u064a\u0633\u064a\u0629 | \u0641\u064a\u0633\u0628\u0648\u0643\u202c", "timestamp": "1493967600" },

[rmdes]

I just returned index.php to its original state but DEBUG still enabled if it can of any help..

[teromene]

Can you try running this ?
https://gist.github.com/teromene/8163ea72d484292730eeaa772f06c89d

[rmdes]

Not sure if this is what you meant : https://bridge.armada.digital/test.php
or do I have to run this on the terminal ?

[teromene]

It actually seems like a bug from the bridge : according to google translate, this means Facebook Welcome Page.
Is your web hosting located in an arabic speaking country ?

[rmdes]

nope in Nederlands actually (digitalocean)

[rmdes]

So just to double-check I got another RSS-bridge running also on digitalocean, also on ubuntu,
and get the exact same "bug" : https://rss.hwc.brussels/?action=display&bridge=Facebook&u=hwcbrussels&format=Html

Mystery !

[LogMANOriginal]

Your PHP settings might be incorrectly setup. You can check that by creating a file 'phpinfo.php' on your server and add this:

<?php echo phpinfo(); ?>

Then open the file in your web browser via http://<your server>/phpinfo.php. Look around for language settings. I'm not sure which one is important, but you can search for 'lang' and look for anything suspicious.

In my case everything is set to en-US,en

[rmdes]

Not fiding the same elements, but this is my php.ini content : https://paste.armada.digital/herifeyiwe.scala

[rmdes]

And here is my phpinfo page output : https://bridge.armada.digital/phpinfo.php

[LogMANOriginal]

Same as mine. Still leaves the question to why Facebook offers you pages in Arabic language...

The language is not specified in the headers so it should default to English or whatever language is appropriate for the location of the requester.

Comparing my results to https://bridge.suumitsu.eu and your server at https://bridge.armada.digital clearly shows that contents depend on the location of the server:

• My local server shows German results
• Feeds from bridge.suumitsu.eu are in French
• Yours are in Arabic

Maybe Facebook doesn't know Netherlands is closer to France than Iran? :rofl:

But seriously it should be possible to force languages by providing content languages in the request header. However in order to still deliver different languages it should make use of languages specified by the host. Maybe the PHP environment variables can be utilized to do that.

[rmdes]

It's crazy right ? the only thing I can think of is : I use https://cloudron.io if that make any difference, it shouldn't I mean this is the only piece of code giving me this kind of trouble, the reason I mention cloudron is because both HWC and Armada run on Cloudron and both have the issue with the facebook bridge...???!!!??. I have asked their dev to have a look when they have time... :/

Not sure which PHP environment variables to modify but I'll do some research

if it was server side, why only appear for facebook bridge? does it mean facebook is sending MORE information that RSS-Bridge just interpret? if so would there be a way to exclude those characters at the server side level ?

[LogMANOriginal]

I made some adjustments to force contents with languages specified in the PHP environment variables. It seems to work for me, though I would like you to test it first. Please replace your FacebookBridge with this one:

https://gist.github.com/LogMANOriginal/14b50e2aa00a6b6029010ffd6e776761

I changed a few lines from 113 to 129. Technically it should forward the language specified by your browser, though I'm not sure about that at all. I'll have to read up on environment variables first.

[LogMANOriginal]

It does in fact forward your browser language to the bridge and to Facebook. This opens up a lot of new possible features :trophy:

'HTTP_ACCEPT_LANGUAGE'
Contents of the Accept-Language: header from the current request, if there is one. Example: 'en'.

-- PHP Manual

[rmdes]

That made my day :))) It's fixed, going to do the same on my second instance but I think it's definitely gone : https://bridge.armada.digital/?action=display&bridge=Facebook&u=medormag&format=Html

second instance I have with same bug : https://rss.hwc.brussels/?action=display&bridge=Facebook&u=medormag&format=Html

Thanks soo much !