Magento-lts: Make OpenMage detectable by Webcrawlers

[Flyingmana]

There are numbers about usage of Magento1 and Magento2
https://twitter.com/gwillem/status/1232603585130778624

What could be useful to see, how many do migrate to OpenMage in comparison how many move to other platforms. Having this number would be helpful to be recognized as serious platform.

• I contacted https://builtwith.com/ via their contact formular
• I created an Issue in https://github.com/AliasIO/wappalyzer

I wonder if we should add some kind of hint, to make OpenMage easier detectable/differentiable.
Current detection approaches seem to go mostly against the html of the loaded page, where we are still mostly the same as the Original

[colinmollenhour]

I think changing the "frontend" cookie to something else like "ecommerce" or "om" (for OpenMage) would be a way that would

• Make OpenMage LTS easily identifiable
• Likely break Magento 1 detection (e.g. this is one of Wappalyzer's main methods)
• Not break anyone's code since the session handling is an internal thing

We could easily add some code to migrate the user's "frontend" cookie to the new one so that no sessions are lost upon upgrade.

[colinmollenhour]

I added pull #991 as a proof of concept.

[tmotyl]

we can also add more hints, like meta tag to default templates of frontend and backend

<meta name="generator" content="OpenMage MageLTS" />

[AlterWeb]

Thinking out of a security perspective isn't it better to not give away what system and version you use?

[tmotyl]

@AlterWeb for professional security scanners you can't really hide what system you use (they can analyze e.g. paths to images, or how html is structured, cookies etc), so if you're not exposing exact version like 1.9.4.4 its not a security risk.

[fballiano]

we can also add more hints, like meta tag to default templates of frontend and backend

<meta name="generator" content="OpenMage MageLTS" />

I wouldn't add more HTML, altho it's just a single tag

[trabulium]

Just a heads up for others. This change caused us a lot of grief debugging Amasty FPC -

the following changes were required:
app/code/local/Amasty/Fpc/Model/Fpc/Front.php: protected $_sessionName = 'om_frontend';
app/code/local/Amasty/Fpc/Model/Fpc.php: 'session' => $_COOKIE['om_frontend']
app/code/local/Amasty/Fpc/Model/Fpc.php: 'session' => isset($_COOKIE['om_frontend']) ? $_COOKIE['om_frontend'] : ''
app/etc/amfpc.xml: om_frontend

[seansan]

@AlterWeb for professional security scanners you can't really hide what system you use (they can analyze e.g. paths to images, or how html is structured, cookies etc), so if you're not exposing exact version like 1.9.4.4 its not a security risk.

This reply kind of defies why you would change the cookie then.

[seansan]

Just a heads up for others. This change caused us a lot of grief debugging Amasty FPC -

the following changes were required:

app/code/local/Amasty/Fpc/Model/Fpc/Front.php: protected $_sessionName = 'om_frontend';

app/code/local/Amasty/Fpc/Model/Fpc.php: 'session' => $_COOKIE['om_frontend']

app/code/local/Amasty/Fpc/Model/Fpc.php: 'session' => isset($_COOKIE['om_frontend']) ? $_COOKIE['om_frontend'] : ''

app/etc/amfpc.xml: om_frontend

Yeah I must admit we too had many broken processes. As Collin says yes it's internal but apparently too much relies on it.

I think it should be reverted. It doesn't really add anything.

Also if you really want it? Why not make it an option in settings. Then at least people can choose not to use it.

Now it's enforced in 20.x amd this just does not make sense

[colinmollenhour]

Thanks for the report. Sorry to hear about the troubles this update caused. However, I think this is an important change and is worth some effort to keep it. If Amasty FPC is the only module broken then perhaps we can request them to create an OpenMage compatible version? Maybe there are other modules broken as well but maybe not and if only one module of thousands is broken and the fix is changing four lines then I think that is pretty reasonable.

We need to be treating OpenMage as a separate product and thinking of it that way and causing the world to treat it that way as well. I'm sure I'm not only speaking for myself when I say I know merchants that have received emails from their credit card processors saying effectively "We detected that you are running Magento 1 which is no longer supported. You need to migrate to a new platform." - It is much easier to explain that OpenMage is not Magento 1 when they are not detected as one and the same by bots.

We don't want OpenMage forever plagued by being detected as Magento. It's really not meant to improve security at all.

[seansan]

Hi! and thanks for the discussion

If this is the case (we want OM to be treated as a "different" product)
then I understand it a bit more

However I doubt a "cookie name" change drives this ...

Btw is is Amasty module + we use a iphone app that mimics Magento sessions
from Highstreet: this is also broken (2nd)

Again: would it be sso bad to make this a config option? Then everybody is
happy.

thanks, Sean

On Wed, Nov 11, 2020 at 10:36 PM Colin Mollenhour [e-mail]
wrote:

Thanks for the report. Sorry to hear about the troubles this update
caused. However, I think this is an important change and is worth some
effort to keep it. If Amasty PFC is the only module broken then perhaps we
can request them to create an OpenMage compatible version? Maybe there are
other modules broken as well but maybe not and if only one module of
thousands is broken and the fix is changing four lines then I think that is
pretty reasonable.

We need to be treating OpenMage as a separate product and thinking of it
that way and causing the world to treat it that way as well. I'm sure I'm
not only speaking for myself when I say I know merchants that have received
emails from their credit card processors saying effectively "We detected
that you are running Magento 1 which is no longer supported. You need to
migrate to a new platform." - It is much easier to explain that OpenMage is
not Magento 1 when they are not detected as one and the same by bots.

We don't want OpenMage forever plagued by being detected as Magento. It's
really not meant to improve security at all.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/OpenMage/magento-lts/issues/990#issuecomment-725674726,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE7I2YMW5FLE5R4XQ55C3TSPL7XNANCNFSM4NFO3KJQ
.

[colinmollenhour]

Perhaps in hindsight this should have only been merged into the 20.0 branch and not the 19.4 branch... Reverting the patch would break existing sessions although that should be easy to do if you're tracking the code with a private git repo:

git revert 51e0abdd79f9a33a66ff9bef488863cb1d3438e5

I'm not sure yet that this warrants yet another config option when it is easy to revert the code change in a private repo but if you're using Composer I suppose it is more difficult. If you submit a PR I'm sure it will get a fair consideration.

Another point is that third-party code should not use the string 'frontend' but instead the proper constant: Mage_Core_Controller_Front_Action::SESSION_NAMESPACE.

[seansan]

See my comment here

We are starting a discussion on programming semantics ... and I dont mean it bad. As I see it

• request comes in: make openmage detectable by webcrawlers
• judgment: prio very low

• quick win to solve it: add a meta tag or add a body class or something

• but changing the cookie name? that one I dont follow ... why a change so to the core of M1? In addition there are now multiple reports of broken elements - OK maybe they should have built it otherwise - but this is the reality

It should not exist in either of the 2 repo's : is my opinion

@colinmollenhour dont want to offend you .. just think we got a little ahead of ourselves here if you look at the original intent, that was not even a bug but a feature req. I do hope you understand.

https://github.com/OpenMage/magento-lts/commit/51e0abdd79f9a33a66ff9bef488863cb1d3438e5#commitcomment-44147035

[tmotyl]

Hi
I agree with colin that we should stear away from being recognized as magento, and the code should use constant and not hardcoded string.

Im ok with reverting this in 1.9 but at last in 20. It should stay.

Also a possible solution is to add a note about it to the „migrate from Magento to Openmage page”.

[ADDISON74]

Such a decision may put this project in a negative light. When it was found out that Magento 1 has continuity many were happy to support this project. Unfortunately, most extension manufacturers have stopped selling M1 modules and to offer support for them even to those sold already. When support is offered, the price paid is higher and higher, as if the developer wants to make you migrate to M2. Due to a change certain websites risk not working and then disappointment occurs. I know websites with around 18 extensions that are functioning without any issue for more than 4 years since M1 version 1.9.2.4 was released. My personal opinion is making this optional for the moment without a proper information for the developers or being sure a backward compatibility is achieved. If you want to keep the modification create a script for looking into a module files and replace the old with the new. We did the same thing with formkeys in custom templates.

[colinmollenhour]

The bots like builtwith.com do not care about the generator meta tag and if M1 didn't have one before then they aren't going to be using it now, so adding this is not going to have any effect, although it won't really do any harm either.
Instead, they use a combination of things like presence or absence or signatures of certain Javascript files or objects, common controller names, common path prefixes like "skin/frontend" etc.. Of these detection methods cookie names are a big one:
https://github.com/AliasIO/wappalyzer/blob/master/src/technologies.json#L8384

So this PR was meant to target that method specifically as it is probably still the least invasive thing to change (e.g. versus renaming VarienForm). Now we can work towards getting these detection sites to treat the "om_frontend" cookie as a strong signal that the site is running OpenMage versus Magento.

Apologies for the unexpected BC break and lack of thorough warning or explanation.. I could see an argument for reverting the change in 19.4, but still think it is a reasonable BC break for 20.0. For the 19.4 version though we could actually be causing more issues for people that have already adapted and now we go back again... So I'm on the fence. If someone offers a PR for 19.4 that reverts the change and detects the "om_frontend" cookie for shop owners that have already upgraded then I'd not be opposed to it.

Regarding extension shops, if they used the proper constant then nothing should be broken, so that is arguably just a bug on their end, but I get your point, they want zero maintenance headache or to just drop support altogether.. Not sure what to say here but that is up to them to decide. I'm a bit hesitant to bend too much to the will of extension shops and shop owners clinging on to buggy extensions or exceedingly old and insecure versions... I think we should focus on improving the core and there are going to be a few bugs or unexpected BC breaks along the way and it will require some minor maintenance from the extension shops. For example PHP 8 support. Otherwise if we are too afraid to change anything then we aren't really offering improvements, just a facade of long-term support.

[Flyingmana]

judgment: prio very low

I want to extend some things here, because my judgement here would be "very high", as beeing recognized as an independent Project which is actually used, it is needed to appear in the popular statistics.
There were several possible partners who wanted numbers of how many are using it. Others did not even care to respond, despite supporting OpenMage would only need as much as adding a Tag to their (Theme) marketplace and they could directly profit from our pure existence.

also let me remind you of our RFC area: https://github.com/OpenMage/rfcs
if you want to enforce certain general decision onto maintainers (like which kind of changes and BC breaks are appropriate in which context), then create and advocate an RFC there.
If you wonder, why we should bend to them: if the majority of the Community Votes for a certain direction, we would be very unwise to ignore this.

[seansan]

Maybe of interest
https://webmasters.stackexchange.com/questions/32023/how-to-hide-from-builtwith/32026#32026

Also: I see my own site on LTS is still recognized as M1 (with the cookie
name change)

On Sat, Nov 14, 2020 at 11:17 AM Daniel Fahlke [e-mail]
wrote:

judgment: prio very low

I want to extend some things here, because my judgement here would be "very
high", as beeing recognized as an independent Project which is actually
used, it is needed to appear in the popular statistics.
There were several possible partners who wanted numbers of how many are
using it. Others did not even care to respond, despite supporting OpenMage
would only need as much as adding a Tag to their (Theme) marketplace and
they could directly profit from our pure existence.

also let me remind you of our RFC area: https://github.com/OpenMage/rfcs
if you want to enforce certain general decision onto maintainers (like
which kind of changes and BC breaks are appropriate in which context), then
create and advocate an RFC there.
If you wonder, why we should bend to them: if the majority of the
Community Votes for a certain direction, we would be very unwise to ignore
this.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/OpenMage/magento-lts/issues/990#issuecomment-727181751,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE7I24JBP54W7G7WZVVL7DSPZKKVANCNFSM4NFO3KJQ
.

[colinmollenhour]

Just an FYI I started on a PR for wappalyzer but it didn't work as expected after a few trials so I have dropped it for the moment..

Regarding the impact of the change, I think that is important to establish so if anyone has any additional info please add it here. So far I am only aware of the following impact:

• Amasty FPC extenstion (@seansan and @trabulium)
• @seansan mentioned an iphone app (is this a private custom tool or public?)

Anything else?

Again, sorry for the unexpected trouble caused by this change. However if the fixes are simple and affect a relatively small number of merchants that can easily revert the changes privately then I think that is not a terrible resolution.

[seansan]

Thanks Colin - that's it I guess.
Ultimately it should be a choice I guess (without merchants having to do
git/revert code)
Let push it forward now.
nb. Im in contact /w buitlwith
nb2. Changes like this maybe nee a vote/some more feedback next time
thx Sean

On Mon, Nov 16, 2020 at 8:35 PM Colin Mollenhour [e-mail]
wrote:

Just an FYI I started on a PR for wappalyzer but it didn't work as
expected after a few trials so I have dropped it for the moment..

Regarding the impact of the change, I think that is important to establish
so if anyone has any additional info please add it here. So far I am only
aware of the following impact:

• Amasty FPC extenstion (@seansan https://github.com/seansan and
  @trabulium https://github.com/trabulium)
• @seansan https://github.com/seansan mentioned an iphone app (is
  this a private custom tool or public?)

Anything else?

Again, sorry for the unexpected trouble caused by this change. However if
the fixes are simple and affect a relatively small number of merchants that
can easily revert the changes privately then I think that is not a terrible
resolution.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OpenMage/magento-lts/issues/990#issuecomment-728278840,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE7I27T6ULNEYENXD2MPBDSQF5JRANCNFSM4NFO3KJQ
.

[seansan]

I had contact with BW guys, and there it is ;) Case closed?

https://trends.builtwith.com/websitelist/OpenMage

[colinmollenhour]

Thanks @seansan!

I think we can leave this issue open a while longer for any other similar services.. I don't use these much but I'm assuming there are quite a few more of them out there.

[seansan]

Sounds good.

which services? then we can give them a nudge ;)

On Mon, Nov 30, 2020 at 6:33 PM Colin Mollenhour [e-mail]
wrote:

Thanks @seansan https://github.com/seansan!

I think we can leave this issue open a while longer for any other similar
services.. I don't use these much but I'm assuming there are quite a few
more of them out there.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/OpenMage/magento-lts/issues/990#issuecomment-735931324,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE7I2Y2LUKVGJ7IR3M3ENTSSPJNHANCNFSM4NFO3KJQ
.

[sreichel]

If Amasty FPC is the only module broken then perhaps we can request them to create an OpenMage compatible version?

Got feedback from Amasty (few days ago) ... Ticket 263626 ...

Hi Sven,

Thank you for getting in touch.

I have forwarded your request to a colleague of mine from the dev team so they can review it.

Once there is any news, I'll get back to you.

Thank you.

and

Hello,

I hope this email finds you well.

I have got an update from the dev team.

Developers created a patch where we improved the compatibility with OpenMage:
https://drive.google.com/file/d/1JJDLN0j8fGWI4ZJsjedm8ul_brUCCP5f/view?usp=sharing

At the moment, I cannot guarantee that the above fix will be added to the official extension release.

If you have additional questions, feel free to ask.

Thank you.

Can someone confirm this fix?

I also asked for supporting OM and waiting for reply.

[rjocoleman]

https://gist.github.com/rjocoleman/1126f93c02a4de1f86accdea34eb4df2

That gist/patch is the crux of the changes (from the linked Google Drive) against the 1.8.23 release (there is more but it's just copyright changes on most of the files). The <session_name> change to app/etc/amfpc.xml is also required.

Looks like it should work.

I'll do some further testing on this today.

[sreichel]

The <session_name> change to app/etc/amfpc.xml is also required.

Should be obsolete with .... ?

-            $sessionName = (string)Mage::app()
-                ->getConfig()
-                ->getNode('global/amfpc/session_name');
+            $sessionName = Mage_Core_Controller_Front_Action::SESSION_NAMESPACE;

[rjocoleman]

Update - so far I have not detected any issues with the current 1.9.4.x branch and Amasty_Fpc v1.8.23 with Amasty's patch from my Gist applied via a composer patch.

Thanks for this!