Suitecrm: Custom files not overriding SuiteP core - 7.9 Beta

[joshuahilton]

Issue

I wanted to override some language strings as well as some tpls in SuiteP.

I edited the login.tpl and en_us.lang.php files and placed the tpls in /custom/themes/SuiteP/tpls/ and the lang file in /custom/languages/

I then did a quick repair which did not do anything as the old strings were still showing. I also did a rebuild javascript languages which also had no effect.

I also edited the footer.tpl and placed it in the custom folder, which DID work, a custom style.css also worked fine.

Expected Behavior

Placing tpl and and lang files in the custom folder did not override the core SuiteP files.

Actual Behavior

Old string values remain.

Possible Fix

Steps to Reproduce

1. Edit a tpl listed above or a lang file and place it in the appropriate custom folder.
2. Run a quick repair

Context

Your Environment

• SuiteCRM Version used: 7.9 Beta
• Browser name and version (e.g. Chrome Version 51.0.2704.63 (64-bit)): 57.0.2987.133 (64-bit)
• Environment name and version (e.g. MySQL, PHP 7): MySQL, PHP 5.5.9
• Operating System and version (e.g Ubuntu 16.04): OSX

[pgorod]

If you specify exactly which files you changed, and what changes you made, I believe your bug report will be much more valuable.

Just because it failed for some files, doesn't mean it will fail for all. And if I go and try one customization and find out that it works, I'm just wasting my time. I'd much rather try the exact one that failed for you, to see if it works for me.

[joshuahilton]

I edited login.tpl and en_us.lang.php files.

In login.tpl, I removed the image tag for the SuiteCRM logo in the top left corner.

In en_us_lang.php I edited the quick create strings for 'Create Account' and 'Create Contact' to be 'Create Establishment' and 'Create Booking' respectively.

[pgorod]

I think your notion of "exactly" is different from my notion of "exactly". :-)

I see 4 login.tpl files in my system, and 146 en_us.lang.php files!

In order to reproduce your issue, I need to know (without having to do a lot of guessing)

• the full path of the initial file
• the full path of the file you created in the custom folder (based on that initial file)
• the exact changes or even the full content of the new file (the problem might be originated by some syntactical detail in your changes)

With this, in two quick minutes (more than I've spent on this chat here) I'm testing your exact situation in my test system.

Thanks.

[daniel-samson]

@joshuahilton the language file should be in custom/extension/application/Ext/en_us.lang.php. You need to run repair and rebuild for this to take effect.

I can confirm that the login.tpl is a bug. modules/Users/Login.php:220 doesn't check for the custom version.

[daniel-samson]

Perhaps the order should be

1. check custom/themes/current theme/login.tpl
2. check custom/modules/Users/login.tpl
3. check themes/current theme/login.tpl
4. check modules/Users/login.tpl

[pgorod]

@daniel-samson it seems you are better than me at guessing what people mean! :-)

About the language file, shouldn't it be in

custom/Extension/application/Ext/Language/en_us.descriptivename.php

so that then it would be consolidated into

custom/application/Ext/Language/en_us.lang.php

during the Quick Repair and Rebuild? I'm following Jim Mackin's book here, though I can't test right now.

I think the general idea is that everything in directories named Ext is auto-generated, except stuff under custom/Extension/*/Ext...

[daniel-samson]

@pgorod - lol ... that's what i meant ;D - but well spotted.

@joshuahilton
custom/application/Ext/en_us.lang.php is the custom location which SuiteCRM loads. You could edit that location if you are not using the extensions framework. @pgorod is correct ... you should edit the custom/Extension/application/Ext/Language/en_us.lang.php and then run a repair and rebuild so that it gets built into custom/application/Ext/en_us.lang.php.

I will check Jim's book before posting...lol

[joshuahilton]

@pgorod @daniel-samson the language file I was overriding was /language/en_us.lang.php so I placed the edited version in /custom/language/en_us.lang.php

If it was meant to be in custom/application/Ext/en_us.lang.php then this will be why it wasn't working :)

[chris001]

@daniel-samson The real issue is that the custom file loading isn't handled consistently at the global framework level, in one base class, as it should. Instead, the Sugar developers copy-pasted the same custom override code, but in slightly different forms, to (estimate) 400 out of 500 of the application code files. Then, this bug happens. Some of the application code files ones which don't have the copy-pasted code, so they don't do the custom loading when they're expected to!
https://en.wikipedia.org/wiki/Copy_and_paste_programming
Good thing, there's a tool to help locate it in the code, to solve it much more easily.
https://github.com/sebastianbergmann/phpcpd

composer require --dev sebastian/phpcpd

You can then invoke it using the vendor/bin/phpcpd executable.
Usage Example
Let's check the SuiteCRM include folder for duplicate code.

$ vendor/bin/phpcpd include
phpcpd 2.0.4 by Sebastian Bergmann.

Found 97 exact clones with 11006 duplicated lines in 79 files:

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetFieldrelate.php:178-193
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetFieldrelate.php:197-212

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateTaskButton.php:108-133
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButton.php:208-233

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButton.php:46-94
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButton_c.php:46-94

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButton.php:114-268
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButton_c.php:100-254

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButton.php:268-298
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButton_c.php:255-285

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateTaskButton.php:83-158
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateNoteButton.php:83-158

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateCampaignLogEntryButton.php:93-123
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopSelectButton.php:128-158

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateTaskButton.php:104-135
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopButtonQuickCreate.php:114-145

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateNoteButton.php:57-148
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopScheduleMeetingButton.php:57-148

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateAccountNameButton.php:68-80
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopCreateLeadNameButton.php:72-84

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopScheduleMeetingButton.php:54-151
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopScheduleCallButton.php:54-151

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopSelectUsersButton.php:85-125
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopSelectContactsButton.php:86-126

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopSelectButton.php:126-164
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopSelectContactsButton.php:129-167

  -     /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopSelectUsersButton.php:149-195
        /home/company/public_html/crm/include/generic/SugarWidgets/SugarWidgetSubPanelTopSelectContactsButton.php:145-191

  -     /home/company/public_html/crm/include/SearchForm/SearchForm.php:456-483
        /home/company/public_html/crm/include/SearchForm/SearchForm2.php:519-546

.................


  -     /home/company/public_html/crm/include/SugarPHPMailer.php:130-157
        /home/company/public_html/crm/include/SugarPHPMailer.php:164-191

  -     /home/company/public_html/crm/include/Dashlets/Dashlet.php:217-237
        /home/company/public_html/crm/include/utils/layout_utils.php:79-99

  -     /home/company/public_html/crm/include/SugarObjects/templates/basic/metadata/SearchFields.php:58-79
        /home/company/public_html/crm/include/SugarObjects/templates/person/metadata/SearchFields.php:94-115

..................


  -     /home/company/public_html/crm/include/SugarObjects/templates/basic/metadata/SearchFields.php:47-79
        /home/company/public_html/crm/include/SugarObjects/templates/company/metadata/SearchFields.php:89-121

  -     /home/company/public_html/crm/include/SugarObjects/templates/person/Person.php:284-333
        /home/company/public_html/crm/include/SugarObjects/templates/company/Company.php:157-206

  -     /home/company/public_html/crm/include/SugarObjects/templates/person/vardefs.php:396-454
        /home/company/public_html/crm/include/SugarObjects/templates/company/vardefs.php:305-363

  -     /home/company/public_html/crm/include/HTMLPurifier/standalone/HTMLPurifier/Lexer/PH5P.php:907-946
        /home/company/public_html/crm/include/HTMLPurifier/standalone/HTMLPurifier/Lexer/PH5P.php:959-998

3.41% duplicated lines out of 322925 total lines of code.

Time: 9.99 seconds, Memory: 428.00MB

The results are horrifying. 11,006 exactly duplicated lines of code, in 79 code files!
And that's only in the one folder, include. The rest of the SuiteCRM application code files reside in several other folders, and contain many tens of thousands more copy-pasted lines of duplicate code.

[pgorod]

@joshuahilton read my post carefully, that path you mention is still not the correct one.

[daniel-samson]

@chris001 We are already aware of this issue. Since we have to keep SuiteCRM upgrade safe, some of the duplication is necessary for example the vardefs, and metadata. Some of this duplication is by design. Since modules can be added or removed. Having duplication of data structures, languages, and other side effect files in this case makes sense.

As for the core (include directory, data, and themes directories etc...). We are working towards cleaning up the code base. May I bring your attention to our sonarqube page. It displays the issues we are focusing on.

The community are more than welcome to help out. If you are removing duplicated code, please first check to see what is using that code and handle it appropriately. Remember that someone in the community may be relying on that piece of code. Also I would recommend that members work on a small parts of the code base and make a pull request for just that small area. This will make it easier to review and test the pull requests.

[joshuahilton]

@pgorod sorry I don't think my reply was clear. I was saying that I now realise the path I was using was the incorrect one.

[chris001]

@daniel-samson Thanks for your reply.

1. In soniqube, the 94.7k "minor code smell" errors of code formatting, 1 min effort each = 1 man-year of human work effort.
   OR let Travis-CI run php-cs-fixer, or run php-cs-fixer manually, on the code, and it fixes all of them in 1 hour.
   Who's in charge of making sure travis is doing all the good things it can do for this application project??
   It seems like nobody is.

1. For the duplicated code, again this is a serious problem which my favorite automated tools help to accomplish 90-99% of these fixes with zero impact (the term I prefer is "side-effects") on the code calling that code.
   Again here, I recommend absolutely requiring, for each changed code file, you must create or update the related phpunit test file.

[daniel-samson]

@chris001 Lets discuss this topic in forums. This issue is not the place for this discussion.

I noticed your suggestion in:
https://suitecrm.com/forum/suggestion-box/13923-tackling-github#46620

I will post my response there.

[pratiksgith]

SonarQube link isn't working

As for the core (include directory, data, and themes directories etc...). We are working towards cleaning up the code base. May I bring your attention to our sonarqube page. It displays the issues we are focusing on.

[Dillon-Brown]

@pratikbhopalgmailcom I believe the old sonarqube stuff is no longer in use. We have Codacy which should also automatically run on every pull request along with Robo tasks if you were wanting to automatically run PHP-CS-Fixer.