Suitecrm: Introduce Composer and Symfony, BackboneJS + UnderscoreJS

[chris001]

These 2 save time with more efficient modern php development, speed up bug fixing and addition of new features and improvements.

Composer
introduces psr-4 autoloading for Sugar classes and proper namespacing
doesn't break backward compatibility
reduces unnecessary "require" directives and allow easier refactoring
allows to update old vendors software like Zend libraries, recaptcha, tcpdf and so on & move it to the vendor dir.

Composer and Laravel5 :
https://github.com/sugarcrm/sugarcrm_dev/pull/233#issuecomment-172608235
"I started to improve SugarCRM 6.5 CE. And i moved at first standart php libraries (phpmailer, tcpdf, zf, tcpdf, e.t.c) to composer https://github.com/butschster/sugarcrm_dev/blob/master/composer.json.

I want to union SugarCRM with Laravel5.
I added console support in SugarCRM, Monolog, generator _ide_helper.php https://github.com/butschster/sugarcrm_dev/blob/master/_ide_helper.php with phpDoc for all beans."

EDIT: additional support to add Composer and Laravel:

https://groups.google.com/forum/#!topic/php-fig/GrLYJaH1ldo (tech lead on Sugar 7 is huge fan of Composer PSR-4 and has added Composer to Sugar 7)

http://academe.co.uk/2013/11/sugarcrm-ce-codebase-improvements-wishlist/ (Jason judge of Tyne and Wear UK and big fan of Composer and Laravel to solve many of the weaknesses of the Sugar CE code that Suite forked).

http://blog.legacyteam.info/2014/10/whats-so-great-about-laravel/ Daniel V explains many of the advantages...

[ghermans]

:+1:

[vjaro]

+1

[JimMackin]

+1 for composer

Related to #782

[defaultbeep]

👍Composer.

Interested to know more about how and why you would include Laravel.
I'm a fan of Laravel and Eloquent myself.

[Danisan]

+1

[fgpalacios]

+1

Fernando.-
El ene 20, 2016 5:35 PM, "Daniel Blanco" [email protected]
escribió:

+1

—
Reply to this email directly or view it on GitHub
https://github.com/salesagility/SuiteCRM/issues/855#issuecomment-173351576
.

[chris001]

@jonathanalbangoodman
Laravel5 is being added now by @butschster to his fork https://github.com/butschster/sugarcrm_dev

Why include Laravel5 in SuiteCRM:

1. Improve the legacy 2004 SugarCRM code http://academe.co.uk/2013/11/sugarcrm-ce-codebase-improvements-wishlist/
2. Many other added benefits such as better development with Homestead, deployment with Settler, etc http://blog.legacyteam.info/2014/10/whats-so-great-about-laravel/
3. Attract a huge community of laravel5 developers to make many awesome modules for SuiteCRM more easily and in less time than before
4. Save time and expense in core and module development
5. Bring millions of more views/eyeballs to SuiteCRM because of being included in Packagist as the popular PHP CRM application with Laravel5 support
6. many more, maybe you can think of some...

[defaultbeep]

@chris001

Cool had a brief look at @butschster's repo.

So Sugar can be wrapped up in a Laravel application extending Laravels behaviour to Sugar? I relatively new to Sugar you're definitely right about it being legacy, would be good to bring it up to speed.

[pribeiro42]

@chris001

The Laravel5 integration will force everyone to re-write extensions, right ?

[butschster]

The Laravel5 integration will force everyone to re-write extensions, right

Maybe no

[chris001]

@pribeiro42
@butschster
IMO probably the best way would be to retain the existing classic sugar 6.5 API for running classic sugar 6.5 modules, and add the new sugar 7 compatible REST v10+ API for new generation modules, and add Laravel5 or 5.1 for the development benefits of 1) (optional) easier REST v10+ module building, 2) (optional) easier suite core development.

[pribeiro42]

@chris001
Who's talking about REST API ? My main concern is the custom modules...

[Mausino]

:+1:

[chris001]

@pribeiro42
It's considered good practice, to wall off the main suitecrm application, from the custom modules, by means of a REST API for secure communication... obviously this also lets you run custom modules as a cloud service

[pribeiro42]

@chris001
And what about those modules you can upload via "Module Uploader" ? Those don't interact with the core of SuiteCRM via REST API, they add tables to the db and might even add custom columns (representing custom fields) to the core SuiteCRM modules...
I'm not trying to undermine your effort of bringing a modern framework to Suite, I'm in favor of that (less so of composer....) just being cautious about the possible implications that may bring to 3rd party developers...

[JoshuaQ]

Moreover, if you are willing to make a change of this scale, we should assess whether it would be more appropriate to use Symfony with silex, given the maturity framework. And if it would be more appropriate to directly touch/remove the module /data and integrating it with an ORM such doctrine. I know it sounds bold, but in my opinion it will be very difficult to maintain in the time this sugar-spaghetti-code if not is evolve in a robust framework that structured and evolves.
Perhaps is very dificult but i think that is the only way that the suitecrm live long time.

[defaultbeep]

Hey @chris001 would custom modules not be served better by an improved internal API?

I think theres a lot of work that could be done to Sugar's core logic before anyone tries to strap a new REST API to it. Big fan of this discussion here, thanks for bringing it up!

[jakubpas]

Thanks for mentioning my pull request from SugarCRM CE. Seems they are not interested in merging this I am going to try to add it here. I propose in this pull request:

https://github.com/salesagility/SuiteCRM/pull/1114

• Introduce installation by composer - you can actually test it now:

composer create-project jakubpas/suitecrm your_application_name

• introduce PS-4 namespaces for SuiteCRM
• moving internal and abandoned classes to namespaced /src (currently SugarLogger and XTemplate)
• moving external libraries to composer's /vendor (currenty Smarty as example)
• some minor refactoring

The" jakubpas/suitecrm" can be replaced by official SuiteCRM packagist location as soon as it will be set up.

[chris001]

It'd be great if SalesAgility would register an account on packagist.org, and add the SuiteCRM project, which would allow new users and developers to easily install via one line, composer create-project salesagility/suitecrm your_application_name.
https://packagist.org/search/?q=suitecrm

[chris001]

How to get SuiteCRM application onto packagist.org, and use github to auto update the SuiteCRM packagist application at every new release:

1. Go to Packagist web site, create your account such as salesagility, login and Submit a Package.
2. Packagist'll ask you for Repository URL (Git/Svn/Hg).
   Paste there [email protected]:salesagility/SuiteCRM.git and click "Check!".
3. Packagist will check your project and return the project name, SuiteCRM.
4. If it's correct, accept it.

How to Update the Packagist

• Every time you do a new release SuiteCRM application code to Github, you must either manually or automatically update the SuiteCRM application on Packagist.org

1. To manually update, go to the salesagility Packagist account, go into the SuiteCRM package, and click "Force Update!". Packagist will go to Github and pull in the updated SuiteCRM application files.

2. To turn on "auto update", which pushes changes from github to packagist automatically, go into the salesagility github account, click "Settings", then "Service Hooks", and click the "Packagist" service. There, update with the salesagility packagist account information:

   User: your Packagist username, salesagility
   Token: your packagist API token, copy from the salesagility Packagist account Settings page.
   Domain: packagist.org
   Ok!

3. Once auto-update finishes pushing code from github.com to packagist.org, the latest release of SuiteCRM will be available on packagist.org for users to easy install from the command line:

composer create-project salesagility/suitecrm MyOrganizationCRM

[B-Prod]

Using Composer and go to Packagist would be a great enhancement. Moreover, it would be really great to require "composer/installers" so we could install the suiteCrm core in a custom folder as it is the application core.

[chris001]

@B-Prod Good idea, as far as I know, composer/installers would enable SuiteCRM to install other software on the crm server, ie the support Portal, a joomla extension. Would be great to save the system administrator the time and effort of installing it themselves, and just hit a button on the Admin page and SuiteCRM installs it for you.

[B-Prod]

I would allow to put the core source code in a specific directory (like "web") instead of the "vendor" one, but without redefining the "vendor-dir" setting so the other packages may continue to be shared along the "vendor" folder.

Moreover, by defining some specific types (eg: "suitecrm:module", "suitecrm:theme" etc.) inside the "suitecrm" namespace, all different package of a certain type may be installed in a dedicated directory.
So the "suitecrm:module" packages would be put in the "modules/contrib" folder for example.

This lifting may need some changes in the current code, but it could definitely allow to have clear and simple suiteCrm project repositories.

I don't figure out a correct way to deal with language packs, as the files are dispatched into a lot of core folders. This is a big issue because when updating the source via Composer, the dispatched files could be deleted.

[chris001]

@B-Prod The language files should be safe, won't get overwritten while upgrading the core SuiteCRM application. Because the language files are in separate files which live alongside the default English language files, for example, SuiteCRM installer languages are installer/language/en_us.lang.php, es_es.lang.php and ru_ru.lang.php so installing the Brazilian Portuguese pack adds pt_BR.lang.php alongside it. Core app upgrade only writes over, at most, the 3 core languages english spanish and russian, only if they have been updated since last core app upgrade.

[chris001]

It would be really great to add the feature for the installer (and the Admin panel, and the upgrader) to programatically install and upgrade language packs through direct download from crowdin. Make it a lot easier for the user, they just select their language from the list, same as when you install Linux, and select your language, locale, and keyboard layout. The language selector action should auto download and use the language pack.

[pgorod]

@chris001 I think there is only one core language (English), Spanish and Russian are now similar to all the other languages, they come in language packs, not in the main app. But @horus68 can confirm that (and the crowdin direct downloads issue).

[horus68]

For SuiteCRM:

• Core only includes english files.
• Languages are created on Crowdin https://crowdin.com/project/suitecrmtranslations
  From Crowdin users can download ready to install language packs
  There is also 3 language packs (German, Italian and Russian) created and released by different authors outside Crowdin.

A system to take this all in an automatic process and directly from the admin area would be perfect.
But then again... SuiteCRM is not automatically updated... and that should be a priority!

Now take Joomla as a languages centralized installation example:

• Joomla is completely updated directly from the admin area, including a system to discovery updated for core and 3rd party extensions
• Languages are installed in the admin menu from a central site that includes all language packs. Updates are also found and installed directly in the admin area. Each language pack can be installed when first installing or at any time. Languages can be single updated.

See guide: J3.x:Setup a Multilingual Site/Installing New Language https://docs.joomla.org/J3.x:Setup_a_Multilingual_Site/Installing_New_Language

But there are a strange workflow is the creation of those language packs (historical motivation here!): they are created by different teams, mostly using Crowdin. Each team leader uploads them to the Joomlacode repository so they can be discovered by the system.

See demo https://demo.joomla.org/

[chris001]

It's possible to make a module which handles automatically downloading the translations directly from crowdin and auto installing them. Also, automatically checking for updates and auto installing them.

[horus68]

@chris001 its a go from me. But I'm not a coder, so I just can test it when done!

[B-Prod]

@chris001 The main issue with language packs is that Composer updates a package by deleting it then installing the latest version.

So the installed language packs would be erased when updating suiteCrm. However it could be possible to add a "post-update-cmd" script that retrieves all the translations and dispatch them, which seems not too crazy.

Another issue would be related to extra modules or themes, as they are currently looked for in the core directory...
A solution may be found in another project composer file: https://github.com/drupal-composer/drupal-project/blob/8.x/composer.json
So we have the core files separated from the extensions.

The definitions of all types should be done there: https://github.com/composer/installers

[chris001]

It's not just the language packs that would need to be dynamically re-installed, but also, all the files that contain application settings: config.php config_override.php composer.json composer.lock. Then composer update of the app itself would work perfectly.

[B-Prod]

For the configuration and other static files, have a look on what is done there: https://github.com/drupal-composer/drupal-scaffold
The idea is to provide a white list with default values (so standard installation) that may be overriden. Those files are not managed by composer and not deleted.
But the language packs seem to work more as "user" files, so files that may be managed through the backoffice and not always with Composer. The issue is that if those files does not stand in a folder apart, but they are dispatched along a lot of core folders...

Who from SuiteCrm is supposed to confirm if this issue takes the right direction or not?
The suggestions here are really interesting but without a strong willing of integrate them, it seems complicated to contribute. Indeed some part of the CRM need to be rewritten...

[chris001]

@jakubpas has started the composer direction, with the application installable by composer and the logger loading by the composer PSR-4 autoloader. There's still a lot to do, to edit every one of the app source files to add namespace and use the composer PSR-4 autoloader, in place of require and include.

Installing and upgrading by Composer could be added with minimal modifications and additions of code to the SuiteCRM application., and in a way that stays backward compatible with the add-on modules already out there.

Composer has hooks for pre and post install, update, and uninstall, for the root package - SuiteCRM - and other packages - 3rd party libraries, the language packs, all of the SugarCRM 6.5 compatible modules.

There's huge added benefits to install and update SuiteCRM with composer. The best one is how the Travis-CI test script would also then be able to do something more substantial and informative with its automatic testing. The Travis script could have composer install and update 3rd party packages, CRM modules, and language packs, during the test run, and if the tests of the new pull request code then run without error, returning a 0 and not throwing any exceptions, it means that the new code is acceptable and doesn't break anything. So ideally Travis and Composer then would automatically catch errors and notify you so that you have the change to fix them before putting out a release.

[jakubpas]

Finally I found a time to move my patch from sugarcrm code to last version of develop branch in SuiteCRM. Here is the pull request:

Move Zend, Smarty to composer, Introduce psr-4, add XTemplate, Sugar_Logger to psr-4 #3237

The commit introduces moving external libraries to composer and also psr-4 in SuiteCRM
This will help keeping CRM code more light, and also help update external libraries to newer - more up to date versions. Also this will help to install SuiteCRM by packagist.
The change does not break compatibility

Description

introduces psr-4 autoloading for Sugar stuff and proper namespacing. As an example two class were moved to /src: simple one file XTemplate class and more complex subdirectory, multifile Sugar_Logger.

reduces unnecessary "require" directives and allow easier code refactoring in modern IDE’s

allows to update in future old vendor's software like Zend libraries, recaptcha, tcpdf and so on & move it to the vendor dir.

composer.json cleanup to help installation from packagist

Zend and Smarty libs were removed and replaced with composer. Some “require” directives for zend and other classes are removed as they are now autoloaded. Minor fixes in .gitignore.

Motivation and Context

This change is step forward to modern php programing techniques and standards in SuiteCRM. The aim is to go as far as it possible without breaking compatibility with older versions.

How To Test This

composer install

then run installator normally
also

cd tests
../vendor/bin/phpunit

[jakubpas]

Packagist updated. You can install app by:
composer create-project jakubpas/suitecrm --stability="dev"

[chris001]

@jakubpas your PR is an awesome step forward with composer!
For anyone who wants to contribute to this effort to modernize the ancient 13 year old legacy code base of SugarCRM 6.5, here are some more packages that need to be moved over to get autoloaded with composer through use the namespaced classes in these packages, removing all occurrences of require/require_once the direct path to the php file. There are some other packages. This is most of them:

    composer require zendframework/zendframework1
    symfony/security-csrf
    google/recaptcha
    google/apiclient
    ezyang/htmlpurifier
    leafo/lessphp
    econea/nusoap
    FriendsOfSymfony/oauth2-php
    onelogin/php-saml
    pclzip/pclzip
    phpmailer/phpmailer
    tecnickcom/tcpdf

[jakubpas]

Once my changes are merged I can play with moving other libraries.

[horus68]

Once my changes are merged I can play with moving other libraries.

And now we need a @salesagility team here!

[chris001]

Totally @horus68 Composer is part of the new advanced technology used by all modern PHP apps including SugarCRM which is part of what makes Sugar 7.x better than Sugar 6.5. Naturally, to become more secure, competitive, and attract more fans, followers, users, and module developers, SuiteCRM needs to add full PSR-4 compliant composer autoloading of classes.

[samus-aran]

Introduce Composer and Laravel5 or BackboneJS + UnderscoreJS

[samus-aran]

We have now set up a new home for suggestions at Trello. All github issues that were labeled 'suggestion' have been moved and will be closed. Certain ones will be progressed within the new Suggestion Box and may be re-opened.

Announcement of moving Suggestions:
https://suitecrm.com/forum/suggestion-box/13691-moving-suggestions-from-github-to

New SuiteCRM Suggestion Box
https://trello.com/b/Ht7LbMqw/suitecrm-suggestion-box

[chris001]

@JimMackin @mattlorimer @salesagility @willrennie @samus-aran
What else you need to merge this.
Thank you

[samus-aran]

@chris001 Time really. In the new Suggestion Box it is already in the discussion column so it is one of our focus. Hopefully we can progress this further in the next couple of weeks.

[chris001]

Just to share some stats about time.

• It's been 5 years since Composer was released.
• 2.5 years since SugarCRM 7.5 added composer.
• Over 1 year since this PR for composer was opened.
• Over 1 year since @jakubpas first carried over his PR against sugarcrm/sugarcrm_dev to salesagility/suitecrm.
• 1 month since jakub redid his PR to catch up with the changes in the code in the last year.
• Any sensible developer considers it a defect to not use composer to manage and update libraries because of the tremendous amount of time wasted in manually trying to keep up with it all.
  .