Suitecrm: [Meta] Move remaining vendored libraries to Composer

[connorshea]

The goal of this issue is to track a bunch of smaller issues that are dedicated to removing vendored libraries from the repository and replacing them with corresponding Composer packages. This serves a few purposes:

• Easier to upgrade.
• Prevents customization of core code from other libraries (which is bad because it prevents us from upgrading the libraries)
• Easier to track security vulnerabilities in the libraries.
• Fewer files in our repo.
• Easier to track when libraries stop getting maintained.

A lot of packages have already been moved in the last year or two, but there are still quite a few left to move.

This list isn't necessarily comprehensive, it's just all of the vendored libraries that I was able to find.

• [x] Smarty (Issue: #6719, PR: #7591)
• [x] SAML2 (PR: #6850)
• [ ] Twitter OAuth (Issue: #7595)
• [ ] Facebook OAuth
• [x] HTMLPurifier (Issue: #3100, PR: #5959) - this was actually added to Composer, but never removed from include/ for 'backwards compatibility'. I'd say we can probably remove it entirely pretty safely. It's already been removed in the develop branch. There's also a file for HTMLPurifier at jssource/src_files/include/HTMLPurifier/standalone/HTMLPurifier/Printer.
• [ ] nusoap (Issue: #7590, PR: #7592)
• [ ] HTTP_WebDAV_Server (Issue: #7596)
• [ ] TCPDF (Issue: #6513)
• [ ] [Crypt_Blowfish](https://github.com/salesagility/SuiteCRM/tree/ef79908dad38b294d2f62924f91aeaa805d8f7a3/include/Pear/Crypt_Blowfish)
• [x] [XML_HTMLSax3](https://github.com/salesagility/SuiteCRM/tree/ef79908dad38b294d2f62924f91aeaa805d8f7a3/include/Pear/XML_HTMLSax3) (#8088)
• [x] Pclzip (PR: #7287) - removed in develop.
• [ ] ytree - vendored at include/ytree, seems to be a PHP library with a bunch of JS files? I'm not entirely sure what this is/where it comes from, or if it's a third party library at all. It might be YAHOO.widget.TreeView? Some of the files are repeated at jssource/src_files/include/ytree.
• [x] mPDF - vendored at modules/AOS_PDF_Templates/PDF_Lib/, it was fixed by #7118 in develop.
• [x] Various Zend packages in Zend/, based on the Version.php file it's at v1.10.8. That's really old, 2.0.0 came out in September 2012, I'm not really sure when 1.10 came out. The oldest version on packagist is 2.0.0, but there is a zf1/ namespace with older versions of Zend packages, e.g. https://packagist.org/packages/zf1/zend-uri has 1.12.3+. We also have some Zend files at modules/AOD_Index/Lib/Zend/Search
  
  
  
  • [x] Zend HTTP (Packagist) (#7672)
  
  
  • [x] Zend Oauth (Packagist) (#7672)
  
  
  • [x] Zend Gdata (Packagist) - This one doesn't actually seem to be used for anything. (#7672)
  
  
  • [x] Zend Validate (Packagist) (#7672)
  
  
  • [x] Zend URI (Packagist) (#7672)
  
  
  • [x] Zend Lucene Search (Packagist) (#7690)
  
  
• [ ] XTemplate (Issue: #8016)
• [ ] ParseCSV (PR: #8006)
• [ ] PHPSQLParser, vendored at include/php-sql-parser.php, it's only used in a small handful of places. It's on a fairly old version of the library, but the github project is here
• [x] XHProf (PR: #8095), vendored at include/SugarXHprof/xhprof_lib. It doesn't seem like the code has been touched in a long time, and I don't think it works on PHP 7. The project seems abandoned.
• [x] XML_HTMLSax3 (PR: #8088)
• [ ] [pChart](http://www.pchart.net/), vendored in modules/AOR_Charts/lib/pChart.

I've also got a list of JavaScript libraries. These are harder, since we don't use npm/yarn and most of these probably don't have corresponding Composer packages:

• [ ] TinyMCE (PR: #6699) - this was moved to composer in develop, but tiny_mce is still available in include/javascript/tiny_mce/ on develop?
• [ ] jQuery - currently vendored at include/javascript/jquery, it seems to be on v2.1.3. There's also jQuery UI, jQuery Cookie, and a variety of other extensions :/
• [ ] JSTree - vendored at include/javascript/jstree (PR: #7296)
• [ ] Mozaik - vendored at include/javascript/mozaik
• [ ] YUI - vendored at include/javascript/yui3, it seems to be at v3.3.0.
  
  
  
  • There's also include/javascript/yui, which is seemingly at v2.6.0. I'm not sure why we're using both...
  
  
  • There's also jssource/src_files/include/javascript/yui3, apparently.
  
  
• [ ] qtip - vendored at include/javascript/qtip, v2.2.1
• [ ] [c3](https://c3js.org/) - vendored at include/javascript/c3, I guess it's a chart library based on D3? It's minified (and only includes the CSS file?) so I have no idea what version it's on. It's only used in modules/Spots/Dashlets/SpotsDashlet/SpotsDashlet.tpl and modules/Spots/tpl/ShowSpots.tpl.
• [ ] TouchPunch - vendored at include/javascript/touchPunch, seems to be a plugin for jQuery UI. v0.2.3
• [ ] php.js - vendored at jssource/src_files/include/javascript/phpjs
  
  
  
  • license.js says 3.25, but that doesn't seem right since php.js never made it that far?
  
  
  • It looks like the project is now locotus, it was renamed here.
  
  
• [ ] RGraph - vendored at include/SuiteGraphs/rgraph
  
  
  
  • The files say '2015-11-02', so I guess whatever version they were on at that date is the one we're on.
  
  

[connorshea]

We also have jQuery 1.6.4 here for some reason: https://github.com/salesagility/SuiteCRM/blob/develop/jssource/src_files/include/javascript/jquery.js

[connorshea]

Ok, so jssource/ is where we want to look for the _source_ files, include/javascript just has the minified versions that'll hopefully be removed soon. (#7340)

It's still weird to me that we have jQuery 1.6.4 AND jQuery 2.1.3, though.

[connorshea]

Somehow I missed that Zend was vendored in the repository, so I've added that to the list. Looks like we're on 1.10.8?
We have the Zend Gdata, Http, Oauth, Uri, and Validate packages. There's also Zend Search in the AOD_Index module, for some reason?: https://github.com/salesagility/SuiteCRM/tree/develop/modules/AOD_Index/Lib/Zend/Search

That one is going to be... fun. The 1.x docs seem to have mostly become 404 pages now, and Composer has 1.12.3 as the earliest version of the Zend packages.

[connorshea]

Turns out there's also a copy of jQuery at modules/jjwg_Maps/javascript/jquery-1.8.0.min.js AND two copies of jQuery at modules/jjwg_Areas/javascript/jquery-1.4.2.min.js and modules/jjwg_Areas/javascript/jquery-1.8.0.min.js.

😒

Oh, and also one at include/javascript/mozaik/colorpicker/js/jquery.js.

So I think that brings us up to a grand total of 8 copies of jQuery in this repo. ಠ_ಠ

[Dillon-Brown]

8 copies of Jquery.... ಠ_ಠ indeed.

[connorshea]

It looks like we also vendor a library here: include/parsecsv.lib.php

[Dillon-Brown]

It looks like we also vendor a library here: include/parsecsv.lib.php

@connorshea Well spotted, removed here: #8006

[connorshea]

Does anyone know where we can get XTemplate via Composer? Or can we remove it entirely?

EDIT: Opened an issue about removing XTemplate at #8016. It's going to take a decent bit of work to remove it, but it hopefully shouldn't be insurmountable. The library is unmaintained and hasn't been updated in over a decade, as far as I can tell. It's probably got security holes like swiss cheese.

[connorshea]

From codecov.yml, I noticed we also vendor php-sql-parser at include/php-sql-parser.php.

Looks like it's on GitHub at https://github.com/greenlion/PHP-SQL-Parser, and we can get it from Packagist here: https://packagist.org/packages/greenlion/php-sql-parser

Edit: I looked into this a bit more, this could probably be removed relatively easily. It's only used in 3 places. I wasn't able to figure out what version of the library was vendored into our repo, but it's definitely quite an old version.

[connorshea]

Found a new one: pChart is vendored in modules/AOR_Charts/lib/pChart.

Looks like we're using v2.1.4, which was the last version released (in 2014).

It looks like we can it from Composer but it's not official: https://github.com/szymach/c-pchart / https://packagist.org/packages/szymach/c-pchart

The Composer package doesn't have v2.1.4 for whatever reason, not sure if that'd be a problem. 2.0.12 or 3.x would be our best options. 3.x has some backwards incompatibilities and I'm not really sure what the deal is with 2.0.12.

[connorshea]

phpjs can probably be removed relatively easily. It just creates some PHP functions in JS, and we only use one as far as I can tell:

• html_entity_decode: AjaxUI.js seems to be the only place where the html_entity_decode JS function is actually used.
• get_html_translation_table: Only used in other phpjs files.
• htmlentities: Doesn't seem to be used anywhere.

[connorshea]

We also have a copy of moment.min.js, well 2 copies. 1 is used by fullcalendar, not really sure what the other one is for.