Friendica: mcrypt is not available in php 7.1

[annando]

We have to check where we use it and have to rewrite parts of the code.

[Quix0r]

With Midnight Commander, a simple search for _mcrypt_ reveals 233 findings, without addons.

It seem to start with install.php wich checks for mcrypt_create_iv() availability.

[annando]

We can use https://github.com/phpseclib/phpseclib with the additionally installed packet https://github.com/phpseclib/mcrypt_compat

I would do that, if someone could tell me how to work with this autoloader thingy. (Means: Where to place the files, how to address them, ...)

[MrPetovan]

These days, the most popular PHP tool to manage third-party library dependencies is Composer.

The default behavior is the following: you set up a simple composer.json file listing the required libraries and their target version (can be a wildcard to enable minor version upgrading). The next time you run the Composer executable at the root of your project with the install mode, Composer will fetch the latest appropriate version of all the required libraries and put them in a vendor folder. Additionally, it will create a vendor/autoload.php file that takes care of the autoloading stuff. We just have to require this file in index.php to have access to all the libraries classes anywhere in the code just by instantiating objects. Finally, a composer.lock file is created with the list of the current versions of the libraries that have been imported.

Now, every subsequent time you run Composer install on your app root dir, it will check if the existing libraries' versions match the one listed in composer.lock and upgrade/downgrade libraries to the exact version mentioned.

However, if you run Composer with the update mode, it will try to fetch new versions according to the version mask set in composer.json and update the composer.lock with the new versions if there's any.

Normally the vendor directory is left blank in each project for Composer to populate, but we can still commit the library content into the Friendica codebase so that users won't have to run Composer manually on their own install. Alternatively, we can provide a care-free download archive with all the libraries pre-loaded for a specific tagged release and leave the Git repository free of library clutter.

I strongly am in favor of this since the current HTML-to-Markdown library is available on Composer as well and would make my life easier for #3274.

[MrPetovan]

The future composer.json could look like this:
json { "require": { "phpseclib/mcrypt_compat": "1.0.*", "league/html-to-markdown": "4.4.*", }, "autoload": { "psr-4": {"Friendica\\": "include/"} } }

Oh yeah, Composer also takes care of creating the autoloading links between namespaces and folders.

[annando]

I guess in the past - before you joined - we had some discussion about this. Wasn't the problem that with using this stuff, everybody who just had cloned the repository (for being up to date) had to use this tool as well to create a workable installation?

I think someone said that with this stuff you shouldn't ship the downloaded libraries with the installation or something similar. (Which would be a "no go" in my eyes)

However, this sounds like I should wait for someone™ who has knowledge in this to create the appropriate folder structure and so on. I guess I could spend days trying to understand this thing.

[annando]

Update: For the current issue I just saw that we only need mcrypt in two places by now:

1. With the old RINO mode 1 (but we already are having the mode 2)
2. In the communication with Diaspora

For the second case I could very simply create wrapper functions. This would be a fast solution.

[tobiasd]

On Tue, 28 Mar 2017 18:17:41 -0700
Hypolite Petovan [e-mail] wrote:

Normally the vendor directory is left blank in each project for
Composer to populate, but we can still commit the library content
into the Friendica codebase so that users won't have to run Composer
manually on their own install.

Yes, I think we should do this! Else the simplicity to install Friendica will crumble to dust. I don't consider composer to be "easy to use".

The worst thing are these curl $someurl | sudo php - install instruction. Just take some random script from the internet and run it as root...

[annando]

Update 2:

I just realized that both:

1. I haven't read @Hypolite's mail completely since he wrote about this thing that I declared as no-go.
2. We are using some mcrypt function in RINO2 as well.

[Quix0r]

Composer is really simple to use. Just enter composer install (after you have installed it) and everything needed is being downloaded to ./vendor/ (with .gitignore you should ignore all there) and prepared there, including requirements for developers (like unit tests).

[Quix0r]

@Hypolite you should also include the minimum PHP version and any extensions (except SPL) you use.

Here is a more complete example: https://github.com/PHPOffice/PhpSpreadsheet/blob/develop/composer.json

It looks on first step confusing but it is really straight-forward. No need to tingle around with the composer.lock file. That is being installed/updated for you on initial composer install run (and then you should commit it) and on any update runs.

[tobiasd]

On Wed, 29 Mar 2017 01:12:29 -0700
Roland Häder [e-mail] wrote:

Composer is really simple to use. Just enter composer install

We have different assumptions about "simple" usage then. Installation is included in usage and we cannot expect everybody to install composer on their home computer just to upload Friendica via FTP to their shared hoster. But I'm sure there are 3rd and 4th party tools just to make the deployment schema more complicated.

[annando]

I also don't see the point why everyone who wants to use Friendica had to do such stuff like installing additional third party tools, especially when they are on systems where this isn't possible.

I know that in the past we had people who said that me mustn't add these downloaded files in our repository when we had worked with this mechanism. I guess this was the reason why we hadn't agreed in the past on doing so.

When we find a solution where everything works and no user or developer had to install additional stuff, then I'm fine with it.

[fabrixxm]

ATM we have an autoloader code which is the composer autoloader code, modified. 3rd part libs are auto-loaded from library.

Last time I added a library, I used composer, then moved code from vendor to library and updated where needed files in include/autoloader

[annando]

@fabrixxm do you think that you could do something similar with the libraries that I mentioned above? One of these we are already using - but in an outdated version.

Additionally I saw that the other encryption library is outdated as well and is using mcrypt functions as well: https://github.com/defuse/php-encryption

We had to update it as well.

[fabrixxm]

yes, the update of php-encryption was notice some months ago.
see this thread https://kirgroup.com/display/61b57c7e205741bbf4be76c647081904

I have done (an then forgot about it) a branch with updated php-encryption 2.0.1 which introduces RINO3, because there was some incompatibilities between major version of php-encryption

[fabrixxm]

do you think that you could do something similar with the libraries that I mentioned above?

yup

[MrPetovan]

When we find a solution where everything works and no user or developer had to install additional stuff, then I'm fine with it.

I disagree with this blanket position. I think we can have two different environments, one for developping with an empty vendor/library folder that requires Composer, and a production snapshot with a pre-populated 3rd-party folder and no need for Composer at all.

I mean, if you already can do a git pull, you can do a composer install as well. Conversely, anybody wanting to set up an instance should only have to copy a release's files into any FTP folder and point a browser to it, and updates should be as easy.

It's definitely possible to maintain both environments to make the developers and users happy. The only missing part currently is the release packaging and hosting, but that shouldn't be too hard to set up if it makes easier for dev to include/update 3rd party libraries.

[annando]

By now we ask every regular user to simply do a git pull to be up to date. Several of them are having this in their crontab. And these are regular users, no developers.

Especially since we are actively deleting old files, users then had to to updates manually - which is more complicated.

We would lose users for a benefit that I don't really see. (In comparism to to this "composer" thing and to ship theses downloaded files in any way)

[annando]

@fabrixxm concerning the "php-encryption" library, we maybe could replace it with "phpseclib"?

[MrPetovan]

By now we ask every regular user to simply do a git pull to be up to date. Several of them are having this in their crontab. And these are regular users, no developers.

Then it's even more straightforward than I thought. My worry was that not all regular users have access to command-line, which means we can't have Composer as the only way to update Friendica 3rd-party libraries. We can ship a standalone version of Composer (that can self-update using the eponymous mode) and ask users to run bin/composer.phar install just after git pull for example.

The benefit of Composer is that we don't have to maintain the library folder by hand nor require any files inside it. This mean we are less vulnerable to 3rd-party bugs since we can easily bump versions with a simple command line.

The simple fact that I manually have to upgrade the HTML to Markdown library from 2.2 to 4.4 is a reason good enough for me to support the addition of Composer. It would also make this mcrypt migration easier.

[annando]

I don't think that this composer thing would help with version incompatibilities between the versions. (What @fabrixxm experienced when he wanted to update the "php-encryption" library from 1.1 to 2)

And we also have to cover the people who just download the master.zip or develop.zip and transfer it via FTP - without shell access.

[annando]

What really votes against shipping a non empty vendor/library folder?

I don't vote against this composer - if someone explains me in simple words how to work with it and how to setup this thing with new libraries.

[MrPetovan]

The main improvement would be not having to move any files around to the library folder and no to have to manually write the autoloader file.

Regarding the content of vendor we ship, it mainly is about redundancy. The codebase already exists in their respective repository, and if it doesn't we'd better find a maintained alternative than keep an unmaintained library in vendor because we once copied it there.

Overall it's not a huge improvement, but it's not a huge drag either since we would only need to package libraries in an archive for every public release.

[MrPetovan]

if someone explains me in simple words how to work with it and how to setup this thing with new libraries.

If you expect people to have composer installed globally:

1. Create a composer.json file listing the libraries Friendica depends on.
2. Run composer install to import Friendica's dependencies into vendor. The imported libraries' own dependancies will be included as well. For example, requiring phpseclib/mcrypt_compat will automatically import phpseclib/phpseclib as well.
3. Commit both composer.json and composer.lock.
4. Periodically run composer update to upgrade libraries and commit the updated composer.lock
5. If a major version change is needed, simply edit composer.json, run composer update, and commit both composer.json and composer.lock.

If you want to bundle Composer with Friendica:

1. First download composer.phar into util and chmod it executable.
2. Create a composer.json file listing the libraries Friendica depends on.
3. Run util/composer.phar install to import Friendica's dependencies into vendor.
4. Commit util/composer.phar, composer.json and composer.lock.
5. Periodically run util/composer.phar self-update and util/composer.phar update and commit the updated util/composer.phar and composer.lock files.
6. If a major version change is needed, simply edit composer.json, run util/composer.phar update, and commit both composer.json and composer.lock.

[Quix0r]

Or: have finished bundled downloadable files (they can be put on any "cloud" like @nextcloud is. Composer provides here the command composer archive which creates the finalized bundle, including all files in ./vendor/.

later in your PHP code sinply include ./vendor/autoload.php and everything 3rd-party is being loaded through autpload-function (or method I think).

[MrPetovan]

later in your PHP code simply include ./vendor/autoload.php and everything 3rd-party is being lazily loaded through autoload-function (or method I think).

... and any other source folder that needs PSR-4 autoload like we intent to switch Friendica Core classes to.

I didn't even know about composer archive, this makes the release bundling even easier than I thought.

[annando]

"even easier" are not the words I would have used. I have the feeling that if we switch to this method, then I will need constant help in installing, including and use of the libraries.

[MrPetovan]

Look, I'm not going to lie to you, you perfectly understood it, this is a change. However, I'm ready to support you all along the way, but I think you greatly underestimate yourself if you think you can't get used to it. It's a change alright, but not a huge one. Here's what's going to change with Composer:

• 1-2 more CLI command to run to install/update a dev instance of Friendica.
• No more require calls to include vendor/src files. I believe this will shave up to 0.2s from some requests where we currently include recursively a whole bunch of files in the off-chance that we'll need them (we don't).
• Classes will have to be instantiated using their namespace or through the system call [code]use[/code]. This may prove the most confusing initially, and I'll be on call to assist with any issue you or anyone may have regarding this. I believe @Quix0r and @fabrixxm will be able to help as well.

[tobiasd]

Then it's even more straightforward than I thought. My worry was that not all regular users have access to command-line,

They don't. Where come the illusion from, that everyone has CLI access nowadays?

[annando]

Every line like require_once 'include/config.php'; then has to be replaced with a line like use \Friendica\Core\Config; - like we already are doing, I guess.

This is not my concern.

I'm thinking about our users. I'm working directly with users in my everyday work. Every additional task that is required to install and/or update Friendica will definitely lead to a loss of installations.

From my point of view we can ship an empty vendor folder when we eventually will have an automatic update procedure - or this composer stuff could be called from the frontend as PHP function call as well.

[fabrixxm]

concerning the "php-encryption" library, we maybe could replace it with "phpseclib"?

could be a solution, yes.. I need to find time to investigate..

[Quix0r]

Well, webmaster can download a prepared bundle which someone of us has built (with composer archive which is an easy command to build fully prepared bundles). Then no webmaster need CLI access, simply upload the unpackaged files as usual. And don't tell me you (us developers) don't know how to use Shell or command prompts (under Windows) ... :-)

@annando no, they (webmaster) still have to download a prepared "binary" package, unpackage it and upload all files (pre-clean existing files as before, of course, too). Nothing for them has changed, only for us. I found composer some days ago and the composer.json (which is your starting point) was a bit confusing but that passed away after a few minutes. It is really easy to learn (see https://getcomposer.org/doc/00-intro.md for instance) and helps keeping track of 3rd-party libraries. You can also specify your own repositories, including (but not limited to) GIT, SVN and many other VCS' more.

[annando]

@fabrixxm I just replaced the mcrypt functions in the Diaspora code with (our old) "phpseclib". This seems to work. So I could try to do this with the RINO code as well.

[Quix0r]

@annando don't mix-up two different things: webmaster|administrator != developer. With webmaster|administrator I mean those who want to install Friendica on their server (let's call them user here). The user (role) wants to download a pre-build package, unpackage it and upload all extracted files to the server and call an installation routine there, alternatively a good-documentated manual installation should be provided, in case of the "automatic" fails for some some reason.

Developers in contrast should have all development tools around (and that is not just an editor) that are needed to "build" the software from "corresponding source code". Many other FLOSS have something like autogen.sh around which builds the build system (configure and Makefile-s) and then ./configure --some-foo-parameter is being called (or cmake, whatever, depending on build system) and then the binary files are being build.

I do this since I have started downloading projects by versioning software. Sure then I don't have something finished. I have then to act like a developer (even when I don't want/can develop on it) by running the same build process as any other developer else.

To not give the user this hazzle, there are so-called "binaray packages" (pre-build, ready-to-install) downloads for many years around (like it happened with Linux kernel for example). Developers don't download these packages, they use VCS and then need to make sure that all used libraries (and their header files) are installed on their system, including all neccessary build tools.

So, what @Hypolite and I are proposing here is (OT, of course but it emerged from it) that Friendica has a real-world release system (like many other FLOSS projects have, too). Users download pre-build packages ("binary packages") and developers the "corresponding source code".

[annando]

Thing is: Currently we are using "git" as an update routine - even for regular users. (we even encouraging them to do so)

We can go to some system with "pre build packages" when we have developed a reliable update routine on our own - not before that.

Especially with users who are working on the "develop" branch (or the "rc"), it would be too much work to do the updates on their own every few days.

[Quix0r]

Why are users on a developer's branch and not on a release (pre-build bundle)? If they [users] to so, they act like a developer and not a user, because they are on a development branch. I do the same with other FLOSS projects, too. I don't complain there (anymore), that the latest code doesn't contain configure, I have to run the "bootstrap" (done by developer normally) which mostly involves installing libfoo-dev and autoconf/automake packages.

Normally, you have a developer branch (here develop), a staging branch (to pre-test the code under live conditions) and release tags (yes, others do it other way, I know). And these release tags are being build ready-to-install "binary packages" (like deb, rpm) from, so regular users have it the easiest way, mark a package for installation and let the package manager do the rest.

Similar with Friendica, would be really straight forward and easy. If there is someone [user] around, that wants to use development branch, then he/she has to behave like a developer, like I do, install all needed packages to build a "binary" (= ready-to-install) package and then use that one.

[Quix0r]

So you are saying, to name it clearly, that some of your known users can make decisions (even when they don't want but you give them veto right) what happens on development branch, even when they don't speak up here.

I cannot belive that, that some users are on an developer branch and seem to have a veto right even they are no developer here ...

[annando]

Just concerning the recent versions:

• 3.5.1-dev (19)
• 3.5.1-rc (4)
• 3.5.1 (32)
• 3.5.2-dev (26)

We don't have that much people developing the code. But these people want to be "on the edge". Since our code is running astonishing stable - even on early development versions - many people rely on this. And they don't want to wait for month for new stuff. (Benefit is that the develop branch is constantly tested)

And I wouldn't want to separate the different persons that much. Developers are developing their stuff and they should ask around the developers and the users if this would be a good feature. And if someone (and it doesn't matter who the person is) would veto against it (with good arguments) then this is something that we have to look at.

These people on the develop branch are our benefit. They are willingly to test our code and they are reporting bugs and/or feature requests. Every project should be lucky to have these people in their project. Even if they cannot code they contribute in their very own way.

I don't believe in a world where the developers are in a golden cage where they don't listen at what the users are saying.

[annando]

I want to have it like Nextcloud is doing:

Means: A system that can be updated via the users in an easy way - with the possibility to select the update channel.

[MrPetovan]

@Quix0r perfectly summarizes my thoughts that I'm too angry today to write respectfully.

[AlfredSK]

Hi,
some thoughts from my perspective as an user and admin. I've installed Friendica via git clone. Why have I done it that way? Well, because it is a very easy process. Much easier and faster than downloading an archive, extracting and moving the files to the document root. Updates are easy, too. In fact, I have configured a cron job for that task.
I'm on develop branch. Not because I want to be a developer - I cannot, because I have no coding skills - but because _I want to help_ testing new features and finding bugs.
What I want to say is there are more "roles" than just users and developers.
The changes mentioned in this thread (composer stuff) are absolutely okay for me. If it helps to make your work easier - go for it. But I want to kindly ask to think of people like me. :) I think we also need test pilots.
Thank you all for your great work!
Best regards,
Steffen

[Quix0r]

I'm willing to host it for you, like you may have already seen in my @nextcloud instance: https://cloud.mxchange.org/index.php/s/mWsAI4u5IuaUhfI

I recommend then to have an expiration date set, as my storage (10 TB traffic are never reached on my server) is not "unlimited" (even the NSA has "only" ~ 2 EB storage, so not endless).

[tobiasd]

On Thu, 30 Mar 2017 04:12:53 -0700
Roland Häder [e-mail] wrote:

Why are users on a developer's branch and not on a release
(pre-build bundle)?

Call them Beta-Testers if you want. They are not developers, but help developing by offering their eyes to spot problems with the cutting edge code, that slipped through the eyes of the developers while the code was not merged into the repository.

[tobiasd]

On Thu, 30 Mar 2017 05:24:21 -0700
Michael Vogel [e-mail] wrote:

I don't believe in a world where the developers are in a golden cage
where they don't listen at what the users are saying.

Thanks :-)

[Quix0r]

Okay, same what I'm doing with many other FLOSS. So I always follow the way then, developers go as I have developer (bleeding-edge) code and not a finalized build package. Same should apply to those beta-testers.

[annando]

Since mcrypt is now removed we can close this issue, can't we?

[MrPetovan]

BAM!