Homebrew-core: httpd migration to core - some issues

[edrozenberg]

I had httpd24 running as daemon on port 80 while updating/upgrading homebrew today. After the upgrade these processes were still running and there was no nice way to shut them down other than killall. Also the launchagent or launchdaemon plist file was still loaded and present.

A bunch of cleanup I did to make sure all the old stuff was gone and then (re)installing the new formula httpd. Some of these steps are redundant, I know:

sudo brew services stop httpd
sudo brew services stop httpd24

brew services stop httpd
brew services stop httpd24

sudo killall httpd
sudo killall httpd24

sudo launchctl unload -w /Library/LaunchDaemons/homebrew.mxcl.httpd24.plist
sudo rm /Library/LaunchDaemons/homebrew.mxcl.httpd24.plist

launchctl unload -w ~/Library/LaunchAgents/homebrew.mxcl.httpd24.plist
rm ~/Library/LaunchAgents/homebrew.mxcl.httpd24.plist

brew uninstall --ignore-dependencies httpd
brew uninstall --ignore-dependencies httpd24

mv /usr/local/var/apache2 /usr/local/var/httpd

brew untap homebrew/apache

brew install httpd

sudo brew services start httpd

[edrozenberg]

I was wrong about it using config from /usr/local/opt/httpd/.bottle/etc/httpd - deleting that part of my comment.

Instead the config location httpd uses has changed from the old location:
/usr/local/etc/apache2/2.4

to the new location:
/usr/local/etc/httpd/

Like the new location name, but took me a while to catch on. The old config files are untouched in the old location so I'll migrate them to the new location.

[edrozenberg]

Various paths are different in the new httpd.conf now, so something like this below would help to get someone started on fixing their config. Basically - moving the new config files to .new, moving the old config files to the new expected location, then checking the differences between the old httpd.conf and the new one, and apply any changes from the old httpd.conf to the new httpd.conf by hand (or using a merge tool).

mv /usr/local/etc/httpd /usr/local/etc/httpd.new
mv /usr/local/etc/apache2/2.4 /usr/local/etc/httpd
rmdir -p /usr/local/etc/apache2

mv /usr/local/etc/httpd/httpd.conf /usr/local/etc/httpd/httpd.conf.myoldconfig
cp /usr/local/etc/httpd.new/httpd.conf /usr/local/etc/httpd/

diff /usr/local/etc/httpd/httpd.conf.myoldconfig /usr/local/etc/httpd/httpd.conf

vim /usr/local/etc/httpd/httpd.conf

[edrozenberg]

Also apachectl configtest helps to troubleshoot config file issues.

[edrozenberg]

The docs location used by default in the new httpd.conf is bad and version dependent:

DocumentRoot "/usr/local/Cellar/httpd/2.4.27_2/share/httpd/htdocs"
<Directory "/usr/local/Cellar/httpd/2.4.27_2/share/httpd/htdocs">

I'm changing this to the "normal" location:

DocumentRoot "/usr/local/var/www/htdocs"
<Directory "/usr/local/var/www/htdocs">

Similarly "bad" locations used for the various script alias and cgi locations. I assume the package was compiled with this /usr/local/Cellar/httpd/2.4.27_2/share/httpd setting instead of using the standard /usr/local/var/www location.

[edrozenberg]

Various changes I made to my httpd.conf

A better ServerRoot that doesn't use the hardcoded httpd version:

ServerRoot "/usr/local/opt/httpd"

DocumentRoot set to a standard location:

DocumentRoot "/usr/local/var/www/htdocs"
<Directory "/usr/local/var/www/htdocs">

ScriptAlias changed to a better location (had to create this directory and copy cgi-bin files to this location):

ScriptAlias /cgi-bin/ "/usr/local/var/httpd/cgi-bin/"

CGI directory changed also:

<Directory "/usr/local/var/httpd/cgi-bin">

Added this to the end of the httpd.conf to load any additional config files (such as the php module config file for ex.):

# Other additional configuration files
IncludeOptional /usr/local/etc/httpd/other/*.conf

[ilovezfs]

@kabel any thoughts here?

[ilovezfs]

php issues are for the php tap (https://github.com/Homebrew/homebrew-php/issues), not here.

[edrozenberg]

PHP formula is still trying to use the homebrew/apache tap instead of the new httpd. I'll submit a new issue for the php formulas.

[ilovezfs]

Various paths are different in the new httpd.conf now, so something like this below would help to get someone started on fixing their config. Basically - moving the new config files to .new, moving the old config files to the new expected location, then checking the differences between the old httpd.conf and the new one, and apply any changes from the old httpd.conf to the new httpd.conf by hand (or using a merge tool).

We're not trying to preserve compatibility with the prior formula or automigrate.

[edrozenberg]

@ilovezfs thanks, wasn't suggesting this for the formula, but as a manual approach to consider for people making the change.

[ilovezfs]

@ilovezfs thanks, wasn't suggesting this for the formula, but as a manual approach to consider for people making the change.

:+1:

[edrozenberg]

Submitted php issue https://github.com/Homebrew/homebrew-php/issues/4493 regarding changes php formulas will need to make to work with the httpd in homebrew core.

[kabel]

A better ServerRoot that doesn't use the hardcoded httpd version

This didn't make the migration to fix the default config's server root:

    # point config files to opt_prefix instead of the version-specific prefix
    inreplace "Makefile.in",
        '#@@ServerRoot@@#$(prefix)#', '#@@ServerRoot@@'"##{opt_prefix}#"

Similarly "bad" locations used for the various script alias and cgi locations. I assume the package was compiled with this /usr/local/Cellar/httpd/2.4.27_2/share/httpd setting instead of using the standard /usr/local/var/www location.

This is because of the mixed messaging with the config.layout: https://github.com/MikeMcQuaid/homebrew-core/blob/880a7f76f2e7a0995f5364675944c78ed57469ab/Formula/httpd.rb#L24-L27 are still not good locations. As mentioned people expect them to be #{var}/www/.

[ilovezfs]

@kabel could I convince you to open a PR? :heart:

[kabel]

I'm getting ready to leave for a football game ATM, but I can put one together later this weekend.

[ilovezfs]

Thanks! Enjoy the game :)

[chrisdeeming]

Thanks for opening this issue, I'm sure it will help.

Disappointed, however, that a fairly routine homebrew upgrade has essentially brought my development setup to its knees without warning.

[ilovezfs]

@chrisdeeming sorry for the pain.

[RandomDSdevel]

Just thought I'd note that I left a comment elsewhere that seems relevant here, even if only somewhat orthogonally/tangentially. Perhaps I'll open a new issue for this…

(P. S.: CC @kabel since I '@'-mentioned you in the linked comment.)

[kabel]

@RandomDSdevel sorry the migration didn't go as swimmingly as possible. Configuration is tough when dealing with something as large as httpd's config. I don't speak in any official capacity, but I don't believe there are any plans to start another tap with the formulae that didn't come along. If you think something is missing you are welcome to open a specific issue or pull request and see what the core maintainers say, or take it one step further and create your own tap.

Regarding mod_fastcgi and mod_fcgi:
Both were replaced in httpd 2.4 with mod_proxy_fcgi. Sure, the configuration changes are a bit to deal with, but it's for the better as this module is handled by the core apache team. For more details you can review:

• https://serverfault.com/questions/783173/differences-between-mod-fastcgi-and-mod-proxy-fcgi
• http://httpd.apache.org/docs/2.4/mod/mod_proxy_fcgi.html
• http://httpd.apache.org/docs/2.4/programs/fcgistarter.html

Regarding the other formulae that didn't make the migration:

mod_bonjour: Not included with macOS anymore. Dead?

mod_python: While _dead_ is probably hyperbole, mod_wsgi does seem preferred. See https://stackoverflow.com/questions/3319545/mod-wsgi-mod-python-or-just-cgi

mod_rpaf: Is not included in any main linux distros

mod_security: Might be worth bringing over. It is included in some main linux distros.

mod_wsgi: The preferred method (from the docs), is to install via pip: APXS=/usr/local/bin/apxs pip install mod_wsgi

[RandomDSdevel]

@kabel:

@RandomDSdevel sorry the migration didn't go as swimmingly as possible. …

     Eh, it's not entirely anybody's fault, particularly not yours. Besides, some of the blame for the slight kerfuffle I frenzied myself into surely falls on _me_ for using something that was deprecated without fully realizing it, anyway.

…Configuration is tough when dealing with something as large as httpd's config. …

     No _kidding!_

…I don't speak in any official capacity, but I don't believe there are any plans to start another tap with the formulae that didn't come along. …

     Oh, well. As I've mentioned off-hand once or twice before in another context, I've been needing to invest some time in setting something more modern up in the place of what I have now anyway.

…If you think something is missing you are welcome to open a specific issue or pull request and see what the core maintainers say, …

     Let me stew over it for a bit, and _then_ we'll see if it turns out I need to do that. Right now, I think I can make do with my pre-existing mod_fastcgi installation for the time being, at least until I replace it and the rest of my current (admittedly actually somewhat broken!) web development setup with something else (that hopefully works without making me scratch my head too much, but that's something _else_ that's out of scope for discussion in this issue.)

…or take it one step further and create your own tap.

⋮

     I _did_ consider and start working on creating my own fork of Homebrew/homebrew-apache from before the merged commits that cemented its deprecation on which to base such a new tap of my own when I initially made the mistake of panicking and suspecting that Homebrew might have removed my mod_fastcgi install out from under my nose as a result of its reaction to the tap migration. I abandoned that effort, however, when I realized getting too deep into it would probably entail making some modifications to mod_fastcgi's formula to make it reference HTTPD 2.4's new location in the Cellar that _could_ have been more extensive than any I've ever attempted before, and I've been focusing more on my efforts to learn C++ than educating myself in Ruby and bash shell script lately (I've done revision bumps and modified patch blocks but can't recall if I've ever messed with build steps, though that _may_ just be my memory drawing a blank for the moment…?)

⋮

Regarding mod_fastcgi and mod_fcgi:
Both were replaced in httpd 2.4 with mod_proxy_fcgi. Sure, the configuration changes are a bit to deal with, but it's for the better as this module is handled by the core apache team. For more details you can review:

• https://serverfault.com/questions/783173/differences-between-mod-fastcgi-and-mod-proxy-fcgi
• http://httpd.apache.org/docs/2.4/mod/mod_proxy_fcgi.html
• http://httpd.apache.org/docs/2.4/programs/fcgistarter.html

⋮

     Thanks for the documentation references, as I hadn't seen that last one while searching around to see what equivalents might exist between the different environments' configuration systems (never _did_ manage to finish figuring that out, but I can deal with that later…)

⋮

mod_bonjour: Not included with macOS anymore. Dead?

mod_python: While dead is probably hyperbole, mod_wsgi does seem preferred. See https://stackoverflow.com/questions/3319545/mod-wsgi-mod-python-or-just-cgi

mod_rpaf: Is not included in any main linux distros

mod_security: Might be worth bringing over. It is included in some main linux distros.

mod_wsgi: The preferred method (from the docs), is to install via pip: APXS=/usr/local/bin/apxs pip install mod_wsgi

     The only less well maintained one of those I'd possibly be interested in using in the future _might_ be mod_security if I find out I ever have a use case for it; I'd just use the less outdated approaches like I tend to when starting something out relatively fresh (in this case, that would be with respect to realizing I need those modules' functionality in the first place, not setting up my entire localhost-internal fake 'server' pipeline again, though, like I've touched upon earlier in this post, I'll probably reset _that_ as _well._) Anything else you find worthy of discussing between yourself and the other maintainers and contributors to discussion this issue and discussion and work in other related issues and PRs, I'll likely just lurk around to watch you all hash out like I tend to with any thread that piques my interest as being at least somewhat relevant to me in the end per its eventual effects.

[ilovezfs]

@miniyarov please refrain from comments that don't actually add any new information.

[ilovezfs]

If you brew update and brew upgrade these issues should now be resolved by https://github.com/Homebrew/homebrew-core/pull/18855.

[ghost]

Hi there,

sorry but i dont know where to post this else. lately with the new migrated httpd to core we experience strange memory problems.

we tested it with clean installed high sierra systems, sierra and el capitan. installing homebrew, httpd and php70 or php71 with --with-httpd option.

then testing some local development sites, in the activity monitor, you see still when the page is fully loaded, httpd does not free memory. if u reload the page, the memory increases, step for step. then suddenly some files cant be loaded, or php error pops up with strange class exists error or out of memory with the info that some trillion whatever bytes where tried to beeing allocated.

i am sure its something with the brew httpd ... starting a webserver mit php -s does not give this problems. xamp installation does not give the porblems too. is there anybody out there experiencing this problems? i am out of ideas how to debug this problem furthermore :/

[ilovezfs]

@kabel do you think this is a thread-safety-mpm-mismatch issue?

[javian]

@dl-at-i22 can you verify which mpm you are using for the system and if php is configured as thread safe ?

[ghost]

@javian @ilovezfs thanks for trying to help.

i used the plain brew install without any specific configuration. this is the output for the mpm:

Server version: Apache/2.4.28 (Unix)
Server MPM: event

And yes, php is configured as thread safe after running brew install php70 --with-httpd

[ilovezfs]

@dl-at-i22 you can try this: https://github.com/Homebrew/homebrew-php/issues/4509#issuecomment-336402497

[ghost]

@ilovezfs thanks a lot!! u saved my day. i did not see that, but with thread safety disabled my system is again running fine.

[tonylegrone]

@edrozenberg do you know if these config changes are safe from getting wiped out by future brew upgrades?

[edrozenberg]

@tonylegrone pretty sure any future upgrades will not overwrite existing config. Just to be safe should consider keeping a backup of the current config especially if it's complex or hard to recreate.

[jbiancot]

@ilovezfs @edrozenberg I don't know who was the genius that came with the idea to replace the /etc/apache2 config for the /etc/httpd, because it F sucks. The /etc/apache2 and similar configration files follow the standard of Debian/Ubuntu which is over 65-70% of the linuxes on the net. The crap of RedHat/Centos is less than 25% and everyday losing more market share. I hope the same genius who decided to change and break the config will rectify and rollback. My 2 cents.

[ilovezfs]

@jbiancot Please read our Code of Conduct and adjust your future communication accordingly.

[tonylegrone]

@edrozenberg Thanks. My config isn't that difficult, just not something I have to remember very often. I do have backups just in case.

[tonylegrone]

@jbiancot isn't wrong even though his comments are out of frustration. This change pulled the rug out from under tons of people and left no documentation that I could find explaining what to do about all the missing brew options. I had no idea something was wrong until I noticed httpd wasn't running in brew services and yet, I could access all my sites and had plenty of httpd workers in my process list. Sucked several hours of my day digging around for explanations so I could be sure of how to fix my setup. Luckily, @edrozenberg was kind enough to post this issue and clear some things up.

[ilovezfs]

Please open a new issue if you're still experiencing any problems.