Hhvm: 3.18 - .hhbc created as root

[craigcarnell]

Since upgrading to 3.18, hhvm is unable to open hhvm.hhbc as it is created as root instead of www-data.

HHVM Version

3.18

Standalone code, or other way to reproduce the problem

Permission on /var/run/hhvm/

-rw-r--r--  1 root     root     4504576 Feb 15 09:40 hhvm.hhbc
-rw-r--r--  1 www-data www-data       4 Feb 15 09:40 hhvm.pid
srwxrw----  1 www-data www-data       0 Feb 15 09:40 hhvm.sock

hhvm process:

ps aux | grep hhvm
www-data  4979  0.7  7.9 759320 162740 ?       Ssl  09:40   0:00 /usr/bin/hhvm -m daemon -u www-data -c /etc/hhvm/server.ini -c /etc/hhvm/php.ini -c /etc/hhvm/config.hdf

Expected result

/var/run/hhvm/hhvm.hhbc to be created under www-data user/group

Actual result

[Wed Feb 15 09:37:08 2017] [hphp] [4331:7fab4b3ff700:1:000001] [] Failed to initialize central HHBC repository:\n  Failed to initialize schema in /var/run/hhvm/hhvm.hhbc(rw-r--r-- root:root): \n  Failed to initialize schema in /var/www/.hhvm.hhbc(rw-r--r-- www-data:www-data): RepoQuery::step(repo=0x7fab58917400) error: 'CREATE TABLE main.magic_cc987335f4691fa6ed98336c2378d8e71a751b22(product TEXT);' --> (14) unable to open database file\n
[Wed Feb 15 09:37:08 2017] [hphp] [4331:7fab4b3ff700:1:000002] [7fab615b9428:7fab615bb02a:d92474:1738098:1746028:1746db0:1747256:17472c9:13a760b:13a7d62:34497af:349cad5:349ce79:1389e1b:156cad0:3491ec6:15a3e1d:15a48c2:dac1b3:133b726:7fab66a736ba:7fab6168a82d] Core dumped: Aborted

[LZL0]

Same issue here. I have just downgraded to 3.17.3 because of that. If you guys need additional testing I am willing to help out.

[LZL0]

ProcessID: 1549
ThreadID: 140594705856256
ThreadPID: 2188
Name: unknown program
Type: Aborted
Runtime: hhvm
Version: tags/HHVM-3.18.0-0-g9b285191feb2bb1558bb6682da135263bd2a9e60
DebuggerCount: 0

ThreadType: Web Request
Assertion Message: Failed to initialize central HHBC repository:
Failed to initialize schema in /var/run/hhvm/hhvm.hhbc(rw-r--r-- root:root):
Failed to open /var/www/.hhvm.hhbc: 14 - unable to open database file

Assertion Failure: /tmp/tmp.Fv54re7NgD/hphp/runtime/vm/repo.cpp:562: HPHP::Repo::initCentral()::__lambda62: assertion `false' failed.
URL: /wp-admin/admin-ajax.php

[mofarrell]

How are you guys running HHVM? Do you run it in different modes? Any steps allowing us to reliably reproduce this error would be useful.

Seeing your configs, and your setup would help figure out how you got into this state. You should be able to blow away the repo. The whole /var/run/ folder is destroyed during a reboot. After deleting it does it fail as you start HHVM?

[LZL0]

I have set up HHVM using EasyEngine. If I mark the 'www-data' user as the owner of hhvm.hhbc (or if I delete the file) and restart the server it will just throw the very same error. However 3.17.3 (and older) is working flawlessly.

[mofarrell]

Do you have your config files? They might be helpful as well. There exist settings to control the owner of the repos.

[LZL0]

Let me check if I recall correctly my master NGINX process runs as root and the workers are running as www-data.

[LZL0]

my server.ini file

; php options

pid = /var/run/hhvm/pid

; hhvm specific

hhvm.server.port = 8000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.server.ip = 127.0.0.1
hhvm.jit_warmup_requests = 0

my php.ini file

; php options
session.save_handler = files
session.save_path = /var/lib/hhvm/sessions
session.gc_maxlifetime = 1440

; hhvm specific
hhvm.log.level = Warning
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.runtime_error_reporting_level = 8191
hhvm.mysql.typed_results = false
hhvm.log.header = true
hhvm.log.natives_stack_trace = true
hhvm.mysql.socket = /var/run/mysqld/mysqld.sock
hhvm.pdo_mysql.socket = /var/run/mysqld/mysqld.sock
hhvm.mysqli.socket = /var/run/mysqld/mysqld.sock

[peraleks]

Probably you forgot to run after you install these commands:
$ sudo /usr/share/hhvm/install_fastcgi.sh

in /etc/hhvm/server.ini add:
hhvm.server.file_socket = /var/run/hhvm/sock

in nginx site config add add to location section:
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock

$ sudo /etc/init.d/hhvm restart

[LZL0]

root@server:~# ps aux | grep hhvm
www-data  1554  5.0 47.5 1831712 975500 ?      Ssl  07:16  29:54 /usr/bin/hhvm --config /etc/hhvm/php.ini --config /etc/hhvm/server.ini --user www-data --mode daemon -vPidFile=/var/run/hhvm/pid
root     12043  0.0  0.0  14096   692 pts/0    S+   17:14   0:00 grep --color=auto hhvm
root@server:~# ps aux | grep nginx
root      1019  0.0  0.4 217180  9112 ?        Ss   07:16   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  1027  0.0  0.7 220252 16396 ?        S    07:16   0:06 nginx: worker process                           
www-data  1028  0.0  0.7 219228 14888 ?        S    07:16   0:07 nginx: worker process                           
www-data  1029  0.0  0.4 217180  9444 ?        S    07:16   0:00 nginx: cache manager process                    
root     12054  0.0  0.0  14100   956 pts/0    S+   17:14   0:00 grep --color=auto nginx

[LZL0]

thank you for your help
unfortunately it still crashes, as I already mentioned, the problem doesn't exist with version 3.17.3

[mofarrell]

This shouldn't be necessary, but:

hhvm.repo.central.file_mode=755
hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data

Should allow you to control the permissions that file is being created with.

[LZL0]

added the config lines& deleted the file in question, as a last step restarted the server
it turns out instead of 755 file permissions the file gets 363
when I change the 777 delete the file and restart the server it displays 411

[LZL0]

if I set it to 1469 the file gets 675
at 2100 it gets 064

[mofarrell]

Omit the file mode option in that case

[LZL0]

it gets 644 permissions but unfortunately the error log contains 0 bytes and it seems like the problem still persists

EDIT: gave the folder and its contents to www-data and changed the permissions on error.log to 777, restarted

[LZL0]

I suppose there must be some kind of change in version 3.18 which causes this

[LZL0]

; php options

pid = /var/run/hhvm/pid

; hhvm specific

hhvm.server.port = 8000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.server.ip = 127.0.0.1
hhvm.jit_warmup_requests = 0
hhvm.server.file_socket = /var/run/hhvm/sock
hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data

It looks like that with the new config HHVM 3.17.3 does crash on start aswell.
I am interested interested in @craigcarnell's opinion.

EDIT: By removing the line 'hhvm.server.file_socket = /var/run/hhvm/sock' HHVM 3.17.3 starts successfully, however HHVM 3.18 still doesn't start as expected.

[WrightAaronM]

I'm having the same issue and have been messing with it for hours now. I'm running hhvm behind nginx on Ubuntu 16.04 with postgresql.

Here is my hhvm server.ini:

; php options

pid = /var/run/hhvm/pid

; hhvm specific

;hhvm.repo.central.file_mode=755
hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data

hhvm.server.port = 9000
;hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc

hhvm.server.source_root = /home/mysite/site/code

But I noticed something interesting. When I comment out the hhvm.server.source_root line and restart hhvm, hhvm stops crashing, and I just get a 404 error from hhvm. I don't know if that's helpful or not.

I'll keep working at it to see what else I can find. Is there anything specific I can give that might help?

[WrightAaronM]

An update:
Running hhvm from the command line works fine, even when adding the exact same options used in my server.ini file

hhvm -m server -p 8080 -d hhvm.server.source_root=/home/mysite/site/code -d hhvm.server.default_document=index.php -d hhvm.repo.central.path=/var/run/hhvm/hhvm.hhbc -d hhvm.repo.central.file_user=www-data -d hhvm.repo.central.file_group=www-data

This works fine, even when accessing the site through the NGINX reverse proxy. The problem only occurs when using HHVM as a service.

[LZL0]

@WrightAaronM does HHVM version 3.17.3 work on your side?

[mofarrell]

@LZL0 Did you make sure you removed the repo?

Also to get correct permissions (755) the following options work:

hhvm.repo.central.file_mode=493
hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data

493 -> 755 in octal

I believe this should fix the issue for the time being. I will look in to why it is not creating the repo with appropriate permissions to begin with.

[LZL0]

@mofarrell it works flawlessly with those config lines now, thank you!

[WrightAaronM]

@LZL0 This might be a dumb question, but how would I roll back to 3.17.3 to check? I installed HHVM through the Linux repo. I got the 3.15 LTS to install (and that works fine), but I can't find anything about installing a specific version other than the LTS versions. Would I have to compile from source to do that?

According to my apt update history log, I was running 3.17.2, which worked fine, then upgraded to 3.18 and that's when it broke.

[mofarrell]

I am bisecting the tree to find which commit changed the behavior.

To get 3.17.3 without building from source you can manually grab the deb file off of dl.hhvm.com, and install it using:
sudo dpkg -i <deb file>

If you are on ubuntu xenial this is the file:
http://dl.hhvm.com/ubuntu/pool/main/h/hhvm/hhvm_3.17.3~xenial_amd64.deb

[WrightAaronM]

@mofarrell Thanks for your help. I was able to install 3.17.3 and everything works correctly with that version.

[LZL0]

@WrightAaronM I would advise you to edit you /etc/hhvm/server.ini file

In my instance:

; php options

pid = /var/run/hhvm/pid

; hhvm specific

hhvm.server.port = 8000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.server.ip = 127.0.0.1
hhvm.jit_warmup_requests = 0
hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data
hhvm.repo.central.file_mode=493

But since you are using port 9000 and some other options, yours should be:
; php options

pid = /var/run/hhvm/pid

; hhvm specific

hhvm.server.port = 9000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.server.ip = 127.0.0.1
hhvm.jit_warmup_requests = 0
hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data
hhvm.repo.central.file_mode=493
hhvm.server.source_root = /home/mysite/site/code

[Firegarden]

I have just setup a fresh Ubuntu 16.04.02 on AWS and the only thing I did was install nginx and HHVM (installed multiple times reversing the install order) and every time HHVM gives this error. It shows up as 502 bad gateway but if you look at the HHVM error log you get that hhvm.hhbc issue. The only fix right now is to set the www-data as owner

You can reproduce this issue very easily as I did 5 times. Launch fresh Ubuntu 16.04 and update packages, install nginx, install hhvm - boom 502 on all php scripts. Doesn't matter if you tcp or socks.

I would love to see a fix for this asap because i don't want to revert 3.17

[Firegarden]

Just to be clear the fix for this right now is to set www-data as the owner and group of /var/www/ and restart services - fixed.

Unfortunately when I reboot it gives the error all over again and you have to reapply the permissions mentioned and restart service.

[Firegarden]

Love for others to be talking about getting 3.18 release build to RUN OUT OF THE BOX ON FRESH UBUNTU 16.04.02

Steps to reproduce

1. Install Ubuntu 16.04 LTS
2. update, upgrade and install nginx
3. install hhvm
4. experience 502 gateway on all php scripts

Maybe I am the only one up right now but I am not sleeping because I am counting on yeah you guessed it HHVM on Ubuntu

The FIX or SOLUTION to get this RESOLVED is to install HHVM 3.17.3
or
sudo chown www-data:www-data /var/www after every reboot

EDIT: I installed 3.17.3 but it gave me a similar problem. Now I find when I reboot the machine it gives me the error and yet when I restart hhvm it works.

[LZL0]

Dunno @Firegarden my www folder is owned by root, while HHVM runs as www-data. Although the server cannot add/write files it works perfectly (and provides optimal file security). Of course I gave www-data the folders which are used to upload files to and which are there for caching the site.

please try& open your /etc/hhvm/server.ini file and replace everything with

; php options

pid = /var/run/hhvm/pid

; hhvm specific

hhvm.server.port = 8000
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc
hhvm.server.ip = 127.0.0.1
hhvm.jit_warmup_requests = 0
hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data
hhvm.repo.central.file_mode=493

After that restart the whole server. For me HHVM 3.18 works flawlessly on Ubuntu 14.04.

[mofarrell]

I have been working on this. I just found the commit introducing the issue https://github.com/facebook/hhvm/commit/9b3ae25e09ffd3217a32d1c94829a566fa6c8f16. I will be looking to get a fix patched on to 3.18, and we will release new binaries.

For now the workaround is to add the following to your configuration:

hhvm.repo.central.file_user=www-data
hhvm.repo.central.file_group=www-data
hhvm.repo.central.file_mode=493

And delete your existing repo.

[mofarrell]

I should have a fix on master for this shortly. We will cherrypick it onto 3.18.1

[LZL0]

Great news @mofarrell, Thank you!

[Firegarden]

Hey sorry for the newbie question but what do you mean by "And delete your existing repo."?

Thank you

[mofarrell]

The existing repo is the file that was having permission problems: /var/run/hhvm/hhvm.hhbc
This is roughly the equivalent to the opcache that is used in zend. The repo that is created by default, serves as a cache so we do not need to keep parsing the php files.

For more advanced use (and more performance), you can additionally enable repo authoritative mode. In that case you precompile your php into a repo and run a global optimization pass on the repo.

[LZL0]

Thanks @mofarrell for the advice.
I've found this https://docs.hhvm.com/hhvm/advanced-usage/repo-authoritative, maybe somebody else can find it useful too.

[Firegarden]

If I reboot my machine I have to manually restart HHVM otherwise the directory /var/run/hhvm does not exist.
EDIT:
I think I was doing something wrong - as soon as I added back the start up script it's now working on reboots.
sudo update-rc.d hhvm defaults

[mofarrell]

3.18.1 is released, and should include the patch.