Hhvm: HHVM fails to start on system restart if Repo Authoritative mode is enabled

[javuto]

HHVM Version

hhvm --version
HipHop VM 3.14.5 (rel)
Compiler: tags/HHVM-3.14.5-0-gc6bf714e6468213eed377527b431e9cd4cec1432
Repo schema: 9ab3fe4281c731189b558d921eb7317b0eb2b2e6

Standalone code, or other way to reproduce the problem

1. Install the HHVM package from http://dl.hhvm.com/ubuntu/pool/main/h/hhvm/hhvm_3.14.5~trusty_amd64.deb in Ubuntu 14.04.
2. Configure HHVM with the following configuration:

; php options
pid = /var/run/hhvm/pid
; hhvm specific
hhvm.server.port = 80
hhvm.server.type = proxygen
hhvm.server.default_document = index.php
hhvm.log.level = Error
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.log.header = true
hhvm.log.access[default][file] = /var/log/hhvm/access.log
hhvm.log.access[default][format] = "%h %l %u %t \"%r\" %>s %b"
hhvm.server.source_root = /var/www
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc

1. Enable HHVM Repo Authoritative mode with sudo hhvm-repo-mode enable "/var/www".
2. Check that HHVM is working properly with the following simple script as index.php:

<?php 
  print "This is a test\n";
?>

1. Restart the machine.

Expected result

HHVM service is running and the index.php is executed properly.

Actual result

HHVM service is not running. No errors in /var/log/hhvm/error.log and it fails to connect to port 80. To bring the service back up, the following commands are needed:

sudo rm /var/run/hhvm/hhvm.hhbc
sudo hhvm-repo-mode enable "/var/www"
sudo service hhvm start

After those, the service seems to be working fine.
This issue was originally noticed within the fbctf platform (https://github.com/facebook/fbctf/issues/234) but this is a much simpler way to reproduce the problem.

[javuto]

How is this a no isolated repro? I gave you the steps to reproduce, code to test and mitigation. What else do you need?

[Orvid]

I think @aorenste missed the fact that 4. included the full source. Either that or his interpretation of an "isolated" repro is a bit more narrow than it could be. Although it is isolated, it does have a dependence on the OS as well.

Either way, I've removed the tag.

[aorenste]

Oops - my bad. I misread the repro steps.

[mofarrell]

@javuto I was able to reproduce the error. Adding hhvm to the System V auto startup fixed it for me (technically ubuntu 14.04 uses upstart for system startup, but that runs all the system v startup things as well).

sudo update-rc.d hhvm defaults
sudo service hhvm restart

For newer ubuntu versions Systemd should be used. This link has a good explanation of all the possible configurations on different linux versions.

If this doesn't fix it for you comment, and I will reopen the issue.

[javuto]

Hey @mofarrell, thank you for looking at this! Adding it to the startup is something we were already doing in the fbctf platform and I can still reproduce the problem with the simpler repro steps, only if HHVM Repo Authoritative mode is enabled.

[mofarrell]

I must have messed up reproducing it before. Now I see what you mean. The sqlite repo is corrupted on reboot.

[mofarrell]

The /var/run directory is a tmpfs filesystem mounted at /run and symlinked to /var/run. It doesn't survive reboot, so the repo file present on reboot is an empty one that is created by the launched hhvm daemon. The hhvm-repo-mode script doesn't do very much to build the repo. My guess is it would be easiest to perform that work in the fbctf setup scripts. I am little confused why that script would be using the /var/run directory rather than /var/cache or something (I might change this)

[javuto]

Spot on @mofarrell, once I changed the location of the file hhvm.hhbc to be /var/cache/hhvm/hhvm.hhbc it worked perfectly after rebooting. Once thing to note is that I had to make sure the permissions were correct because otherwise the user www-data won't be able to access the file and it is logged in the error file as:

[Thu Dec  1 20:30:17 2016] [hphp] [16524:7f2226b4ffc0:0:000001] [] Failed to initialize central HHBC repository:\n  Failed to initialize schema in /var/cache/hhvm/hhvm.hhbc: \n  Failed to open /var/www/.hhvm.hhbc: 14 - unable to open database file\n

Thank you!