Homestead: Homestead as a replacement for lullabot/trusty32-vm

[deviantintegral]

I'm considering abandoning the https://github.com/Lullabot/trusty32-lamp vagrant box I maintain in favour of Homestead, as it's built on Ubuntu 14.04 which EOL's soon. For details, see https://github.com/Lullabot/trusty32-lamp/issues/50

To reach feature parity, there's a few items we'd need to add to Homestead.

Database Snapshots

One of the core features the the box includes is fast copy-on-write branching and snapshots of MySQL databases. We often deal with large databases in local development, and being able to quickly roll back without a full table or database import / export is a huge time saver. It works with a combination of a thinly-provisioned LVM volume and the Logical MySQL Manager set of scripts.

The good news is that Homestead already uses LVM, but the volumes use all of the PV space. Since Vagrant uses VMDKs and not VDIs, Virtualbox can't expand the volume, and reducing the root logical volume would require booting into single-user mode and remounting / readonly.

Performance profiling

Blackfire.io is included with Homestead, but it's very expensive ($29/USD a month) if you want to trigger a profile from your browser. You can profile from the CLI, but that's tricky when submitting form POSTs that may have many fields, requires escaping, and so on. xhgui is the tool I've used before, and Homestead already includes Mongo support.

mDNS support

Supporting mDNS means you don't have to deal with /etc/hosts hacking in most cases. It's also very useful for on-device mobile testing where you'd have to edit DNS records since you can't edit a hosts file. I found https://github.com/laravel/homestead/pull/103, but I don't think we need that much code. Simply installing avahi-daemon exposes the hostname as hostname.local automatically. I was looking through the repo for where base packages are managed, but all I could find were scripts for optional packages.

Docs for xdebug

xdebug is included, but not documented on how to use it. I know this isn't homestead specific, but I think documenting how to debug both browser and CLI requests would be helpful to end users.

Before I start on any of these, could I get a 👍 or 👎 ? I think xhgui and mdns could be optional, but I don't see a way to support database snapshots without altering the base box itself.

[svpernova09]

I'm 👍 on all these but somewhat concerned about modifying the volumes. I don't want to change the build process for Homestead https://github.com/laravel/settler in regards to running 1 packer command to build a base box to verify.

As long as we can maintain the simple build process we have today I'm 👍

Feel free to submit PRs to Settler but since we don't enable issues there any discussion needed can be hashed out in an issue here.

[svpernova09]

I know you're still waiting on me to test laravel/settler#194. Thanks for your patience!

[svpernova09]

Do you need anything else in Settler to satisfy this or is it safe for me to build a new base box?

[deviantintegral]

I think we're good to go! I'm going to work on some docs improvements at https://github.com/laravel/docs/blob/5.8/homestead.md today - I'd like to leave this open until that's done.

[svpernova09]

Sure no worries! I just wanted to check to make sure you weren't waiting on me. I think I may build a new version of settler and release over the weekend. That way you'd be able to test anything you may need in this repo (If you're using it)

[svpernova09]

@deviantintegral,

Here's what I'm running into:

MacOS + Parallels, This also happens on WIndows 10 + Hyper-V, yet VMware throws a packer bug (haven't dug into it yet)

Any suggestions? This didn't happen on Virtualbox, Running another build to verify.

    parallels-iso: Preparing to unpack .../05-libfcgi-perl_0.78-2build1_amd64.deb ...
    parallels-iso: Unpacking libfcgi-perl (0.78-2build1) ...
    parallels-iso: Selecting previously unselected package libcgi-fast-perl.
    parallels-iso: Preparing to unpack .../06-libcgi-fast-perl_1%3a2.13-1_all.deb ...
    parallels-iso: Unpacking libcgi-fast-perl (1:2.13-1) ...
    parallels-iso: Selecting previously unselected package libencode-locale-perl.
    parallels-iso: Preparing to unpack .../07-libencode-locale-perl_1.05-1_all.deb ...
    parallels-iso: Unpacking libencode-locale-perl (1.05-1) ...
    parallels-iso: Selecting previously unselected package libhtml-template-perl.
    parallels-iso: Preparing to unpack .../08-libhtml-template-perl_2.97-1_all.deb ...
    parallels-iso: Unpacking libhtml-template-perl (2.97-1) ...
    parallels-iso: Selecting previously unselected package libhttp-date-perl.
    parallels-iso: Preparing to unpack .../09-libhttp-date-perl_6.02-1_all.deb ...
    parallels-iso: Unpacking libhttp-date-perl (6.02-1) ...
    parallels-iso: Selecting previously unselected package libio-html-perl.
    parallels-iso: Preparing to unpack .../10-libio-html-perl_1.001-1_all.deb ...
    parallels-iso: Unpacking libio-html-perl (1.001-1) ...
    parallels-iso: Selecting previously unselected package liblwp-mediatypes-perl.
    parallels-iso: Preparing to unpack .../11-liblwp-mediatypes-perl_6.02-1_all.deb ...
    parallels-iso: Unpacking liblwp-mediatypes-perl (6.02-1) ...
    parallels-iso: Selecting previously unselected package libhttp-message-perl.
    parallels-iso: Preparing to unpack .../12-libhttp-message-perl_6.14-1_all.deb ...
    parallels-iso: Unpacking libhttp-message-perl (6.14-1) ...
    parallels-iso: Selecting previously unselected package mysql-server.
    parallels-iso: Preparing to unpack .../13-mysql-server_5.7.25-0ubuntu0.18.04.2_all.deb ...
    parallels-iso: Unpacking mysql-server (5.7.25-0ubuntu0.18.04.2) ...
    parallels-iso: Setting up libhtml-tagset-perl (3.20-3) ...
    parallels-iso: Setting up libhttp-date-perl (6.02-1) ...
    parallels-iso: Setting up libevent-core-2.1-6:amd64 (2.1.8-stable-4build1) ...
    parallels-iso: Processing triggers for ureadahead (0.100.0-20) ...
    parallels-iso: Setting up libencode-locale-perl (1.05-1) ...
    parallels-iso: Setting up libio-html-perl (1.001-1) ...
    parallels-iso: Setting up liblwp-mediatypes-perl (6.02-1) ...
    parallels-iso: Processing triggers for libc-bin (2.27-3ubuntu1) ...
    parallels-iso: Setting up libaio1:amd64 (0.3.110-5) ...
    parallels-iso: Setting up liburi-perl (1.73-1) ...
    parallels-iso: Processing triggers for systemd (237-3ubuntu10.20) ...
    parallels-iso: Setting up libhtml-parser-perl (3.72-3build1) ...
    parallels-iso: Setting up libcgi-pm-perl (4.38-1) ...
    parallels-iso: Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    parallels-iso: Setting up libhttp-message-perl (6.14-1) ...
    parallels-iso: Setting up mysql-client-core-5.7 (5.7.25-0ubuntu0.18.04.2) ...
    parallels-iso: Setting up libfcgi-perl (0.78-2build1) ...
    parallels-iso: Setting up libhtml-template-perl (2.97-1) ...
    parallels-iso: Setting up mysql-server-core-5.7 (5.7.25-0ubuntu0.18.04.2) ...
    parallels-iso: Setting up libcgi-fast-perl (1:2.13-1) ...
    parallels-iso: Setting up mysql-client-5.7 (5.7.25-0ubuntu0.18.04.2) ...
    parallels-iso: Setting up mysql-server-5.7 (5.7.25-0ubuntu0.18.04.2) ...
    parallels-iso: update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
    parallels-iso: Renaming removed key_buffer and myisam-recover options (if present)
    parallels-iso: Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
    parallels-iso: Setting up mysql-server (5.7.25-0ubuntu0.18.04.2) ...
    parallels-iso: Processing triggers for libc-bin (2.27-3ubuntu1) ...
    parallels-iso: Processing triggers for systemd (237-3ubuntu10.20) ...
    parallels-iso: Processing triggers for ureadahead (0.100.0-20) ...
    parallels-iso: + apt-get install -y thin-provisioning-tools bc
    parallels-iso: Reading package lists...
    parallels-iso: Building dependency tree...
    parallels-iso: Reading state information...
    parallels-iso: bc is already the newest version (1.07.1-2).
    parallels-iso: The following NEW packages will be installed:
    parallels-iso:   thin-provisioning-tools
    parallels-iso: 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
    parallels-iso: Need to get 359 kB of archives.
    parallels-iso: After this operation, 1,352 kB of additional disk space will be used.
    parallels-iso: Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 thin-provisioning-tools amd64 0.7.4-2ubuntu3 [359 kB]
    parallels-iso: Fetched 359 kB in 1s (408 kB/s)
    parallels-iso: Selecting previously unselected package thin-provisioning-tools.
    parallels-iso: (Reading database ... 139319 files and directories currently installed.)
    parallels-iso: Preparing to unpack .../thin-provisioning-tools_0.7.4-2ubuntu3_amd64.deb ...
    parallels-iso: Unpacking thin-provisioning-tools (0.7.4-2ubuntu3) ...
    parallels-iso: Setting up thin-provisioning-tools (0.7.4-2ubuntu3) ...
    parallels-iso: Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    parallels-iso: + git clone -b ubuntu-18.04 https://github.com/Lullabot/lmm.git /opt/lmm
    parallels-iso: Cloning into '/opt/lmm'...
    parallels-iso: + ln -s /opt/lmm/lmm /usr/local/sbin/lmm
    parallels-iso: + mkdir -p /vagrant-vg/master
    parallels-iso: + lvcreate -L 40G -T vagrant-vg/thinpool
    parallels-iso:   Using default stripesize 64.00 KiB.
    parallels-iso:   Volume group "vagrant-vg" not found
    parallels-iso:   Cannot process volume group vagrant-vg
==> parallels-iso: Unregistering virtual machine...
==> parallels-iso: Deleting output directory...
Build 'parallels-iso' errored: Script exited with non-zero exit status: 5

==> Some builds didn't complete successfully and had errors:
--> parallels-iso: Script exited with non-zero exit status: 5

==> Builds finished but no artifacts were created.
packer build --only=parallels-iso ubuntu-18.04-amd64.json: 17:35

[svpernova09]

Confirmed working on Virtualbox:

    virtualbox-iso: + apt-get install -y thin-provisioning-tools bc
    virtualbox-iso: Reading package lists...
    virtualbox-iso: Building dependency tree...
    virtualbox-iso: Reading state information...
    virtualbox-iso: bc is already the newest version (1.07.1-2).
    virtualbox-iso: The following NEW packages will be installed:
    virtualbox-iso:   thin-provisioning-tools
    virtualbox-iso: 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
    virtualbox-iso: Need to get 359 kB of archives.
    virtualbox-iso: After this operation, 1,352 kB of additional disk space will be used.
    virtualbox-iso: Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 thin-provisioning-tools amd64 0.7.4-2ubuntu3 [359 kB]
    virtualbox-iso: Fetched 359 kB in 1s (433 kB/s)
    virtualbox-iso: Selecting previously unselected package thin-provisioning-tools.
    virtualbox-iso: (Reading database ... 139307 files and directories currently installed.)
    virtualbox-iso: Preparing to unpack .../thin-provisioning-tools_0.7.4-2ubuntu3_amd64.deb ...
    virtualbox-iso: Unpacking thin-provisioning-tools (0.7.4-2ubuntu3) ...
    virtualbox-iso: Setting up thin-provisioning-tools (0.7.4-2ubuntu3) ...
    virtualbox-iso: Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
    virtualbox-iso: + git clone -b ubuntu-18.04 https://github.com/Lullabot/lmm.git /opt/lmm
    virtualbox-iso: Cloning into '/opt/lmm'...
    virtualbox-iso: + ln -s /opt/lmm/lmm /usr/local/sbin/lmm
    virtualbox-iso: + mkdir -p /vagrant-vg/master
    virtualbox-iso: + lvcreate -L 40G -T vagrant-vg/thinpool
    virtualbox-iso:   Using default stripesize 64.00 KiB.
    virtualbox-iso:   Thin pool volume with chunk size 64.00 KiB can address at most 15.81 TiB of data.
    virtualbox-iso:   Logical volume "thinpool" created.
    virtualbox-iso: + lvcreate -V10G -T vagrant-vg/thinpool -n mysql-master
    virtualbox-iso:   Using default stripesize 64.00 KiB.
    virtualbox-iso:   Logical volume "mysql-master" created.
    virtualbox-iso: + mkfs.ext4 /dev/vagrant-vg/mysql-master
    virtualbox-iso: mke2fs 1.44.1 (24-Mar-2018)
    virtualbox-iso: Discarding device blocks: done
    virtualbox-iso: Creating filesystem with 2621440 4k blocks and 655360 inodes
    virtualbox-iso: Filesystem UUID: 792e01cb-a9c7-4357-9970-d613d41f8234
    virtualbox-iso: Superblock backups stored on blocks:
    virtualbox-iso:     32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
    virtualbox-iso:
    virtualbox-iso: Allocating group tables: done
    virtualbox-iso: Writing inode tables: done
    virtualbox-iso: Creating journal (16384 blocks): done
    virtualbox-iso: Writing superblocks and filesystem accounting information: done
    virtualbox-iso:
    virtualbox-iso: + echo /dev/vagrant-vg/mysql-master\t/vagrant-vg/master\text4\terrors=remount-ro\t0\t1
    virtualbox-iso: + mount -a
    virtualbox-iso: + chown mysql:mysql /vagrant-vg/master
    virtualbox-iso: + systemctl stop mysql

If we can't figure out what's causing it we'll need to roll back the changes.

[deviantintegral]

That's really surprising, given that all of the lvm configuration is inside the VM and should be using the same config from https://github.com/laravel/settler/blob/master/http/preseed.cfg. I wonder where the string vagrant-vg comes from? Perhaps that name can change depending on the provider. Can you throw a sudo lvs right before that lvcreate command to see what the configs are?

I'm heading away for the long weekend, but can help dive into this when I'm back Tuesday.

[svpernova09]

Yeah, looks like it was just the wrong volume group name:

    parallels-iso: + sudo lvs
    parallels-iso:   LV     VG           Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
    parallels-iso:   root   ubuntu-18-vg -wi-ao----  17.67g
    parallels-iso:   swap_1 ubuntu-18-vg -wi-ao---- 976.00m

[svpernova09]

It looks like the volume group names vary by provider, which means this is going to be really fun...

I changed the Volume Group to for parallelsubuntu-18-vg and now Virtualbox fails. Looks like Parallels is getting ubuntu-18-vg for the Volume Name, and Virtualbox it's vagrant-vg

[svpernova09]

@deviantintegral, got it figured out! Shouldn't be an issue any longer, just needed to set the volume group name in the preseed.

[svpernova09]

Just finished releasing https://github.com/laravel/homestead/releases/tag/v8.3.0 which requires the base box v7.2.1 which has all of the changes we've been working on. Thanks a ton for the effort. Closing this as complete for now. If we discover anything broken feel free to open a new issue.