Vvv: MariaDB provisioning fails and causing VVV to not provision sites

[coolamit]

Expected Behavior

Fresh install of VVV would provision all custom sites specified in vvv-custom.yml and those sites would work normally as would all default sites provisioned by VVV including the VVV Dashboard.

Current Behavior

While provisioning, the script throws errors when its trying to set up MariaDB. Once the script has finished running, no sites are found to be provisioned including the VVV Dashboard.

Steps to Reproduce (for bugs)

1. Do a clean install of VVV on develop branch.
2. Once provisioning is done, check for http://vvv.test in browser.

Installation log:
https://gist.github.com/coolamit/981d3e35081a88c35a3178aa044d6e04

Your Environment

• VVV version: v3.0.0-git::develop
• VVV Git Branch: develop
• Vagrant version: 2.2.4
• VM Provider name: VirtualBox
• VM Provider version: 5.2.28
• Operating System and version: Windows 10 Pro

[tomjn]

Would it be possible to get a copy of logs/syslog? In the meantime eliminating the /vagrant share should help with this in general

[bensahagun]

Got this similar problem too when I'm trying to replicate my VVV setup on my mac (Windows setup works fine). Tried to copy my VVV folder on mac, did vagrant destroy and vagrant up --provision.
Here's the log: https://gist.github.com/benedicksahagun/cb89490385d231302302660fdf179dcb

Gist

VVV provisioning failure

VVV provisioning failure. GitHub Gist: instantly share code, notes, and snippets.

[DrewAPicture]

Did you update the box? I was hitting this too after the 3.0 release yesterday and updating the box on a fresh check out fixed it up.

[tomjn]

hmmm I should add something that makes it hard fail provision if the distribution is trusty

[tomjn]

Done! Now if it's running on trusty it'll halt provisioning and tell you what to do

[coolamit]

Did you update the box? I was hitting this too after the 3.0 release yesterday and updating the box on a fresh check out fixed it up.

Yeah it was a clean install, I had run vagrant destroy & git pull on develop branch earlier.

Here's part of the log I posted in a gist in my original post above:

==> default: Box 'varying-vagrant-vagrants/ubuntu-18.04' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'varying-vagrant-vagrants/ubuntu-18.04'
    default: URL: https://vagrantcloud.com/varying-vagrant-vagrants/ubuntu-18.04
==> default: Adding box 'varying-vagrant-vagrants/ubuntu-18.04' (v1557101534) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/varying-vagrant-vagrants/boxes/ubuntu-18.04/versions/1557101534/providers/virtualbox.box
    default: Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com
    default:
==> default: Successfully added box 'varying-vagrant-vagrants/ubuntu-18.04' (v1557101534) for 'virtualbox'!
==> default: Importing base box 'varying-vagrant-vagrants/ubuntu-18.04'...

As you can see, its not pulling trusty but bionic.

[tomjn]

No worries @coolamit I saw that, but I hadn't realised it would still attempt to provision on 14

[tomjn]

The /vagrant share removal was merged into develop, can you do a vagrant halt and a vagrant up --provision after pulling down the latest commits and verify if it makes a difference?

Also @sbrody had this issue, leaving a gist of the provision here for the record:

https://gist.github.com/sbrody/5f0a3ebd2a3b0d15333f4798045b466f

Gist

Output for Vagrant up --provision

Output for Vagrant up --provision. GitHub Gist: instantly share code, notes, and snippets.

[sbrody]

Hi - I've just run the latest version and unfortunately am getting the same problem

[tomjn]

Thanks for checking the latest commits, out of curiosity, if you halt the machine and delete your database/data folder, then reprovision, does it give the same error?

Also, if you're running the develop branch and do not have this issue, 👍 this comment, knowing how big a proportion of users are impacted is useful

[sbrody]

Hi - the error is slightly different but is still a problem with mariadb-server - do you want a log?

[tomjn]

@sbrody if you could

Sidenote: We're mulling release VVV 3 before this gets resolved so that the master branch is broken for some people, rather than all people, and release a 3.0.1 once the issues resolved/improved, thoughts?

[sbrody]

Whichever gets it up and running quicker is preferable for me.
Latest syslog and relevant terminal output here https://gist.github.com/sbrody/b9b90c374af09eba5584df86b72a8237

Gist

Vagrant mariadb error - syslog

Vagrant mariadb error - syslog. GitHub Gist: instantly share code, notes, and snippets.

[sbrody]

OK - I removed and reinstalled Vagrant and VVV and removed existing databases and it now works

[mathieuhays]

Hi I'm having the same issue and I had no luck solving it. I'm getting the same log output as @benedicksahagun.

My environment:

WordPress|develop ⇒ vagrant status
__ __ __ __
\ V\ V\ V / Varying Vagrant Vagrants
 \_/\_/\_/  v3.0.0-git::develop

Platform:   platform-darwin13.4.0 shell:/bin/zsh vagrant-hostsupdater vagrant-vbguest
Vagrant:    v2.2.4, VirtualBox: v5.2.28r130011

I tried VirtualBox v6 but this one failed right after the vhost-updater.

This is a fresh install.

My log output in case that helps: https://gist.github.com/mathieuhays/99b92d71b522623c945912907679da40

My syslog: https://gist.github.com/mathieuhays/07305a77c5bd345c12fd37c6941ccc4a

I tried to revert back to the 2.6.0 tag but the provision stopped working as explained somewhere else (regarding the PHP packages being removed).

I tried the trick of removing the database/data folder without luck.

Gist

VVV v3 provision dump

GitHub Gist: instantly share code, notes, and snippets.

Gist

syslog vvv v3

GitHub Gist: instantly share code, notes, and snippets.

[ParhamG]

Same here, I cannot get any of the VVV 3(develop) or 2.6.0 to work. I removed the database/data folder and did destroy and then up --provision and it didn't solve: https://gist.github.com/ParhamG/c70a83f5359ff945f35dd6811f4cdae8

Gist

gist:c70a83f5359ff945f35dd6811f4cdae8

GitHub Gist: instantly share code, notes, and snippets.

[tomjn]

@mathieuhays @ParhamG @benedicksahagun as an aside, what kind of filesystem are you trying to spin VVV up on? Is it an SSD/HD/USB/Network Share?

@ParhamG, v2.6 won't provision, we're only working with v3. Can you update your gist to include the full log? The VVV Splash has super useful information but it's been cut off deliberately in your log :( If you have a syslog or a mysql error log, adding those to the gist would be handy too

tip: You can add multiple files to a single gist by clicking the add file button at the bottom

[mathieuhays]

@tomjn I'm setting VVV up in my user folder on an SSD.

[tomjn]

@mathieuhays can you run the tree command on the database/data directory? same for everybody else, I want to see what the folder/file structure looks like. Add it to a gist/update the original gist

For example:

tree database/data/

The output should be a tree of folders/files with ascii/indentations

[mathieuhays]

@tomjn I've updated my "provision dump" gist to include the filestructure. I'm using Mac OS FYI. The mysqlfolder is empty (not sure what the treecommand equivalent is for mac)

[ParhamG]

@tomjn Mine is SSD/Windows 10. I have updated the gist with the full log and also added syslog, log/mysql/mariadb-bin.index and the output from tree: https://gist.github.com/ParhamG/c70a83f5359ff945f35dd6811f4cdae8

[tomjn]

@mathieuhays the tree command should be available on MacOS, you might need to install it via homebrew though. When you say your mysql folder, is that the logs/mysql or the database/data folder?

[mathieuhays]

@tomjn I'm talking about the data/mysql folder. The content of log/mysql/mariadb-bin.index is ./ON.000001

[IanMezza]

Possible Solution

After failed provisioning:

 default: Errors were encountered while processing:
    default:  mariadb-server-10.3
    default:  mariadb-server
    default: E: Sub-process /usr/bin/dpkg returned an error code (1)
    default: Main packages check and install failed, halting provision
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong

I logged in via ssh with vagrant ssh and did as follows:

sudo rm /var/lib/mysql/tc.log
/usr/bin/mysql_secure_installation

which allowed me to set the root mysql user password as root.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!

The I did vagrant reload --provision without destroying the VM. There is something in between the provisioning that didn't allow to set up MariaDB properly. Once this is fix, the rest of the process goes well.

Your Environment

• VVV version: v3.0.0-git::develop
• VVV Git Branch:develop
• Vagrant version: v2.2.4
• VM Provider name: VirtualBox
• VM Provider version: v5.2.28r130011
• Operating System and version: macOS Mojave Version 10.14.4

[ParhamG]

The solution above didn't work for me:

vagrant@vvv:~$ /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

[tomjn]

@IanMezza was this on a fresh install, or after upgrading VVV2?

[IanMezza]

This was on a fresh install.

[willenglishiv]

+1 also on a fresh install of VVV and getting the mariadb errors during provisioning.

[willenglishiv]

https://gist.github.com/willenglishiv/3794b29ff592acb69a9c20a63b60309d

[tomjn]

This PR https://github.com/Varying-Vagrant-Vagrants/VVV/pull/1805 adds a new option to vvv-custom.yml that disables the synced folder for databases. If general: db_share_type is set to false, it won't attempt to mount the database/data folder, and instead use the internal VM as it did in VVV 2.

This does mean you would need to back up the databases and restore them to update the box, but, it should sidestep most of the issues here.

Ideally I'd like to figure out a solution, but people have jobs to do, so lets take a blocker and turn it into just an inconvenience. I've tested the PR locally, but I'd like to see if it helps someone who is unable to provision at the moment

/cc @willenglishiv @ParhamG @mathieuhays

To switch to that PR:

git fetch origin
git checkout modify/dbsharedoptions

[ParhamG]

@tomjn That didn't work for me, not sure if it is the same error though: https://gist.github.com/ParhamG/522a6b819f6c5c26c9f8c1c02246eb62

I also tried develop earlier today with VBox 6.06 and no luck: https://gist.github.com/ParhamG/a132ce7652af993be29c66f32c7386a4

Gist

VVV3-db_share_type, VBox 5.2.28

VVV3-db_share_type, VBox 5.2.28. GitHub Gist: instantly share code, notes, and snippets.

Gist

VVV 3, VBox 6.0.6

VVV 3, VBox 6.0.6. GitHub Gist: instantly share code, notes, and snippets.

[tomjn]

@ParhamG was that a fresh install or reprovision of an existing install? If it was a reprovision, was the VM already up when you ran the commands?

I noticed this in the log:

 default: /var/lib/mysql => C:/vvv/database/data

Which indicates the DB folder is mounted, which the option is supposed to prevent

edit: I see you're running with a customfile:

Running Custom Vagrant file with additional vagrant configs at C:/vvv/Customfile

Can you update your gist with that file? What does it do?

[willenglishiv]

@tomjn I'm getting the same error as @ParhamG with the mariadb failing. log incoming.

https://gist.github.com/willenglishiv/c0207534c739a71dd0ff9c67a3bd0651

[tomjn]

hmmm that was a fresh provision, with the database stored entirely inside the VM, yet it still failed to provision 🤔

[tomjn]

@willenglishiv can you do a vagrant destroy then modify the vagrant file, specifically line 562, and change it from this:

  config.vm.synced_folder "log/", "/var/log", owner: "root", group: "syslog", mount_options: [ "dmode=777", "fmode=666" ]

to this:

#  config.vm.synced_folder "log/", "/var/log", owner: "root", group: "syslog", mount_options: [ "dmode=777", "fmode=666" ]

Then reprovision and tell me if you have any luck?

[mathieuhays]

@tomjn I commented out the line in the vagrantFile. My provision is not 100% done yet but it definitely didn't fail on the mariadb install.

Edit: Yep. The provision completed successfully. (using the branch you mentioned earlier)

[ParhamG]

@tomjn My bad, I had added general: db_share_type: false to vvv-config.yml instead of vvv-custom.yml. Fixing that and reprovisioning didn't help though. I am now trying the vagrantfile mod above. I do vagrant destroy and then vagrant up -- provision every time.

[tomjn]

For reference it should look like this in vvv-custom.yml:

# General VVV options
general:
  # Backup the databases to the database/backups subfolder on halt/suspend/destroy, set to false to disable
  db_backup: true
  # Import the databases if they're missing from backups
  db_restore: true
  db_share_type: false

[ParhamG]

@tomjn Provisioning finished successfully! I am now trying to restore from the backups in database/backup using vagrant ssh -c "db_restore" and I get:

Start MariaDB Database Import
No custom databases to import
Connection to 127.0.0.1 closed.

Am I missing something?

[tomjn]

it'll be trying to restore the files from database/backups folder, if there's nothing in that folder then there's nothing to import. The backups themselves are just sql files so PHPMyAdmin or any MySQL client can import them, the db_restore script just loops through the folder feeding the SQL files into mariadb

As an aside, was this with the line commented out that I mentioned? Or just the db_share_type parameter?

[ParhamG]

I have .sql backups in there, not sure why it is not picking them up. I will try manual import using PHPMyAdmin.

This was using both fixes, the commented out line and the db_share_type parameter.

I went through the upgrade and then all these issues because I was getting 504 Gateway Time-out when I tried to access the dashboard and it is still the same :( do you have any idea on this?

[tomjn]

A 504 gateway timeout seems unrelated to any of the issues reported here so far, and looks like a new issue.

Keep in mind that disabling the logs folder mapping by commenting out that line of the vagrantfile is not a permanent fix, it also means you no longer have access to any of the logs. It's purely a debugging tactic.

As for the 504 gateway issue, this is too generic an error code to really know what happened. The best I can suggest without more information is to try turning it off and on again. Because of the commented out vagrantfile line, you'd have no access to any logs for debugging purposes. I'd suggest undoing the vagrant file change and reprovisioning

[tomjn]

That PR has been merged into develop so you can switch back to the develop branch

[ParhamG]

Thanks @tomjn . Switched back to develop and provisioned and everything looks good.

As for the databases not being imported looks like it is now expecting init-custom.sql and the backups folder in database/sql/. Moving them there fixed the import.

[tomjn]

@ParhamG so that's with the log folder share commented out and the DB inside the VM?

As for the databases not being imported looks like it is now expecting init-custom.sql and the backups folder in database/sql/. Moving them there fixed the import.

Ah, that makes sense :) I think it's worth looking into moving that folder in the vagrant file

[ParhamG]

I restored Vagrantfile to its original state and the log folder share is not commented out anymore, I guess that was only required for the initial provisioning to work? db_share_type is still set to false.

I guess I should also try starting fresh with the log folder share commented out but db_share_type: true?

[tomjn]

I made a PR to auto-move the backups folder to the right location https://github.com/Varying-Vagrant-Vagrants/VVV/pull/1808

[tomjn]

@ParhamG recent changes to the develop branch mean the mysql log is no longer on a shared/mapped folder, you might be able to set db_share_type to true and provision, but no guarantee

[mathieuhays]

I managed to have it working with db_share_type to false and the log line commented out. I tried again with db_share_type removed (defaulting to true then I guess) and it started failing again with the same errors as above.

[tomjn]

@mathieuhays can you undo the vagrantfile change and update to the latest develop branch commit, then try again with db_share_type set to false?

[mathieuhays]

@tomjn It's working :-)

[willenglishiv]

@tomjn downloading latest develop branch and db_share_type: false full provision. What's the downside now? I have to back up my own databases before starting the VM back up? same with the log files?

[tomjn]

@willenglishiv the goal of having a shared db data folder is that you can destroy the VM without loosing data. With db_share_type set to false, this is no longer the case as the database is stored inside the VM. This means when you update your box you'll need to backup beforehand, then restore afterwards. The db_backup and db_restore options should back everything up on a vagrant halt if set to true. This isn't any different to VVV 2.

Note that this doesn't impact a normal vagrant up/provision/halt/suspend/resume.

Ideally, keeping that set to false is a temporary stopgap until the underlying problem is identified and solved.

[exlame]

db_share_type to false is working well on Windows 10 Home with fresh installs. This temporary fix is all we need because DB backups are enough for now to have a working solution. Thanks for your job on vvv3 and your fast fixes.

[tomjn]

I think for now the best approach to getting this resolved is to get a machine with this issue infront of either myself or @LoreleiAurora. I'll be at WCEU at the contributor day doing VVV things, as will a few other VVV contributors. Or if anybody knows of a machine or configuration tweak that can guarantee replicating the problem. It's difficult to remote debug

Thanks for your job on vvv3 and your fast fixes.

Thanks 👍 and thank you for helping test!

[coolamit]

Fresh install on develop branch with db_share_type: false works.

[Mte90]

I have the same issue and investigating.

Seems that the user with id doesn't exist 9001 so this is blocking to set the permissions.

[tomjn]

That's due to a recent change, you'll need to recreate the VM again to use
develop/3.1

On Thu, 13 Jun 2019 at 10:37, Daniele Scasciafratte <
[email protected]> wrote:

I have the same issue and investigating.

Seems that the user with id doesn't exist 9001 so this is blocking to set
the permissions.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/Varying-Vagrant-Vagrants/VVV/issues/1773?email_source=notifications&email_token=AAAOLZZWPWPRENZDS5SH4VTP2IILXA5CNFSM4HLHYVP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXTD4ZA#issuecomment-501628516,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAOLZ6DWHS5PQH5QCOZ35LP2IILXANCNFSM4HLHYVPQ
.

[Mte90]

Confirmed a destroy fix the issue, maybe we need an alert inside VVV?

[tomjn]

There's no way to detect when to show an alert, we only know the current setup, not the past

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.