Ddev: Pantheon ddev pull fails after download, during import.

[ultimike]

What happened (or feature request):

I tried to do a ddev pull, got the following error:

~/Sites/artist [master*]$ ddev pull
You're about to delete the current database and files and replace with a fresh import. Would you like to continue (y/N): y
Importing database... 
Could not perform import: Failed to run docker-compose [-f /Users/michael/Sites/artist/.ddev/docker-compose.yaml exec -T db bash -c cat /db/*.sql | mysql], err='exit status 1', stdout='ERROR 1227 (42000) at line 6894: Access denied; you need (at least one of) the SUPER privilege(s) for this operation
', stderr='' 

I had just done a ddev start on an existing codebase that I hadn't used DDEV for previously.

What you expected to happen:

The pull to be complete.

How to reproduce this:

I could share the project with someone, I suppose...

Version: Please include the output of ddev version, docker version and the project's .ddev/config.yaml.

~/Sites/artist [master*]$ ddev version
domain  ddev.local                     
cli     v0.13.1                        
web     drud/nginx-php-fpm-local:v0.9.3
db      drud/mariadb-local:v0.7.1      
dba     drud/phpmyadmin:v0.2.0         
router  drud/ddev-router:v0.5.0        
commit  v0.13.1  

~/Sites/artist [master*]$ docker version
Client:
 Version:   17.12.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    c97c6d6
 Built: Wed Dec 27 20:03:51 2017
 OS/Arch:   darwin/amd64

Server:
 Engine:
  Version:  17.12.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   c97c6d6
  Built:    Wed Dec 27 20:12:29 2017
  OS/Arch:  linux/amd64
  Experimental: true

~/Sites/artist [master*]$ cat .ddev/config.yaml 
APIVersion: "1"
name: artist
type: drupal7
docroot: ""
php_version: "7.1"
webimage: drud/nginx-php-fpm-local:v0.9.3
dbimage: drud/mariadb-local:v0.7.1
dbaimage: drud/phpmyadmin:v0.2.0
router_http_port: "80"
router_https_port: "8443"
provider: pantheon

# Certain ddev commands can be extended to run tasks before or after the ddev
# command is executed.
# See https://ddev.readthedocs.io/en/latest/users/extending-commands/ for more
# information on the commands that can be extended and the tasks you can define
# for them.
# hooks:
#   post-import-db:
#     - exec: "drush cc all"

Anything else do we need to know:

Nothing comes to mind.

Related source links or issues:

I saw https://github.com/drud/ddev/issues/466, but it doesn't sound like the same thing.

Please use a complexity rating of 1-5 (5 is high) for a feature request. (High complexity implies more PR planning)

[rfay]

Wow, that looks like it's trying to use a different database than "db" in the "db" container. Please check to see that the settings.php is in fact loading settings.local.php, where we overload it, and that the settings.local.php has our contents in it - it should say #ddev-generated in there. But I'm thinking that all happens after the db is loaded.

One thing to try is to rm -r .ddev, and then do the ddev config pantheon again. In case there's something wrong with the docker-compose.yml. I don't think that's the case at all.

I'm happy to take a look if it's appropriate to share with my pantheon account, randy at randyfay.com

[ultimike]

Well - I think you're on the right track because the settings.php doesn't have anything about the settings.local.php, nor is there a settings.local.php (did I mention this is a Drupal 7 site?)

Still make sense to remove .ddev and re-config it?

-mike

[rfay]

It doesn't do any harm to do re-config. Doesn't do any harm to study it from the settings.php end either.

I'm pretty baffled by that error message though. Almost sounds like you have a pantheon dump that is specifying a particular database name.

One thing I'd try is doing a new archive/dump/backup of both files and db on Pantheon's end.

[ultimike]

At what point should the settings.local.php appear? During the ddev config pantheon step? If so, it's not happening...

Once it was complete, I did a ddev start, then a ddev pull, with the same result as above.

I checked, and it looks like the DB was imported, so I manually created a settings.local.php file and now everything appears to be working.

One caveat that may explain the weirdness: this site has some SQL views that get created via a custom drush command. I have had issues importing this database previously (I didn't think about that until today).

It might make sense to just close this issue, as it is entirely possible the site's DB SQL views are to blame.

[rfay]

It actually appears after the import of the db, and I don't think you're getting that far.

sql views sounds like a great culprit! Again, if it's possible and you're willing to share the db or the pantheon site, I'll try it.

I wonder if we need some extra permissions for the views? Is it possible the views refer to a different db?

Thanks for all your work on this.

[ultimike]

My client isn't keen on sharing the codebase - sorry.

The SQL Views have always been a bit of a pain to deal with - including some weird permission stuff. Would it explain the settings.local.php not being generated as well?

[rfay]

Yes, the failure to load means that ddev never gets to the point where it generates the settings.local.php.

It's a lot to ask, but if you know how to put together a random database that generates this, I'd love to figure out how to solve this. But it does seem like you've got it here.

We can close this when it works for you, but my inexperience with sql views makes me insecure about the reason for this failure. Glad it's working for you though!

[ultimike]

I wish I could say that I had to the time to put together a quick DB for you, but I'm pretty swamped for the next few weeks... :(

I can tell you that we have a drush command to create the SQL views that looks something like this:

function mymodule_drush_command() {
  $items = array();

  // The 'create-sql-views' command
  $items['create-sql-views'] = array(
    'description' => "Re-create SQL Views.",
  );
  return $items;
}
function drush_mymodule_create_sql_views() {
  _mymodule_create_sql_views();
}

function _mymodule_create_sql_views() {
  $sql = "CREATE OR REPLACE VIEW {this_is_the_sql_view_name} AS select blah blah blah;";
  db_query($sql);
};

Hopefully that's enough to get you going.

[rfay]

Thanks!

[rfay]

My bet is that this will be solved in https://github.com/drud/ddev/pull/693

[rfay]

Closing this one unless we hear otherwise, assuming #693 solves it. Note that this was a D7 site though, which by default would not have the include of settings.local.php, which could cause this kind of breakage. Happy to reopen!

[texvetadmin]

Howdy! I just recreated this issue. It's (almost) exactly the same (The line that the error occurs on differs from the original poster's issue.):

TexVet-Leistiko-iMac:texvet jonathanleistiko$ ddev pull
You're about to delete the current database and files and replace with a fresh import. Would you like to continue (y/N): y
Importing database... 
ERROR 1227 (42000) at line 2639: Access denied; you need (at least one of) the SUPER privilege(s) for this operation 
Could not perform import: Failed to run docker-compose [-f /Users/jonathanleistiko/Documents/local_dev/ddev_texvet_local/texvet/.ddev/docker-compose.yaml exec -T db bash -c cat /db/*.sql | mysql -udb -pdb db], err='exit status 1', stdout='', stderr='ERROR 1227 (42000) at line 2639: Access denied; you need (at least one of) the SUPER privilege(s) for this operation' 
TexVet-Leistiko-iMac:texvet jonathanleistiko$ 

Docker is running. I'm certain that the DB's name is "pantheon" – Pantheon's Connection Info > Database > DB Name says so.

My organization is willing and able to share access to our Pantheon site/database/etc. In the meantime, I'm working through the notes above (and suggestions in other articles) to see if I can get the local copy of the site working correctly.

Extra info that may or may not be relevant: We did not set up the Pantheon site ourselves. Instead we hired Kalamuna to migrate an existing site from Blackmesh to Pantheon. Part of that included getting CiviCRM to work correctly. That involved setting up some symbolic links or somesuch.

Version:

Mac OS 10.13.4

TexVet-Leistiko-iMac:texvet jonathanleistiko$ ddev version
domain ddev.local
cli v0.17.0
web drud/nginx-php-fpm-local:v1.2.2
db drud/mariadb-local:v0.9.0
dba drud/phpmyadmin:v0.2.0
router drud/ddev-router:v0.5.0
commit v0.17.0

TexVet-Leistiko-iMac:texvet jonathanleistiko$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:13:02 2018
OS/Arch: darwin/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:22:38 2018
OS/Arch: linux/amd64
Experimental: true

Complexity Rating: 2?

[rfay]

@texvetadmin thanks for the report. If there's any possibility of sharing the pantheon site with me for debugging it would be marvelous. randy (at) randyfay.com. Does yours have any SQL views in it?

[texvetadmin]

I just shared our Pantheon site with you.
I hate to admit this, but I'm not sure what you mean when you ask if it has any SQL views in it. We're using the Views module and we have many views that make SQL queries, so I'm inclined to say "Yes, it has SQL views.," but I may be misunderstanding what you're asking.

[rfay]

Thanks for that! I'm able to recreate the issue. The problem line is

/*!50003 CREATE*/ /*!50017 DEFINER=pantheon@%*/ /*!50003 TRIGGER civicrm_activity_before_insert before insert ON civicrm_activity FOR EACH ROW BEGIN

Since we GRANT ALL to the db user, I'm studying to try to figure out what we're missing here.

[rfay]

Please try this @texvetadmin:

• ddev start
• ddev ssh and mysql -uroot -proot and GRANT SUPER ON *.* TO 'db'@'localhost' IDENTIFIED BY 'db'; and FLUSH PRIVILEGES;
• Exit back out to host
• ddev pull

I see that you have a huge db and files there. If you want to just import the db you can ddev import-db --src ~/.ddev/pantheon/texvet/texvet_dev_<latest>_*sql.gz

If this works for you (works for me) we'll consider that as just a routine change.

I do note these complaints on the file import, but neither macOS tar nor gnu-tar does any better on the tarball provided by pantheon:

Could not perform import: failed to extract provided archive: Failed to create file /Users/rfay/workspace/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php, err: open /Users/rfay/workspace/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php: file name too long

[texvetadmin]

I'm giving your recommended commands a go now. I'm up to "ddev pull"
After "thinking" for a good chunk of time, it's on "Generating settings.local.php file for database connection." I'll let you know how it goes.
I sincerely appreciate your assistance. Thank you very much!

[texvetadmin]

Short answer: It worked. Thank you!

Longer answer: It failed. Boo! But it's totally an issue specific to the files. There's a file with a file name that's too long. I have to track it down and delete it.

Importing files... 
Could not perform import: failed to extract provided archive: Failed to create file /Users/jonathanleistiko/Documents/local_dev/ddev_texvet_local/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php, err: open /Users/jonathanleistiko/Documents/local_dev/ddev_texvet_local/texvet/sites/default/files/civicrm/templates_c/en_US/%
/0A1/0A1AE998%%string:2101+S+IH-35,+Suite+216,+Austin,+TX+78741+from+2:30+to+4.

The+Power+Partners+are+hosting+a+job+club+where+you+will+learn+useful+tips+for+your+job+search.+You+can+also+talk+with+our+featured+employers+and+Veteran+Service+Organization+providers.+Feel+free+to+bring+your+resume+for+review..php: file name too long 
TexVet-Leistiko-iMac:texvet jonathanleistiko$ 

I have little doubt that once I track that down (and other ridiculous files like that one), it'll work just fine.

[rfay]

Opened drud/mariadb-local#55 to solve this generally.

BTW @texvetadmin I don't think those long filenames actually cause a failure that would be in your way; take a look and see if most everything you need isn't there in sites/default/files. Mostly with big sites like that having every single one of the files is a waste. People have requested a ddev pull --database-only in https://github.com/drud/ddev/issues/405, I imagine we'll get that going.

[rfay]

Closing this as a duplicate of https://github.com/drud/ddev/issues/852 (which was formerly https://github.com/drud/mariadb-local/issues/55). I'm pretty sure that when we start importing using the root-privileged mysql user this will be solved.

[rfay]

@texvetadmin @ultimike I think both of your problems should be fixed in today's v0.19.0 release, and hope you'll be able to try it out and confirm. @texvetadmin thanks to your generosity in sharing your pantheon project I was already able to give a quick run on yours and it worked out fine. Thanks for that!