Docker-sync: Unison extremely slow for MySQL imports

[brucemead]

When importing an MySQL dump into my MySQL container I am struggling to get over 10-100KiB/s compared to my native 1-3MiB/s. Without docker-sync I get around 50-100KiB/s so clearly something with my setup is incorrect.

Any help would be greatly appreciated.

docker-sync.yml
~
version: "2"
options:
compose-file-path: 'docker-compose.yml'
compose-dev-file-path: 'web/.docker-compose-project-override.yml'
verbose: true
syncs:
mysql-data-sync:
src: './data/mysql'
dest: '/var/lib/mysql'
sync_host_port: 10871
mysql-config-sync:
src: './config/mysql'
dest: '/etc/mysql'
sync_host_port: 10872
mysql-log-sync:
src: './logs/mysql'
dest: '/var/log/mysql'
sync_host_port: 10873
web-sync:
src: './web'
dest: '/var/www/html/web'
sync_excludes: ['.git', 'var/cache', 'var/full_page_cache', 'var/session', '.idea']
sync_host_port: 10874
~

docker-compose.yml
~
version: "2"
services:
installer:
image: vortexcommerce/magento-installer:beta2
volumes_from:
- nginx
volumes:
- ./config/installer/php/php.ini:/usr/local/etc/php/php.ini
- ./config/installer/magerun/n98-magerun.yaml.tmpl:/etc/n98-magerun.yaml.tmpl
- ./config/installer/magento/local.xml.tmpl:/etc/local.xml.tmpl
- ./config/installer/magento/fpc.xml.tmpl:/etc/fpc.xml.tmpl
- ./config/installer/bin/substitute-env-vars.sh:/bin/substitute-env-vars.sh
- ./config/installer/bin/install.sh:/bin/install.sh
links:
- "cache:rediscache"
- "sessions:redissession"
- "fullpagecache:redisfullpagecache"
- "mysql:mysql"
entrypoint: /bin/install.sh
nginx:
image: nginx:latest
ports:
- "80:80"
- "443:443"
links:
- "php:php"
volumes:
- ./logs/nginx:/var/log/nginx
- ./config/ssl/cert.pem:/etc/nginx/ssl/cert.pem
- ./config/ssl/cert.key:/etc/nginx/ssl/cert.key
- ./config/nginx/nginx.conf:/etc/nginx/nginx.conf
- ./config/nginx/sites-enabled/default.conf:/etc/nginx/conf.d/default.conf
- ./config/nginx/includes:/etc/nginx/includes
- ./config/nginx/custom-locations:/etc/nginx/custom-locations
volumes_from:
- php
php:
image: vortexcommerce/php:7.0-beta1
links:
- "cache:rediscache"
- "sessions:redissession"
- "fullpagecache:redisfullpagecache"
- "mysql:mysql"
volumes:
- web-sync:/var/www/html/web:rw
mysql:
image: mysql:5.7
ports:
- "3306:3306"
volumes:
- mysql-data-sync:/var/lib/mysql:rw
- mysql-config-sync:/etc/mysql:rw
- mysql-log-sync:/var/log/mysql:rw
cache:
image: redis:latest
fullpagecache:
image: redis:latest
sessions:
image: redis:latest
volumes:
mysql-data-sync:
external: true
mysql-config-sync:
external: true
mysql-log-sync:
external: true
web-sync:
external: true
~

web/.docker-compose-project-override.yml
~
version: "2"
services:
installer:
image: vortexcommerce/magento-installer:m1-0.1
php:
image: vortexcommerce/php:7.0.14 # Production has 7.0.8
links:
- "elasticsearch:elasticsearch"
mysql:
image: mysql:5.5.54 # Production has 5.5.38
elasticsearch:
image: vortexcommerce/elasticsearch:1.5.2
ports:
- "9200:9200"
- "9300:9300"
~

[EugenMayer]

If you are syncing /var/lib/mysql you are doing it wrong conceptually .. in you case you are completely missunderstanding docker sync and docker volumes in general. Please search the issue queue for a similar question with a brief answer

[henscu]

@brucemead, @EugenMayer is referencing #148. I also did not realise that MYSQL is not a good use case for docker-sync. I will change my WordPress setup to only use docker-sync for my html text files.

[EugenMayer]

**but in general, syncing /var/lib/mysql is not a bad use-case for docker-sync, it is wrong for more or less anything you do.

• It will be wrong for production use under linux ( host mount, slower ) use named volumes here
• it will be unbelievable slow with d4m, with dingy whatsover. For anything not being docker-sync, it will be even worse

---

Beside that, it is wrong conceptually in any way you look at it.

1. if you want to persisten the mysql /var/lib/mysql folder over container-lifecycle, use named volumes. So basically "reusage" over docker-compose down / up ( down does not remove named volumes )
2. If you are syncing it back to the host so you can "reuse" it in production on a different server ( push it production ) you will soon or less, blow it up. Never ever sync mysql-dbs using /var/lib/mysql - always use dumps. This is a complete nogo

Those are the only scenarios i can think of, and both are not at all solved using a host-mount ever. And host-mount means docker-sync, yeah and things become worse.

You btw. do not want to sync log either, and nothing else.

@mickaelperrin had a similar question back in days, use this rule of thump:

Only sync data which is persistent ( as code ) - do not sync data you can generate or will through away after a container lifecycle ( like logs ). If you want to have a look at the logs do:

docker exec -it <containername> tail -f /var/log/php.log

in a shell

Advanced topics in the same area:
Look at my answers on stackoverlfow:

• http://stackoverflow.com/questions/41585397/deploy-web-app-in-docker-data-container-vs-volume/41585585#41585585
• http://stackoverflow.com/questions/41566244/can-containers-share-a-framework/41576040#41576040
• http://stackoverflow.com/questions/41480336/advantages-of-docker-layer-over-plain-os/41480486#41480486