Dokku: bin/post_compile executes before storage is mounted

[thelonecabbage]

Description of problem:
Example case is where bin/post_compile might run some script that downloads some large amount of data off the internet. Obviously you don't want it to do this every time, so you use dokku storage to create a mount, and store the data there, so that subsequent deploys can check for the existence of the file before re-downloading.

But storage doesn't mount the paths until after bin/post_compile. Each time a deploy is triggered data is re-downloaded and then overwritten by the storage mount.

Environment details (AWS, VirtualBox, physical, etc.):
VirtualBox, Flask, Python3

How was dokku installed?:
Vagrant

How reproducible:
100%

Steps to Reproduce:

1. create a hello world flask app
2. create a bin/post_compile file

#!/usr/bin/env bash
date >> /opt/test/datetime.lst

1. dokku storage app /opt/testhost:/opt/testguest
2. git push ...

Actual Results:
/opt/testhost remains empty
/opt/testguest remains empty

Expected Results:
/opt/testhost & /opt/testguest should contain the same file with a row for each date-time that the deploy was initiated

Additional info:

[michaelshobbs]

I think a couple things are going on here. I tried to reproduce the issue and came to a different state in that the deployment actually failed. (see below)

root@dokku:/tmp/python-flask# dokku apps:create python-test
Creating python-test... done
root@dokku:/tmp/python-flask# mkdir /tmp/python-test/
root@dokku:/tmp/python-flask# dokku storage:mount python-test /tmp/python-test:/opt/test
root@dokku:/tmp/python-flask# cat bin/post_compile
#!/usr/bin/env bash
date >> /opt/test/datetime.lst

root@dokku:/tmp/python-flask# git push [email protected]:python-test master
Pseudo-terminal will not be allocated because stdin is not a terminal.
Counting objects: 9, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), 795 bytes | 0 bytes/s, done.
Total 9 (delta 0), reused 0 (delta 0)
-----> Cleaning up...
-----> Building python-test from herokuish...
-----> Adding BUILD_ENV to build environment...
-----> Python app detected
-----> Installing python-2.7.11
       $ pip install -r requirements.txt
       Collecting Flask==0.9 (from -r requirements.txt (line 1))
       Downloading Flask-0.9.tar.gz (481kB)
       Collecting Jinja2==2.6 (from -r requirements.txt (line 2))
       Downloading Jinja2-2.6.tar.gz (389kB)
       Collecting Werkzeug==0.8.3 (from -r requirements.txt (line 3))
       Downloading Werkzeug-0.8.3.tar.gz (1.1MB)
       Collecting gunicorn==0.17.2 (from -r requirements.txt (line 4))
       Downloading gunicorn-0.17.2.tar.gz (360kB)
       Installing collected packages: Werkzeug, Jinja2, Flask, gunicorn
       Running setup.py install for Werkzeug: started
       Running setup.py install for Werkzeug: finished with status 'done'
       Running setup.py install for Jinja2: started
       Running setup.py install for Jinja2: finished with status 'done'
       Running setup.py install for Flask: started
       Running setup.py install for Flask: finished with status 'done'
       Running setup.py install for gunicorn: started
       Running setup.py install for gunicorn: finished with status 'done'
       Successfully installed Flask-0.9 Jinja2-2.6 Werkzeug-0.8.3 gunicorn-0.17.2

-----> Running post-compile hook
remote: bin/post_compile: line 2: /opt/test/datetime.lst: No such file or directory
To [email protected]:python-test
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to '[email protected]:python-test'

Not sure why you had a different result.

In any case, this approach won't work because the storage plugin doesn't mount on build. Instead, you might try using an app.json entry to execute a predeploy or postdeploy script. Docs are here

[michaelshobbs]

@josegonzalez if we wanted to support this, we would have to include build in the available phases in the storage plugin and filter out -v ... in the build case for dockerfile apps.

[josegonzalez]

@michaelshobbs I think we should support this. The bin/post_compile step is pretty normal for python applications, and I think supporting this seems like a "no-brainer".

Note: I'm leaving the implementation to you :P

[josegonzalez]

@thelonecabbage note: we need to revert this feature as it broke certain applications. We've instead added documentation on how to handle it manually going forward. Sorry for any trouble that causes you!