Polis: Docs on getting compiled clients into static file server in docker-compose

[metasoarous]

Right now the docker-compose.yml is very nearly working (including the math worker; see pol-is/polisServer#43), but we still need to have a way to get the compiled client apps into the static fileserver.

@ballPointPenguin Do you have thoughts on this? I could imagine us either creating a new dockerfile which bases itself on the nginx image, but clones down and builds the client repos, and copies over the compiled assets. Does that make sense to you? Is there something else you'd recommend?

CC @rohanrichards.

[rohanrichards]

I'm sure you guys have already considered this but what's stopping you doing this the way uzzal does it with polis-container: https://github.com/uzzal2k5/polis_container ?

[metasoarous]

@rohanrichards I'm not sure how @uzzal2k5 actually created the static asset servers. It looks like he manually built an image on his dockerhub, but I don't see the code for it in the repository he pushed, and there doesn't seem to be any way to rebuild or update the images with more up to date client assets. So this is likely just as missing from his setup as from the one in polisServer. Please feel free to take inspiration and steel whatever is useful though.

[rohanrichards]

hmm, reading through his docker-compose.yml file it just looked like he was pointing the server at the clientparticipant container and I thought that was all that was required, I might not be fully understanding the problem.

[metasoarous]

Well, that's probably all that's required if you are fine with whatever version of the app was compiled into the static js asset in the clientparticipation image he created. But if you want to be able to rebuild those client asset images with updated code, there needs to be a way to do that.

[ballPointPenguin]

I agree with @metasoarous that a new dockerfile (and/or script) which clones and builds the client repos and copies your new local assets into it seems like a good approach.

I honestly haven't tried to do that yet, I had been focused on the API Server, Postgres, and Math containers, and I'd like to get that de-coupled from Heroku so it can be run completely locally or in new infrastructure.

[ballPointPenguin]

I will take some time this evening to test client side / static file server docker stuff. But don't let me step on your toes @rohanrichards if you are already working on it. Maybe we can compare notes and see what feels best.

[rohanrichards]

@metasoarous thanks for clarifying, I have a better idea of what's required. @ballPointPenguin don't worry about stepping on my toes, I will be hacking at this today but I have limited docker (and polis) experience so I doubt my progress will be impactful.

[rohanrichards]

I have made a brief list of what I think needs to be done to achieve the above, let me know if I'm heading in the right direction and any help filling in the blanks would be appreciated

• pol-is/polisClientParticipation needs Dockerfile to set up npm and run npm i and gulp dist(?)
• pol-is/polisClientAdmin needs the same(?)
• pol-is/polisServer needs a new container in docker-coompose.yml for both of the above which pulls down the latest git and runs the dockerfiles
• the static container needs to reference the new clientParticipant container somehow
• nginx in the static container needs a config (I have no experience with nginx) to serve these files(?)
• polis-server container needs a reference (network link?) to the new clientAdmin container and environment variable set for STATIC_FILES_HOST (is this correct? In uzzal's container he points the polis-server container at the clientParticipant so I'm not sure what nginx's role should be here, what needs to be served where?)

[rohanrichards]

@ballPointPenguin left you my progress over on the gitter: https://gitter.im/pol-is/polisDeployment

[ballPointPenguin]

Thank you for going through all of this @rohanrichards
I had a long discussion with @metasoarous last night and I think you are on the right track. Some refactoring may be necessary in a few of the repos to accommodate for variable endpoints, and other considerations.
I think the next task for me is to integrate Nginx.

A couple high-level considerations:

• Ideally there could be a "push-button" solution to pulling down pre-built containers from Docker Hub and running them all with docker-compose. The biggest impediment to this (as I see it) is that some of the static stuff needs to know its host at build time, not run time. So "push-button" can probably only work for pre-defined hosts, eg. pol.is and/or localhost.
• The build-it-yourself way: This approach should work well for most use cases, in which docker-compose pulls all the code from github repos and builds each project (or at least the static ones) given a environment vars file that specifies things like host.
  Optionally, one could run a subset of the services, e.g. just the db and math, whilst developing with a local node and JS client.

I would like to map out the connection points between each of the services.

[uzzal2k5]

@rohanrichards I'm not sure how @uzzal2k5 actually created the static asset servers. It looks like he manually built an image on his dockerhub, but I don't see the code for it in the repository he pushed, and there doesn't seem to be any way to rebuild or update the images with more up to date client assets. So this is likely just as missing from his setup as from the one in polisServer. Please feel free to take inspiration and steel whatever is useful though.

Please see this link: https://github.com/uzzal2k5/polis_container/blob/master/polis-build.sh

[uzzal2k5]

@metasoarous @rohanrichards
Hi @metasoarous , I could send PR , if you wish me to contribute. I asked you before regarding Dockerize pol-is application , but you told me , you have person who is working with that, Thats why I did not sent you PR , @rohanrichards - if you follow my repository carefully, you will understand what i did there.

https://github.com/uzzal2k5/polis_container

https://github.com/uzzal2k5/polisserver
https://github.com/uzzal2k5/polisClientAdmin
https://github.com/uzzal2k5/polisClientParticipation
https://github.com/uzzal2k5/polismath
https://github.com/uzzal2k5/polisleiningen
https://github.com/uzzal2k5/polispostgres

[patcon]

After a hectic couple weeks to get it right (preserving history, and getting all the repos into a monorepo) we finally have a working docker development environment! https://github.com/pol-is/polisServer/pull/65

It's living on https://github.com/pol-is/polisServer/tree/dev (will become master in future)

I'm going to close this issue, but I just wanted to say thank you so SO much @rohanrichards and especially @uzzal2k5 for all your groundwork on docker. We weren't able to focus attention on it at the time when you were working on it, but your proof-of-concept eventually helped light the fire under everyone :)

@uzzal2k5 I know it's not much, but we added a thank-you to the readme 🙂 (I'll add you in an upcoming PR @rohanrichards)
https://github.com/pol-is/polisServer/blob/dev/docs/deployment.md#docker-deployment

If you felt inclined, we're now moving a bit more quickly on things, and would be really grateful for any of your spare attention. If you felt like reviewing what we have on the dev branch, or joining the conversation in Gitter, we'd be excited for that 🎉 Hope you're both doing well!
https://gitter.im/pol-is/polisDeployment

[patcon]

Oh hey, and we're working on a build server here: https://github.com/pol-is/polisServer/issues/67