Polis: Allow for creation of speedier docker builds

[patcon]

Re-ticketed from @crkrenn's suggestion in https://github.com/pol-is-trial-balloon/polis/issues/8#issuecomment-619417169 :100:

This PR will build on code currently in https://github.com/patcon/polisServer/tree/docker-dev

Right now, building the whole multi-container setup via docker takes at least 15 minutes on a relatively powerful digitalocean droplet (2GB + 2 cores). Building on a less powerful local machine takes even longer.

We might want to speed this up. We can share ideas on how we might do that.

[patcon]

More details on my build time using docker-machine for remote build

# digitalocean docker machine created with:
~$ docker-machine create --driver digitalocean --digitalocean-access-token $DIGITALOCEAN_TOKEN --digitalocean-size s-2vcpu-2gb default

~$ time GIT_HASH=$(git log --pretty="%h" -n 1) docker-compose up --detach
>>> [lots of output]
>>> GIT_HASH=$(git log --pretty="%h" -n 1) docker-compose up --detach  6.94s user 1.32s system 1% cpu 13:24.58 total

So in ideal conditions of a DO instance (a place where bandwidth and CPU isn't much of an issue), this build takes ~14minute.

[patcon]

@crkrenn suggested:

• I will migrate my work in progress to hang off this branch. Goals include:
  
  
  
  1. A faster build if possible.
     
     
     
     
     
     1. Evaluation whether node client-admin and node client-participation have advantages over file-server
     
     
     
  
  

We should def try! Having said that, I suspect avoiding file-server won't offer much speed-up -- both a "dev server" and "prod build" approach will spend most of their time building. So the time to install is pretty comparable, i think?

All that's happening extra with the "file-server" approach is (1) the "deploy", which is just copying files locally from dist/ via grunt, and (2) spinning up a bare nginx container and copying a file in, then spinning up another near-bare container and installing 2 npm packages for file-server, then copying a small number of minified/gzipped files across containers.

These all feel relatively quick. It always felt like npm install and gulp took up the most time, and we can't avoid that, i think?

Ideas for quicker builds

Some things we could do (some or all):

1. Improve use of docker cache model. Get onto the same version of nodejs, and try to ensure shared cache layers between containers.
2. Ditch npm for yarn. Should be faster see below comment: npm ci
3. Ditch gulp&co for webpack. Should be faster (and gets more maintenance these days). see: https://github.com/pol-is/polisClientParticipation/issues/34
4. Allow MVP build. Maybe building a less fully-featured version (e.g. without math or reports).
5. Figure out BuildKit. Add parallelism to builds. Not sure how it works, but seems it can use docker-compose.yml files
6. Push images to repository via CI. Allow images to pulled directly from registry without building. https://github.com/pol-is/polisServer/issues/67
7. [something else?]

(1) docker cache, (2) yarn, (3) webpack seem like things we might want to do anyhow, so I'll prob personally start there, to see which speed ups while doing work already "on the critical path" of work :)

[ballPointPenguin]

My 2 cents:

1. agree
2. I don't think yarn is still faster than npm. Npm has done a lot to improve and become more yarn-like in the last few releases, especially with npm ci, but I like yarn and wouldn't strongly object to using it.
3. agree, possibly look into parcel as an alternative to webpack
4. no opinion
5. no opinion
6. agree
7. possibly configure a build server, such as a dedicated cloud instance, or a local machine that has power to really build these containers, either per release tag, or even per PR branch

[crkrenn]

For tweaking and debugging interface changes, running node locally is much faster.

I haven’t migrated this to the monorepo, but you can run the install and run scripts below. You can make a tweak to the code, run ./start again, and you are up and running very quickly because all of the external packages are fixed in images.

https://github.com/crkrenn/polis-deploy/tree/feature/docker_compose_localhost_no_math/docker_compose

-Chris

---

Q: Why is this email [hopefully] five sentences or less?
A: http://five.sentenc.es

From: Patrick Connolly [e-mail]
Reply-To: pol-is/polis-issues [e-mail]
Date: Monday, April 27, 2020 at 7:49 PM
To: pol-is/polis-issues [e-mail]
Cc: Ckl [e-mail], Mention [e-mail]
Subject: Re: [pol-is/polis-issues] Allow for creation of speedier docker builds (#139)

@crkrennhttps://github.com/crkrenn suggested:

• I will migrate my work in progress to hang off this branch. Goals include:

  • A faster build if possible.

    • Evaluation whether node client-admin and node client-participation have advantages over file-server

We should def try! Having said that, I suspect avoiding file-server won't offer much speed-up -- both a "dev server" and "prod build" approach will spend most of their time building. So the time to install is pretty comparable, i think?

All that's happening extra with the "file-server" approach is (1) the "deploy", which is just copying files locally from dist/ via grunt, and (2) spinning up a bare nginx containerhttps://github.com/patcon/polisServer/blob/docker-dev/fileServer/nginx.Dockerfile and copying a file in, then spinning up another near-bare container and installing 2 npm packages for file-server, then copying a small number of minified/gzipped files across containers.

These all feel relatively quick. It always felt like npm install and gulp took up the most time, and we can't avoid that, i think?

Ideas for quicker builds

Some things we could do (some or all):

1. Improve use of docker cache model. Get onto the same version of nodejs, and try to ensure shared cache layers between containers.
2. Ditch npm for yarn. Should be faster
3. Ditch gulp&co for webpack. Should be faster (and gets more maintenance these days)
4. Allow MVP build. Maybe building a less fully-featured version (e.g. without math or reports).
5. Figure out BuildKit. Add parallelism to builds. Not sure how it works, but seems it can use docker-compose.yml fileshttps://github.com/docker/buildx#-f---file-file
6. Push images to repository via CI. Allow images to pulled directly from registry without building.
7. [something else?]

(1) docker cache, (2) yarn, (3) webpack seem like things we might want to do anyhow, so I'll prob personally start there, to see which speed ups while doing work already "on the critical path" of work :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/pol-is/polis-issues/issues/139#issuecomment-620345864, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABOJ5V4IZ46AYXI34QHMRP3ROY74TANCNFSM4MR6HOLQ.

[patcon]

1. parcel. i'm down to investigate! haven't yet heard of it, but i've shamefully not kept up :)
   
   
   
   • only things that comes to mind for webpack are (1) how many folks already are familiar, or (2) some higher-level frontend dev frameworks already use it (create-react-app, next.js), for which leaning on might be a nice eventual way to standardize our toolchain. Not sure if that feels worth it though
   
   
2. ...
3. ...
4. ...
5. Good idea. Would GitHub Actions (or CircleCI or Travis) count as a build server if it's pushing to Docker Hub? e.g., https://github.com/marketplace/actions/publish-docker https://github.com/pol-is/polisServer/issues/67

[ballPointPenguin]

Github actions, or the others, would be a good path to explore for what I'm calling "build server". I think. I haven't tried GitHub actions yet but I'm hopeful.

Regarding webpack, yes it is widespread and that is an advantage. It is also quite complex and has a fast-changing and fast-growing surface area of plugins and extensions and in my experience one can get quite bogged down in webpack configuration.
Webpack is a beast. A powerful beast, but should be treated with some trepidation.
Parcel exists mainly as a counter to this, and has done well to be slim, minimal, and easy. But it's not the right fit for every project. I'm open to either/both/something else. I just want to insert a word of caution about webpack maintenance overhead.

All that to say that Webpack probably is the "best worst thing".

[ballPointPenguin]

This is nice comparison chart
https://github.com/FlorianRappl/bundler-comparison

[patcon]

ah gotcha. makes sense :+1: i've only used it with create-react-app, and it's zero-config like parcel for that, so that explains my different experience. Either works for me

[colinmegill]

Thanks everyone :)

For frontend builds, I would only support using webpack. I don't like enjoy
using it (at all), but we already use it in some of the repos in polis.

NPM has replicated most of the features that differentiated Yarn at this
point, and we should stick with NPM.

On Fri, May 1, 2020 at 5:19 PM Patrick Connolly [e-mail]
wrote:

ah gotcha. makes sense 👍 i've only used it with create-react-app, and
it's zero-config like parcel for that, so that explains my different
experience

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pol-is/polis-issues/issues/139#issuecomment-622572783,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AANQGGMU6GS6J56MOFECY3DRPM4F7ANCNFSM4MR6HOLQ
.

[patcon]

oh hey just realized webpack part of this convo falls under an existing ticket, so let's move that convo to https://github.com/pol-is/polis/issues/90 :)

[crkrenn]

In general, and particularly for this project, I think that sticking with "popular" (and as a result, usually well-maintained) packages is a good idea.

https://github.com/FlorianRappl/bundler-comparison#results shows that webpack is much more broadly used.

For a better resourced project, something that is clearly up and coming like parcel, could be a good choice, and I'm sure that it has advantages.

My 2 cents.

[patcon]

I think we've done everything proposed in here, so closing! Just ask to reopen if you feel differently