Vector: Audit and cleanup current Docker build process

[binarylogic]

I'd like audit, cleanup, and improve our current Docker build process. I originally set this up when we launched Vector and someone with more knowledge in this domain can probably identify improvements in this process.

Specifically:

1. Can/should we support more underlying distros?
2. Are there best practices we are not following as it relates to structuring our containers? (Ex: using ENTRYPOINT, location of the Vector binary and config files.
3. Review how users should provide their own configuration for Vector.
4. Can we reduce the image size?
5. Currently the release-docker CI step uses the local CI build artifacts. I would like to confirm that this is correct and that we don't want to pull from S3 instead?

Before we begin work I'd like to settle on a light specification so we know exactly what needs to be done here. Someone with more knowledge in this domain will be able to do this.

[mre]

Are there best practices we are not following as it relates to structuring our containers. (Ex: using ENTRYPOINT, location of the Vector binary and config files.

One thing I was confused when looking at the Dockerfile of docker-slim is that we copy the binary to /usr/bin (see here). We use /usr/local/bin for alpine, though.
According to this post, /usr/local/bin is for normal user programs not managed by the distribution package manager, e.g. locally compiled packages; so I guess we want to stick to that convention? Unless we plan to add it to upstream Debian, in which case /usr/bin should be fine.

Review how users should provide their own configuration for Vector.

Think that's okay. With https://github.com/timberio/vector/pull/900 in place, I think the default location should be /etc/vector/vector.toml, which we already mention in the docs.

Can we reduce the image size?

If we wanted really tiny images, we could use scratch builds.
Just linking the previous discussions here: https://github.com/timberio/vector/pull/172 and https://github.com/timberio/vector/issues/834.

[binarylogic]

Closing this since the issues were addressed in https://github.com/timberio/vector/pull/985. The scratch image work can be tracked via #834,.