Logstash: Vagrant vs. Docker - fight!

[jakelandis]

Goal: Decide on a strategy for test isolation for Logstash integration tests as part of the strategy for moving from Travis to Jenkins: https://github.com/elastic/logstash/issues/8079

Factors to consider:

• Isolation - two tests running concurrently should not be able to impact each other
• Clean slate - each test run from the same state
• Integrates with Jenkins CI
• Trouble shooting, e.g. faithfully reproduce any error and/or debug actual error
• Simplify a developer's workflow to run the full suite of tests
• Run multiple in parrallel (for CI performance)
• Ability for integration tests to run on Linux or Windows (though Windows will require additional work out of scope)
• Support for developers that run Windows, Mac, or Linux as primary work device
• Support for 3rd party services such as Kafka and Elastic search to be included as part of the integration tests
• Allow 3rd party developers to use same framework (but not necessarily the same resources owned by elastic.co)
• As fast, or faster then current Travis implementation

I will add 3 comments with the leading options.

Please express your preference with 👍 and 👎 and 😕 for +0. _(note - top vote does not "win", but rather provides a means to express opinion)_

_please edit and add more factors below_

Additional Factors:

[jakelandis]

Vagrant + VirtualBox

• Pros
  
  
  
  • Existing jobs within elastic
  
  
  • Well maintained elastic vagrant boxes
  
  
  • Identical locally vs. running in CI
  
  
  • Debuggable since running local
  
  
  • Re-use by 3rd party devs.
  
  
• Cons
  
  
  
  • Jenkins CI must run on bare metal (standard EC2 doesn't allow vm's inside vm's)
  
  
  • VirtualBox and Hyper-V are not compatible, which is problematic for developers on Windows
  
  
  • Requires a clean up lifecycle (for example the host did not call vagrant destory -f)
  
  
  • Requires Vagrant and Virtual box installed (personal history has proved some specific versions needed to work together properly)
  
  

_please edit and add more pros/cons below__

• Additional Pros
• Additional Cons

[jakelandis]

Vagrant + AWS

• Pros
  
  
  
  • Only requires ephemeral instances on demand ~ $.10 per set of integration test
  
  
  • Scales very well, can have hundreds of parallel run if needed
  
  
  • Easy to adopt additional Linux OS's, framework exists for running isolated tests on Window's AMI (requires considerable work to realize)
  
  
  • AWS configuration options, Vagrant does not require much AWS knowledge
  
  
  • Re-use by 3rd party devs.
  
  
  • Only requires Vagrant installed locally
  
  
• Cons
  
  
  
  • Requires AWS credentials
  
  
  • Requires pre-configured security group that allows for remote debugging
  
  
  • Requires a clean up life-cycle (for example the host did not call vagrant destroy -f)
  
  
  • Less deterministic costs. ~ $.10 per set of integration test - can add up
  
  
  • Requires code to be committed to github (or some rsync strategy)
  
  

_please edit and add more pros/cons below__

• Additional Pros
• Additional Cons

[jakelandis]

Docker

• Pros

  • Lowest friction to existing infrastructure. (Docker already lives on the Jenkins slaves)
  • Lower overhead since don't need a full VM every-time
  • Identical locally vs. running in CI
  • High level of control over execution environment
  • Only requires Docker installed locally
  • Debuggable since running local
  • Re-use by 3rd party devs.

• Cons

  • Requires a clean up life-cycle for orphaned containers
  • Less like a production setup (Docker users will the official Docker image, else, likely a VM)
  • Requires Docker knowledge to troubleshoot (v.s. Vagrant, where you vagrant ssh into the box)

_please edit and add more pros/cons below__

• Additional Pros
• Additional Cons

[drewr]

Thanks for weighing the trade-offs here. Docker is preferred, and dynamically launching EC2 instances is to be avoided.

[frankschmitt]

@jakelandis One question regarding this Docker Con:

"Requires Docker knowledge to troubleshoot (v.s. Vagrant, where you vagrant ssh into the box)"

Forgive my ignorance (I've never used vagrant), but isn't vagrant ssh essentially the same as
docker exec -it <your_running_container> bash
to get an interactive session ?

[jakelandis]

@frankschmitt - That is most likely my personal experience bias showing, I don't personally have much experience with Docker, but quite a bit with Vagrant, so it _feels_ easier.

In fairness, it is a con of all 3 approaches, each of which requires domain knowledge to troubleshoot. You are right the two are essentially the same, and depending on your background is either easier or harder to troubleshoot on one platform or the other.

[jakelandis]

Docker wins.

The changes to support Docker for both Unit and Integration tests have been backported to 5.5, and the Jenkins changes should be landing soon.

The README for the integration tests have been updated to include instructions for how to run via Docker.