Logstash: Acceptance tests are timing out since June 22 (5 days)

[jordansissel]

https://logstash-ci.elastic.co/job/elastic+logstash+5.0+multijob-acceptance/

Many of these tests are failing due to what I _think_ are job timeouts. We have a 6-hour timeout on jobs and the recent aborts are all around 5hr40min mark

I don't know why they're getting stuck, yet. The last output in the console log looks like this:

Bundle complete! 4 Gemfile dependencies, 16 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Acceptance: Running the tests
bootstraping all non experimental VM's defined in acceptance/Vagrantfile for debian: ["debian-8", "ubuntu-1204", "ubuntu-1404"]
[Acceptance specs] running on ["debian-8", "ubuntu-1204", "ubuntu-1404"]
Build was aborted
Finished: ABORTED

In other cases, the last output looks like it timed out waiting for a job to fetch from git:

Fetching upstream changes from [email protected]:elastic/logstash.git
 > git --version # timeout=10
using GIT_SSH to set credentials used for connecting to slaves
 > git -c core.askpass=true fetch --tags --progress [email protected]:elastic/logstash.git +refs/heads/*:refs/remotes/origin/*
Build was aborted
Finished: ABORTED

[purbon]

good catch @jordansissel thanks a lot!

for the speed side, I think https://github.com/elastic/logstash/pull/5538 might help improving speed, getting your review there would be very nice, as @ph and @untergeek could not still do it.

on the other side, @elasticdog could you check why git might be timing out?

[jordansissel]

@purbon I wonder if we could also help this along by splitting up the job per platform. For example, this output:

[Acceptance specs] running on ["debian-8", "ubuntu-1204", "ubuntu-1404"]

If we split these into 3 separate jobs, then we would run less risk of job timeout, right? Or at least, we could see which platform was problematic based on the job?

[elasticdog]

Looking at the workspace for the 5.0 branch acceptance tests, I think an old .git/ directory was hanging around that might have been pissed off for whatever reason (hetzner has been having some network flakiness lately)...I removed it and the current job is running again.

[purbon]

@elasticdog jobs are running again, but looks like they take too much time already.

[purbon]

@jordansissel this might help having less concurrent load at the same time for sure, we should see a right way to show this from user perspective too.

on the other side, @jordansissel @suyograo could one of us please review https://github.com/elastic/logstash/pull/5538 ? this will reduce our running time here around 50% helping also the CI infra and user experience for tests.

[purbon]

working on the actuall situation for being stuck, looks like the deb package might be broken somehow for debian-8, will be opening a concrete issue when I find out root cause.

[purbon]

Actually this issue is due to an interactive question being ask during the update process in debian-8, see:

vagrant@packer-virtualbox-iso-1452100235:~$ ls
logstash-5.0.0-alpha3.deb
vagrant@packer-virtualbox-iso-1452100235:~$ sudo dpkg -i logstash-5.0.0-alpha3.deb 
Selecting previously unselected package logstash.
(Reading database ... 52840 files and directories currently installed.)
Preparing to unpack logstash-5.0.0-alpha3.deb ...
Unpacking logstash (1:5.0.0~alpha3-1) ...
Setting up logstash (1:5.0.0~alpha3-1) ...
Using provided startup.options file: /etc/logstash/startup.options
Sending all logs to /tmp/tmp.ZZjKXcWAwA
Successfully created system startup script for Logstash
vagrant@packer-virtualbox-iso-1452100235:~$ sudo dpkg -i /logstash-build/logstash-5.0.0-SNAPSHOT.deb 
(Reading database ... 65284 files and directories currently installed.)
Preparing to unpack .../logstash-5.0.0-SNAPSHOT.deb ...
Unpacking logstash (1:5.0.0-1) over (1:5.0.0~alpha3-1) ...
Setting up logstash (1:5.0.0-1) ...

Configuration file '/etc/logstash/logstash.yml'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** logstash.yml (Y/I/N/O/D/Z) [default=N] ? 

so the call is waiting there forever!

[elasticdog]

Good find @purbon! I'll wait to see that things get passing again before closing this out.

[purbon]

@elasticdog actually been the superpowers of @dliappis who helped find it. He deserve the credit!

[purbon]

closing this as builds are running smooth again.