Stack: Broken integration tests: No CI?

[sjakobi]

I just ran stack test --flag stack:integration-tests :stack-integration-test on my local machine and it turned out that a test named multi-test failed with

...
       An index of the generated HTML coverage reports is available at /tmp/stack-integration-multi-test13787/.stack-work/install/x86_64-linux/lts-6.0/7.10.3/hpc/index.html
       Main.hs: user error (Didn't get expected report for multi-test-suite-test)

A bit of investigation revealed that the integration tests aren't run by either Travis or AppVeyor.

There's this bit in .travis.yml, but it maybe doesn't do what it's intended to do?

     cabal)
       cabal configure --enable-tests --enable-benchmarks -v2 --ghc-options="-O0 -Werror";
       cabal build;
       cabal test;
       cabal check;
       cabal sdist;
       cabal copy;
       cd test/integration; <-- HERE
       true stack setup;
       true stack test;
       ...

Are the integration tests possibly intentionally not run?

Is there some infrastructure that I don't know about, where the integration tests are run regularly?

Do we agree that the integration tests should run both on Travis and on AppVeyor, ideally as many platforms as possible?

[borsboom]

Unfortunately the integration tests take too long so Travis and AppVeyor both time out. We are running them for Linux on a private Jenkins server, and I'm working on bringing up a public Jenkins server so that we can give more visibility to these tests.

[sjakobi]

Thanks, that's good to know!

What about the other platforms though? Do you run the integration tests during the release process?

I guess it would be great to have CI for all the platforms that stack supports but I'm not sure that's a realistic goal…

[borsboom]

Yes, etc/scripts/release.hs always runs the integration tests prior to
creating the bindist. I agree it would be great to have integration tests
run on all platforms for every push, but that's not trivial to set up.

On Tue, Jul 5, 2016 at 11:18 AM Simon Jakobi [email protected]
wrote:

Thanks, that's good to know!

What about the other platforms though? Do you run the integration tests
during the release process?

I guess it would be great to have CI for all the platforms that stack
supports but I'm not sure that's a realistic goal…

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2331#issuecomment-230559283,
or mute the thread
https://github.com/notifications/unsubscribe/ACEGJbMavtvXSm_140RvoXsJy5cGGuWlks5qSqAHgaJpZM4JErZU
.

[borsboom]

https://jenkins-public.fpcomplete.com/job/stack-integration-tests/ is now basically set up. It just builds the master branch when pushed, and e-mails in the case of build failure. It's not yet integrated with Github pull requests.

[sjakobi]

Thanks, @borsboom, that's super useful!

Is it correct that the build results are (and will be) only visible to members of the Commercial Haskell organizations? Is this something that would change once the Jenkins server is integrated with PRs?

I'm also a bit surprised that e.g. build #6 took only 23 minutes, although it certainly benefitted from not having to rebuild any dependencies apart from stack's extra-deps.

I think that should make it possible to fit the integration tests into the Appveyor build, which. e.g. here took only 18 of a maximum of 60 minutes (AFAIK).
If it turns out that the builds time out sporadically, it might be possible to run the integration tests conditionally, depending on how much time was spent on compilation and unit tests.

Oh, and it could of course be that the Jenkins build uses more powerful hardware than Appveyor…

[Blaisorblade]

Is it correct that the build results are (and will be) only visible to members of the Commercial Haskell organizations?

Guess not (any more?)—the link seems to work for me (once I give them access to some of my GitHub info, not 100% sure whether that's needed), and I'm nobody:

https://jenkins-public.fpcomplete.com/job/stack-integration-tests/14/console

Of course that's read-only which seems correct.

[borsboom]

The authentication is mainly there as a deterrent to bots and malicious scripts (an open Jenkins server gets hammered pretty quickly). Any authenticated user has full read-only access currently (although in the future there may be some jobs that are hidden from the general public).

We did try to run integration tests on AppVeyor at one point, but they were always timing out. If someone wants to take a stab at getting it working again, that would be great. Another approach would be setting up a Jenkins slave running Windows on AWS.

[sjakobi]

I've moved the Windows CI part into #2391 so I think this can be closed.

[borsboom]

I'm re-opening because I want to get it running integration tests for PRs as well. However, I'm running into trouble with Jenkins' Github Pull Request Builder plugin. I think I'm hitting https://github.com/janinko/ghprb/issues/347, and so far the workarounds I've tried haven't worked. I think I'll leave it for now and wait until the upstream issue is resolved before trying it again, but if anyone out there has experience with this plugin, I'd love some help getting it working nicely (its documentation is out of date and somewhat lacking).

[snoyberg]

Closing now that the integration tests are running on Azure, and can be manually run on PRs