Pact-jvm: Results are not published to pact broker when using @RunWith(FilteredPactRunner) with jvm-provider-junit

[valemany]

Note: It works fine when using the normal @PactRunner.

[uglyog]

Pushed a fix for the issue.

[bethesque]

I don't think this is a bug! Verification should only be considered complete when ALL the interactions in a pact are verified. Verifying only some does not give you confidence that the consumer/provider will work in real life.

If the consumer team publishes 10 interactions, and the provider team only verifies 8 of them, and then everyone deploys because they think all the tests have passed... and then one of the unverified interactions breaks... bad things!

[uglyog]

Sigh, trust @bethesque to open a can of worms ;-)

So how do we determine when the pact is verified if we can filter the interactions and these can run in separate tests, maybe even on separate build agents?

[uglyog]

We might have to track the results per interaction, and post that to the broker.

[bethesque]

I started off with big sky design that reported each interaction and its status, and then dialled it back to the bare basics when I implemented it. So we can increase the complexity if need be.

How common is this "run the verifications in separate processes" thing?

[bethesque]

@valemany what is your use case for verifying with the filter?

[valemany]

Thanks for looking into this guys.

I agree we should only see results when all pacts are verified. The reason why I use the FilteredPact is because it gives us the ability to run a single suite of test instead of running all the pacts. I think it would be good to have a number of total pacts and if I run the tests and total of number of pacts are verified then it means i ran the whole suite so I can get my results published, otherwise its fine not to publish results if I am only running a few tests if that makes sense.

[bethesque]

@valemany are you running filtered interactions in your CI, or just locally? I'd recommend only publishing the verification results from your CI, not when running on your local machine. You can see an example of how you might do this here (obviously you'd need to adapt it for the JVM) https://github.com/pact-foundation/pact_broker/wiki/Provider-verification-results

[valemany]

@bethesque

We are using PactFiltered in both CI and Locally, because its part of the class annotation so you cant really use different runners locally and in CI without a code change.

Let me give you more context on why we chose to use the FilteredPact as our default runner.

When we where using the normal runner we had the following issue:

One person created a new pact that was failing, he wanted to do full troubleshooting so he published it (it was work in progress). Another person opened a PR including a new pact that was working, but his PR failed because the pact from the other person failed (because normal PactRunner runs everything) Thats what FilteredPact gives us, the ability to enable pacts when we feel they are ready and removes the dependency for other people wanting to publish pacts that are not yet working on the provider side for troubleshooting purposes.

[carlosmmelo]

In short:

We wanted to seperate the tests on multiple files instead of one big huge file like so:

If you dont filter the patcs on each one of the files, its gonna run ALL the patcs from the broker on each file in the provider. Since we dont have ALL the tests on each file but rather the correspondent test per Controller for example (also better organized) then the "missing tests" are gonna be a false negative cause it didnt find them.

This is why the FilteredPact runner is useful for us.

[carlosmmelo]

I think the initial commit from @uglyog was ok, but to cover your concern @bethesque (which is very valid) I think @uglyog should add a safe check checking if ALL test from broker were executed (between the whole run/files) then publish, else exception or some warning saying we are missing (to define or add) tests from the broker.

[bethesque]

Ah! I think understand. You're choosing which pacts to run, not filtering interactions within a pact?

[carlosmmelo]

that is exactly right! @bethesque

[bethesque]

I agree, it makes perfect sense to publish the verifications for the (full) pacts that have actually been run.

[carlosmmelo]

for example, Given we have 8 tests in total:

File number 1:

@PactFilter(["request 1", "request 2", "request 3","request 4"])
@RunWith(FilteredPactRunner)
@Provider("My_Provider")
@PactBroker(host = "{broker_ip}", port = "80")
@TestFor(APIController_1)

File number 2:

@PactFilter(["request 5", "request 6", "request 7", "request 8"])
@RunWith(FilteredPactRunner)
@Provider("My_Provider")
@PactBroker(host = "{broker_ip}", port = "80")
@TestFor(APIController_2)

[valemany]

@bethesque that is correct.

[martinreus]

We also have some provider tests which are separated in different Test Suites. This enables us to keep the suites as small as possible.

I would agree to the idea that, once all states of a Pact are tested, the Pact file should be considered as verified for a consumer<->provider

Any updates on this issue?

[bethesque]

I've just realised there has been some misunderstanding due to incorrect terminology. A pact is the entire file, where as an interaction is a request/response pair. One pact => many interactions. What you're asking is to split the execution of those interactions across many test files, and once they're all verified, to publish the results back to the broker.

[valemany]

@martinreus That is correct. "Once all states of a Pact are tested, the Pact file should be considered as verified for a consumer<->provider"

[valemany]

@bethesque "What you're asking is to split the execution of those interactions across many test files, and once they're all verified, to publish the results back to the broker." <= That works for me.

[valemany]

@uglyog what do you think ^ ?

[uglyog]

My concern is working out when all the interactions have been verified. If the tests are running distributed across different machines, we need to accumulate the results in a central place, and then publish them once we have checked that all interactions are verified. We can only do this by comparing the results against the original pact file, and then we need to combine them to publish.

This adds a lot of complexity. We can reduce this somewhat by using the build directory as a temporary store, but then the tests have to run on the same machine.

[valemany]

We can reduce this somewhat by using the build directory as a temporary store, but then the tests have to run on the same machine. <= I think that would work for most ppl/cases. That definitely works for me and my team.

[CraigAtWork]

Hello, has there been any resolution on this thread? We have a similar configuration where we use the JUnit-based runner and we use @PactFilter to break our unit tests for each data state into separate classes.

With the above configuration, is there a mechanism to publish the results back to the broker?

[uglyog]

No progress on this issue to date.

[ngentse]

Hello, we have the same problem.
Does anyone have a workaround for this?

[krutoystas]

We have the same problem too. We have 13 controllers in our project and we use pact-jvm. I think, it will be better to use RestPactRunner in several test classes with common controlling

[uglyog]

Version 3.6.5 has been released with this feature

[HiTAsm]

Hello, I'm using 3.6.5 version but this does not work. The pact file is considered as verified even if some states are not tested.
I'm using PactRunner as the FilteredPactRunner is deprecated.
Is there a flag or annotation to use in order to make this work or should it work natively in 3.6.5 ?

[uglyog]

@HiTAsm in what way does it not work?

[HiTAsm]

@uglyog, from what I understood from this thread is once we break our tests to multiple seperate classes where we filter on each of the states that we want to test, the pact would be considered as verified only and only if all the interactions are tested.
In my use case, I have a test class where I filter on a particular state without testing the other states, once I run the test, the pact is considered as verified even though not all interactions within the pact were tested so I was expecting that the verification fails

[HiTAsm]

@uglyog, after upgrading to 3.6.13, the behaviour I described now works.