Argo: Fix flakey test TestFunctionalSuite/TestStopBehavior

[alexec]

TestFunctionalSuite/TestStopBehavior - e2e
level=info msg="logging debug diagnostics to file:///tmp/log/argo-e2e/FunctionalSuite-TestStopBehavior.log"
level=info msg="Submitting workflow" workflow=stop-terminate
level=info msg="Workflow submitted" uid=3368a032-4644-44d0-9e13-8c8dd1d866f6 workflow=stop-terminate
level=info msg="Waiting for condition" condition="to start" timeout=5s workflow=stop-terminate
level=info msg=... condition="to start" message= phase= timeout=5s type=ADDED workflow=stop-terminate
level=info msg=... condition="to start" message= phase=Running timeout=5s type=MODIFIED workflow=stop-terminate
level=info msg="Condition met" condition="to start" timeout=5s workflow=stop-terminate
level=info msg="../../dist/argo -n argo stop stop-terminate"
level=info msg="workflow stop-terminate stopped"
level=info
level=info msg="Waiting for condition" condition="to finish" timeout=30s workflow=stop-terminate
level=info msg=... condition="to finish" message= phase=Running timeout=30s type=ADDED workflow=stop-terminate
level=info msg=... condition="to finish" message= phase=Running timeout=30s type=MODIFIED workflow=stop-terminate
level=info msg=... condition="to finish" message= phase=Running timeout=30s type=MODIFIED workflow=stop-terminate
level=info msg=... condition="to finish" message= phase=Running timeout=30s type=MODIFIED workflow=stop-terminate
level=info msg=... condition="to finish" message= phase=Running timeout=30s type=MODIFIED workflow=stop-terminate
level=info msg=... condition="to finish" message= phase=Running timeout=30s type=MODIFIED workflow=stop-terminate
level=info msg=... condition="to finish" message= phase=Running timeout=30s type=MODIFIED workflow=stop-terminate
level=info msg=... condition="to finish" message="Stopped with strategy 'Stop'" phase=Failed timeout=30s type=MODIFIED workflow=stop-terminate
level=info msg="Condition met" condition="to finish" timeout=30s workflow=stop-terminate
level=info msg="Checking expectation" test=TestFunctionalSuite/TestStopBehavior workflow=stop-terminate
level=error msg="Cannot get pod" node=A pod=stop-terminate-3693143150 test=TestFunctionalSuite/TestStopBehavior workflow=stop-terminate
functional_test.go:387: 
Error Trace:    functional_test.go:387
                            then.go:53
                            then.go:30
                            functional_test.go:379
Error:          Not equal: 
                expected: "Succeeded"
                actual  : "Error"

                Diff:
                --- Expected
                +++ Actual
                @@ -1,2 +1,2 @@
                -(v1alpha1.NodePhase) (len=9) "Succeeded"
                +(v1alpha1.NodePhase) (len=5) "Error"

Test:           TestFunctionalSuite/TestStopBehavior

https://18739-100982449-gh.circle-artifacts.com/0/FunctionalSuite-TestStopBehavior.log

[alexec]

@simster7 do you want to own this? I'll just skip today.

[simster7]

Sure I’ll own

[alexec]

With argosay:v2, this seems to have become a problem again. @simster7 do you want to own?

I'm a bit worried that this code changes behaviour with a change to a image tag.

https://app.circleci.com/pipelines/github/argoproj/argo/3797/workflows/ba1c40b4-78f4-4fa5-bc2d-d526fa2ab6f5/jobs/22741/tests
https://22741-100982449-gh.circle-artifacts.com/0/FunctionalSuite-TestStopBehavior.log

[simster7]

level=info msg=... condition="to finish" message="Stopped with strategy 'Stop'" phase=Error timeout=30s type=MODIFIED workflow=stop-terminate

Increase the timeout? Test passes locally

[alexec]

This test is flakey and now failing about 1/3 builds:

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1,2 +1,2 @@
                                -(v1alpha1.NodePhase) (len=6) "Failed"
                                +(v1alpha1.NodePhase) (len=9) "Succeeded"

                Test:           TestFunctionalSuite/TestStopBehavior

[alexec]

Note that I'm seeing Failed rather than Error.

[alexec]

FunctionalSuite-TestStopBehavior.log

[simster7]

Will take another look

[simster7]

Closing in favor of #3823