Flank: Failed gcloud commands do not end the run

[bootstraponline]

With debug = false, all the gcloud commands can fail and flank will happily report success. We should probably check to see if there's anything in the error stream, and if so end the process with a non-zero exit code.

[renas]

Yeah, it needs to stop the execution if an error occurs (with non-zero exit code). I've started looking into this. I'll keep you updated.

[bootstraponline]

Cool. I think https://github.com/TestArmada/flank/issues/21 needs to be fixed first.

[renas]

Yeah fixes for both will be included in the next release.

On Thu, Jun 1, 2017 at 12:43 PM, bootstraponline [e-mail]
wrote:

Cool. I think #21 https://github.com/TestArmada/flank/issues/21 needs
to be fixed first.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/TestArmada/flank/issues/22#issuecomment-305600028,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKGpT_CAv_brr7rLhPefG7dTVsv03kYks5r_xRpgaJpZM4NtcQ-
.

[thalescm]

Also, there's a possibility on a 429 from firebase when starting testing. Happened to me.
Was running 20 shards, only 11 of them actually ran, and 9 were blocked due to reaching a specified quota.
Here's the log in case anyone wanna see https://gist.github.com/thalescm/e7366651790d94a7b0efa3e2b22ad396 (I've changed some names to not have any pii).

The process exited with 0, when it shouldn't, as you can see in the end of the gist.

[renas]

Thanks for reporting this.

On Jun 8, 2017 10:46 AM, "Thales Machado" [e-mail] wrote:

Also, there's a possibility on a 429 from firebase when starting testing.
Happened to me.
Was running 20 shards, only 11 of them actually ran, and 9 were blocked
due to reaching a specified quota.
Here's the log in case anyone wanna see https://gist.github.com/thalescm/
e7366651790d94a7b0efa3e2b22ad396 (I've changed some names to not have any
pii).

The process exited with 0, when it shouldn't, as you can see in the end of
the gist.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/TestArmada/flank/issues/22#issuecomment-307176970,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKGpdO82c5ew4--vq_h_HUbR8LvDdt0ks5sCDNpgaJpZM4NtcQ-
.

[bootstraponline]

ERROR: (gcloud.firebase.test.android.run) Http error while creating test matrix: ResponseError 429: Insufficient test quota

what does Insufficient test quota even mean? I thought Firebase Test Lab was unlimited.

[bootstraponline]

In #test-lab on the firebase slack, it was explained that Spark and Flame plans have limits on the amount of tests that may be run.

• https://firebase.google.com/pricing/

Virtual Device Tests: 10 tests/day
Physical Device Tests: 5 tests/day

4 devices in a matrix limit

[renas]

We will cover this case too. If "ERROR" is returned by gcloud we will end the run.

[thalescm]

Nice! Thanks!

[mrooney]

https://firebase.google.com/docs/test-lab/command-line#script_exit_codes has some useful hints on what exit codes you might want to catch and bail on. It would be nice if there was an option to bail the entire set of shards on an unrecoverable failure; if you need a green build to pass in CI, then a single shard failing with one of those could be cause to stop paying for the other shards and fail fast.

[bootstraponline]

if you need a green build to pass in CI, then a single shard failing with one of those could be cause to stop paying for the other shards and fail fast.

I have an example of how to cancel shards on failure in the poc runner

[Macarse]

@renas status update?

[renas]

@Macarse no updates yet. We are thinking of how to best deal with this. We will need to factor in the retries that gcloud does after it throws an error. I'll keep you updated.

[runningcode]

Any update? I'm seeing this happen a lot right now since firebase test lab is experiencing high latency.

[bootstraponline]

this is something I think I've solved in the kotlin_poc runner. It's missing a bunch of features that regular flank has though.