Flank: Does not scale to large test suites

[bootstraponline]

I have a test suite with 112 tests. These tests run on Firebase test lab on every pull request. Currently, we're using a single VM. The configuration is defined in fastlane. Our apps, tests, and build automation are all open source.

gcloud test runs directly on a single VM and takes 0:09:39 (112 tests).
flank with 112 VMs takes 0:13:24.

Flank increased our cost by 18x and is 3m 45s slower.

$ cat config.properties
debug-prints:true
deviceIds: NexusLowRes
os-version-ids: 25
orientations: portrait
locales: en
shard-timeout: 5
shard-duration: -1

The stdout is filled with / [102/113 files][ 2.0 MiB/ 2.1 MiB] 93% Done. It looks like more commands require the quiet flag. The performance loss is because we're:

• fetching XML results from each VM
• spinning up 1 python process per VM
• trying to access the Walmart only flank bucket

In the 1 test per VM model, everything we need is already in stdout. The test name, success/failure and a link to the full report. Skipping XML fetching would be a huge speed win.

In terms of avoiding spinning up the processes, you can check out this proof of concept:

• https://gist.github.com/bootstraponline/0b8fe4907c36ebb7521538e83a1ed91a#file-shard-py

I'm spinning up multiple test VMs from a single python process by extending the SDK with a new shard command.

[renas]

The flank bucket issue we are working on fixing. We'll add the quiet flag to the commands. As for fetching the xml files we can make that an option. We'll make a new release with these three updates next week.

As far as the python processes, thanks for sharing your script. Would be really nice to do it from a single python process. We'll do some experiments next week. I'll keep you updated.

[bootstraponline]

There was a discussion about a Java API for scheduling test runs on the Firebase slack. It turns out Android Studio has an integration with Firebase Test Lab. Using the existing Java lib would potentially eliminate the need for Python.

The relevant code is in:

• CloudAuthenticator.java
• CloudTestsLauncher.java

Unfortunately yogurtearl noticed the client code is contained in a jar.

• google-api-services-testing-v1-revsnapshot-1.20.0.jar

[bootstraponline]

Firebase is going to look into creating an unofficial Java API that we'd be able to use. I'll post here once there's more info available.

[renas]

Awesome!

On Fri, Jun 9, 2017 at 2:13 PM, bootstraponline [e-mail]
wrote:

Firebase is going to look into creating an unofficial Java API that we'd
be able to use. I'll post here once there's more info available.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/TestArmada/flank/issues/14#issuecomment-307501369,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKGpakSm0hreaNfE-2Xdzp4sSt-RPh0ks5sCbWNgaJpZM4NoAn1
.

[bootstraponline]

FYI I checked all the branches and studio-master-dev has the latest changes. The new link to the jar is:

• https://android.googlesource.com/platform/tools/studio/google/cloud/testing/+/studio-master-dev/firebase-testing/lib/google-api-services-testing-v1-revsnapshot-1.20.0.jar

master - 1 year ago
idea133 - 1 year ago
idea133-milestone - 1 year ago
studio-1.0-dev - 1 year ago
studio-1.0-release - 1 year ago
studio-1.1-dev - 1 year ago
studio-1.1-release - 1 year ago
studio-1.2-dev - 1 year ago
studio-1.2-release - 1 year ago
studio-1.3-dev - 1 year ago
studio-1.3-release - 1 year ago
studio-1.4-dev - 1 year ago
studio-1.4-release - 1 year ago
studio-master-dev - 5 months ago
studio-master-release - 1 year ago
tools_r22.2 - 1 year ago
ub-tools-idea133-release - 1 year ago

[bootstraponline]

Flank Kotlin scales to large tests.