Collect: Look at parallelizing instrumentation tests

Created on 10 Jan 2020  ·  15Comments  ·  Source: getodk/collect

Right now our tests take 20m+ to run. This is causing timeout issues with CI and I don't think we want to have to keep bumping our timeout values every time the test suite grows. We do also need an audit of test coverage we can move from Espresso to Robolectric/JUnit but I think we expect this suite to stay at least this big in the long term.

It'd be good to look into parallelizing our tests. I think the focus for this should be for CI (with Firebase Test Lab). Parallelizing locally is potentially a bad idea due to the hit on people's dev environments. Locally I think continuing to break tests into concentrated domain areas will help devs have smaller sets of tests to run while working on features/refactors.

help wanted
Source
picture seadowg

Most helpful comment

It's great that you're thinking about other opportunities for improvement, @chidauri. To add a little more context, we have the style and unit tests run before instrumented tests so that we don't do an instrumented test run if there are obvious issues with the build. We get a limited number of runs on Firebase Test Lab.

picture lognaturel on 29 Jan 2020
👍2

All 15 comments

@seadowg Can i help with this?

chidauri picture chidauri on 29 Jan 2020
🎉1

You're very welcome to have a play with it @chidauri. The next most useful step would be to do some research on how we could set this up so even if you're not able to get it working just sharing that would be valuable!

seadowg picture seadowg on 29 Jan 2020
🚀1

@seadowg can we remove dependency of unit test to run instrumented tests ,and make them run in parallel, because before merging a commit into master unit tests are already being taken care of ?

chidauri picture chidauri on 29 Jan 2020

@chidauri maybe it's not clear from the issue description but I just meant running our instrumentation tests in parallel (i.e. ./gradlew connectedAndroidTest).

seadowg picture seadowg on 29 Jan 2020
👍2

It's great that you're thinking about other opportunities for improvement, @chidauri. To add a little more context, we have the style and unit tests run before instrumented tests so that we don't do an instrumented test run if there are obvious issues with the build. We get a limited number of runs on Firebase Test Lab.

lognaturel picture lognaturel on 29 Jan 2020
👍2

@seadowg @lognaturel can flank be used, i think it is created just for this?

chidauri picture chidauri on 29 Jan 2020

@chidauri yeah it looks like it would definitely be worth a try. I think to experiment you'll need to set up your own Firebase account. Test Lab is limited but free on so you should be able to give it a go!

seadowg picture seadowg on 30 Jan 2020
👍1

@seadowg
Without flank, the instrumented tests took 22 min 27 sec
Flank results:

3 shards

each shard took approx 17 mins

5 shards

each shard took approx 10 mins

But there seems to be problem here, because 1 out of 3 shards failed in first case, and 3 out of 5 shards failed in second case.
generalSharedPreferencesUpgradeTest is the only test which failed in both the cases, and for different reasons.
Trace for 3 shards case:
junit.framework.AssertionFailedError: expected:<http://localhost:45115/> but was:<https://opendatakit.appspot.com> at junit.framework.Assert.fail(Assert.java:50) at junit.framework.Assert.failNotEquals(Assert.java:287) at junit.framework.Assert.assertEquals(Assert.java:67) at junit.framework.Assert.assertEquals(Assert.java:74) at org.odk.collect.android.SharedPreferencesTest.generalSharedPreferencesUpgradeTest(SharedPreferencesTest.java:71) at java.lang.reflect.Method.invoke(Native Method) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at androidx.test.runner.AndroidJUnit4.run(AndroidJUnit4.java:104) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56) at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2075)

Trace for 5 shards case:
junit.framework.AssertionFailedError: expected:<yes> but was:<no> at junit.framework.Assert.fail(Assert.java:50) at junit.framework.Assert.failNotEquals(Assert.java:287) at junit.framework.Assert.assertEquals(Assert.java:67) at junit.framework.Assert.assertEquals(Assert.java:74) at org.odk.collect.android.SharedPreferencesTest.generalSharedPreferencesUpgradeTest(SharedPreferencesTest.java:71) at java.lang.reflect.Method.invoke(Native Method) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at androidx.test.runner.AndroidJUnit4.run(AndroidJUnit4.java:104) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at org.junit.runner.JUnitCore.run(JUnitCore.java:115) at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56) at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392) at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2075)

chidauri picture chidauri on 31 Jan 2020
🎉1

@chidauri ah amazing thanks for trying this out!

It does look like some of our tests still don't clear state before running. This could lead to tests polluting others or could mean some of our tests actually (probably accidently) rely on data that some other test has set up. You'll see that we use the ResetStateRule in a lot of tests. Experimenting with adding this to tests that are failing might help you make some progress.

seadowg picture seadowg on 31 Jan 2020
👍1

@seadowg That fixed generalSharedPreferencesUpgradeTest, but some tests already use ResetStateRule and still give error, mostly in FillBlankFormTest.java like formsWithDate_ShouldSaveFormsWithSuccess

androidx.test.espresso.NoMatchingViewException: No views in hierarchy found matching: with id: org.odk.collect.android:id/save_exit_button

View Hierarchy:
+>DecorView{id=-1, visibility=VISIBLE, width=1080, height=1920, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=WM.LayoutParams{(0,0)(fillxfill) sim=#10 ty=1 fl=#81810100 pfl=0x20000 wanim=0x10302f6 needsMenuKey=2 colorMode=0}, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3}
|
+->LinearLayout{id=-1, visibility=VISIBLE, width=1080, height=1788, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@c4a427e, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2}
|
+-->ViewStub{id=16908679, res-name=action_mode_bar_stub, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@754e3df, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+-->FrameLayout{id=-1, visibility=VISIBLE, width=1080, height=1722, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@a0d282c, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=66.0, child-count=1}
|
+--->FitWindowsLinearLayout{id=2131296305, res-name=action_bar_root, visibility=VISIBLE, width=1080, height=1722, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@144088a, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2}
|
+---->ViewStubCompat{id=2131296317, res-name=action_mode_bar_stub, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@94ddefb, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+---->ContentFrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=1080, height=1722, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@13a9b18, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}
|
+----->RelativeLayout{id=2131296514, res-name=llParent, visibility=VISIBLE, width=1080, height=1722, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@a116fd7, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=6}
|
+------>Toolbar{id=2131296734, res-name=toolbar, visibility=VISIBLE, width=1080, height=154, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@ca50c4, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3}
|
+------->AppCompatImageButton{id=-1, desc=‎‏‎‎‎‎‎‏‎‏‏‏‎‎‎‎‎‏‎‎‏‎‎‎‎‏‏‏‏‏‎‏‏‎‏‏‎‎‎‎‏‏‏‏‏‏‏‎‏‏‏‏‏‎‏‎‎‏‏‎‏‎‎‎‎‎‏‏‏‎‏‎‎‎‎‎‏‏‎‏‏‎‎‏‎‏‎‏‏‏‏‏‎‎Navigate up‎‏‎‎‏‎, visibility=VISIBLE, width=154, height=154, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.Toolbar$LayoutParams@6584bad, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+------->AppCompatTextView{id=-1, visibility=VISIBLE, width=441, height=74, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.Toolbar$LayoutParams@f751ee2, tag=null, root-is-layout-requested=false, has-input-connection=false, x=198.0, y=40.0, text=1560_IntegerData, input-type=0, ime-target=false, has-links=false}
|
+------->ActionMenuView{id=-1, visibility=VISIBLE, width=419, height=154, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.Toolbar$LayoutParams@9133273, tag=null, root-is-layout-requested=false, has-input-connection=false, x=661.0, y=0.0, child-count=3}
|
+-------->ActionMenuItemView{id=2131296544, res-name=menu_save, desc=Save Form, visibility=VISIBLE, width=154, height=132, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.ActionMenuView$LayoutParams@e7db530, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=11.0, text=, input-type=0, ime-target=false, has-links=false}
|
+-------->ActionMenuItemView{id=2131296540, res-name=menu_goto, desc=Go To Prompt, visibility=VISIBLE, width=154, height=132, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.ActionMenuView$LayoutParams@bdaf5a9, tag=null, root-is-layout-requested=false, has-input-connection=false, x=154.0, y=11.0, text=, input-type=0, ime-target=false, has-links=false}
|
+-------->OverflowMenuButton{id=-1, desc=‎‏‎‎‎‎‎‏‎‏‏‏‎‎‎‎‎‏‎‎‏‎‎‎‎‏‏‏‏‏‎‏‏‏‏‏‏‏‏‎‏‏‎‏‏‏‏‎‏‎‎‎‏‏‎‏‏‏‎‏‏‎‏‎‏‎‎‎‎‎‎‏‎‎‎‏‏‏‎‏‏‏‏‎‎‎‏‎‎‎‏‎‏‎More options‎‏‎‎‏‎, visibility=VISIBLE, width=111, height=132, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=androidx.appcompat.widget.ActionMenuView$LayoutParams@cca472e, tag=null, root-is-layout-requested=false, has-input-connection=false, x=308.0, y=11.0}
|
+------>ProgressBar{id=2131296613, res-name=progressBar, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@f6482cf, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+------>AppCompatImageView{id=-1, visibility=VISIBLE, width=1080, height=28, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@8c0f45c, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=154.0}
|
+------>LinearLayout{id=2131296620, res-name=questionholder, visibility=VISIBLE, width=1080, height=1568, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@fc44365, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=154.0, child-count=1}
|
+------->ODKView{id=-1, visibility=VISIBLE, width=1080, height=1568, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@837a83a, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}
|
+-------->NestedScrollView{id=2131296590, res-name=odk_view_container, visibility=VISIBLE, width=1080, height=156, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@bd07ceb, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}
|
+--------->LinearLayout{id=-1, visibility=VISIBLE, width=1080, height=156, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@696fa48, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}
|
+---------->IntegerWidget{id=11, visibility=VISIBLE, width=1080, height=149, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@a3ab0e1, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=7.0, child-count=3}
|
+----------->AudioVideoImageTextLabel{id=7, visibility=VISIBLE, width=1080, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@4b7fcc7, tag=0, , root-is-layout-requested=false, has-input-connection=false, x=0.0, y=7.0, child-count=1}
|
+------------>RelativeLayout{id=-1, visibility=VISIBLE, width=1080, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@7e70492, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2}
|
+------------->LinearLayout{id=-1, visibility=VISIBLE, width=1080, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@f6cd7de, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3}
|
+-------------->FrameLayout{id=2131296719, res-name=text_container, visibility=VISIBLE, width=1080, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@7d82c8c, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}
|
+--------------->AppCompatTextView{id=8, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=true, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@4e5cfd5, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=, input-type=0, ime-target=false, has-links=false}
|
+-------------->AppCompatImageView{id=2131296481, res-name=imageView, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@eea93ea, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+-------------->AppCompatTextView{id=2131296551, res-name=missingImage, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@3ee76db, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=, input-type=0, ime-target=false, has-links=false}
|
+------------->LinearLayout{id=2131296531, res-name=media_buttons, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@6d604b6, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2}
|
+-------------->AudioButton{id=2131296333, res-name=audioButton, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@953a5b7, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=, input-type=0, ime-target=false, has-links=false, is-checked=false}
|
+-------------->MaterialButton{id=2131296754, res-name=videoButton, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@7a58124, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=, input-type=0, ime-target=false, has-links=false, is-checked=false}
|
+----------->LinearLayout{id=9, visibility=VISIBLE, width=1080, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@24b8b90, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=7.0, child-count=3}
|
+------------>RelativeLayout{id=2131296722, res-name=text_layout, visibility=VISIBLE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@63b94af, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2}
|
+------------->AppCompatImageView{id=2131296467, res-name=help_icon, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@d05d0bc, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+------------->AppCompatTextView{id=2131296468, res-name=help_text_view, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@e7b5845, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=, input-type=0, ime-target=false, has-links=false}
|
+------------>LinearLayout{id=2131296465, res-name=guidance_text_layout, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@d36cccb, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=1}
|
+------------->AppCompatTextView{id=2131296466, res-name=guidance_text_view, visibility=VISIBLE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@2bd3ca8, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=, input-type=0, ime-target=false, has-links=false}
|
+------------>AppCompatTextView{id=2131296758, res-name=warning_text, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@4fe33c1, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=, input-type=0, ime-target=false, has-links=false}
|
+----------->EditText{id=10, visibility=VISIBLE, width=1014, height=142, has-focus=true, has-focusable=true, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=true, is-focusable=true, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@c470766, tag=null, root-is-layout-requested=false, has-input-connection=true, editor-info=[inputType=0x1002 imeOptions=0x6 privateImeOptions=null actionLabel=null actionId=0 initialSelStart=1 initialSelEnd=1 initialCapsMode=0x1000 hintText=null label=null packageName=null fieldId=0 fieldName=null extras=null hintLocales=null contentMimeTypes=null ], x=33.0, y=7.0, text=5, input-type=4098, ime-target=true, has-links=false}
|
+------>AppCompatImageView{id=2131296668, res-name=shadow_up, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@d636aa7, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+------>FrameLayout{id=2131296580, res-name=navigation_view, visibility=VISIBLE, width=1080, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.RelativeLayout$LayoutParams@b794afd, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=1722.0, child-count=1}
|
+------->LinearLayout{id=2131296358, res-name=buttonholder, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@d9499f9, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3}
|
+-------->AppCompatTextView{id=2131296454, res-name=form_back_button, desc=Back, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@ec59d3e, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=Back, input-type=0, ime-target=false, has-links=false}
|
+-------->View{id=-1, visibility=VISIBLE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@c01079f, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
+-------->AppCompatTextView{id=2131296455, res-name=form_forward_button, desc=Next, visibility=GONE, width=0, height=0, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is-focused=false, is-focusable=true, is-layout-requested=true, is-selected=false, layout-params=android.widget.LinearLayout$LayoutParams@bce0ec, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, text=Next, input-type=0, ime-target=false, has-links=false}
|
+->View{id=16908336, res-name=navigationBarBackground, visibility=VISIBLE, width=1080, height=132, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@f7bdcb5, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=1788.0}
|
+->View{id=16908335, res-name=statusBarBackground, visibility=VISIBLE, width=1080, height=66, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, layout-params=android.widget.FrameLayout$LayoutParams@dfa4f4a, tag=null, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0}
|
at dalvik.system.VMStack.getThreadStackTrace(Native Method)
at java.lang.Thread.getStackTrace(Thread.java:1538)
at androidx.test.espresso.base.DefaultFailureHandler.getUserFriendlyError(DefaultFailureHandler.java:96)
at androidx.test.espresso.base.DefaultFailureHandler.handle(DefaultFailureHandler.java:59)
at androidx.test.espresso.ViewInteraction.waitForAndHandleInteractionResults(ViewInteraction.java:322)
at androidx.test.espresso.ViewInteraction.desugaredPerform(ViewInteraction.java:178)
at androidx.test.espresso.ViewInteraction.perform(ViewInteraction.java:119)
at org.odk.collect.android.support.pages.FormEntryPage.clickSaveAndExit(FormEntryPage.java:49)
at org.odk.collect.android.regression.FillBlankFormTest.formsWithDate_ShouldSaveFormsWithSuccess(FillBlankFormTest.java:141)
at java.lang.reflect.Method.invoke(Native Method)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.CopyFormRule$CopyFormStatement.evaluate(CopyFormRule.java:57)
at org.odk.collect.android.support.ResetStateRule$ResetStateStatement.evaluate(ResetStateRule.java:41)
at androidx.test.rule.GrantPermissionRule$RequestPermissionStatement.evaluate(GrantPermissionRule.java:134)
at androidx.test.rule.ActivityTestRule$ActivityStatement.evaluate(ActivityTestRule.java:531)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at androidx.test.runner.AndroidJUnit4.run(AndroidJUnit4.java:104)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2075)
chidauri picture chidauri on 3 Feb 2020

@chidauri are the same tests failing every time? It could be that those tests are indeed "flakey"and that running in parrellel has just revealed that.

seadowg picture seadowg on 5 Feb 2020

@seadowg The results are inconsistent, sometimes all the tests pass but sometimes one of the tests fails.

chidauri picture chidauri on 7 Feb 2020

@seadowg right my question is that is it the same set of tests that sometimes fail? If so it could be that those tests might be able to be improved to not be flakey. It might be worth sharing your current work in a draft PR so we can see what the setup looks like as well!

seadowg picture seadowg on 7 Feb 2020
👍1

@seadowg after running significant number of times, it is not the same set of tests which fail. I've been able to run only 6 times so far, due to the limits :(

Edit: If we use num-flaky-test-attemps: 1, the test which fails on the first run passes the next time, and this is consistent. I'll open a draft PR with a working solution.

chidauri picture chidauri on 7 Feb 2020
👍1

Hello @grzesiek2010, you have been unassigned from this issue because you have not updated this issue or any referenced pull requests for over 15 days.

You can reclaim this issue or claim any other issue by commenting @opendatakit-bot claim on that issue.

Thanks for your contributions, and hope to see you again soon!

getodk-bot picture getodk-bot on 5 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pranavgupta1234 picture pranavgupta1234  ·  4Comments
lognaturel picture lognaturel  ·  4Comments
yanokwa picture yanokwa  ·  3Comments
mapland picture mapland  ·  4Comments
dr0pdb picture dr0pdb  ·  5Comments