Spek: Android Project Test do not run if there is an IT block

[LewisRhine]

@RunWith(JUnitPlatform::class)
class ExampleUnitTest : Spek({
    given("a test") {
        val test = Test()
        on("thisIsTrue is false"){
            test.thisIsTrue = false
            it("should be false"){
                assertEquals(false, test.thisIsTrue)
            }
        }
    }
})

Resluts in
"0 of 1 test"
"Process finished with exit code 0"

If I don't have an IT block the test will run fine.

@RunWith(JUnitPlatform::class)
class ExampleUnitTest : Spek({
    given("a test") {
        val test = Test()
        on("thisIsTrue is false"){
            test.thisIsTrue = false
            assertEquals(false, test.thisIsTrue)
        }
    }
})

[raniejade]

@LewisRhine are there logs you can paste here?

[LewisRhine]

@raniejade I don't really get any log output. it seems like it just doesn't see the test if there is an IT block.

[raniejade]

@LewisRhine Ahh, I remember now. It's an issue with JUnit 4, you can only at most 2 group nestings since it assumes that tests are defined as class methods.

[LewisRhine]

@raniejade Oh, I see. Am I guessing there is no way around it ad of now?

[raniejade]

Yeah, unfortunately not. We are thinking of ways to fix this, one of our priority right now is improving android support.

[igorwojda]

Shame, because I really liked this framework ;/

Any update on his?
Any timeline on this?

[raniejade]

@panell Sorry, the team has been very busy with personal stuff in the past month. I'll start something next week, it's going to be a big change and hopefully it lessen the issues being felt on android.

[igorwojda]

thanks for the info. I really appreciate quick update.

[artem-zinnatullin]

Just to make it clear, @LewisRhine did you use on() intentionally? Because any kind of nesting with groups like context/describe/given works fine with JUnit4 and fits most of the cases.

For instance we don't have any usages of on() actions in few hundreds of specs (thousands of test cases).

[raniejade]

@artem-zinnatullin how deep is the nesting?

For instance we don't have any usages of on() actions in few hundreds of specs (thousands of test cases).

Fck, yeah JUnit 4 doesn't support dynamic tests (tests discovered during execution).

[artem-zinnatullin]

@raniejade I'm currently working with spec that has 12 nested groups.

Spek itself works just fine but Mockito starts going crazy on verify() calls, looks like it's caused by class names that Kotlin compiler generates for such deeply nested lambdas.

[igorwojda]

For me it failing with very at basic stage, exactly like this @LewisRhine described at the top.
One think I want to add is that Android Strudio seems to be in incorrect state.

Adding Screenshot
https://drive.google.com/a/babylonhealth.com/file/d/0B4hXruznbpCVaF9uaEVvUHdHY1k/view?usp=sharing

@RunWith(JUnitPlatform::class)
object ObjectSampleSpec: Spek({
    given("something") {
        on("event") {
            it("verify") {
                assert(true)
            }
        }
    }
})

[raniejade]

@panell Looks like JUnit 4 is still the culprit, it does require the test tree to be known before hand. on is implemented as a dynamic test in JUnit 5.

[igorwojda]

So just to be 100% clear. Is there any way to use Speak with Android tests?

I have a sample test project with sample tests, but I wasn't able to make it works.
https://github.com/igorwojda/migrate-java-tests-to-kotlin

[raniejade]

@igorwojda If you're using @RunWith(..) it's a no. Do you have https://github.com/raniejade/spek-idea-plugin installed?

[igorwojda]

To be precise I am using Android Studio. So the Spek plugin is installed. I have removed @RunWith(..), no I am getting Test framework quit unexpecredly errror

"C:\Program Files\Android\Android Studio\jre\bin\java" -Didea.launcher.port=7532 "-Didea.launcher.bin.path=C:\Program Files\Android\Android Studio\bin" -Dfile.encoding=UTF-8 -classpath "C:\Users\igor.wojda\AppData\Local\Android\sdk\platforms\android-25\android.jar;C:\Users\igor.wojda\AppData\Local\Android\sdk\platforms\android-25\data\res;C:\Users\igor.wojda\AndroidStudioProjects\migrate-java-tests-to-kotlin\app\build\intermediates\classes\debug;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\net.bytebuddy\byte-buddy\1.6.14\871c3e49dc6183d0d361601c2f1d11abb1a6b48c\byte-buddy-1.6.14.jar;C:\Users\igor.wojda\.android\build-cache\5c18f1a673aa37060172f8b7eb6ae61c7596408d\output\res;C:\Users\igor.wojda\.android\build-cache\5c18f1a673aa37060172f8b7eb6ae61c7596408d\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\110558991186986bad9674c79415cdc509e95d9b\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\110558991186986bad9674c79415cdc509e95d9b\output\res;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\net.bytebuddy\byte-buddy-agent\1.6.14\ba1e5ba3a84fb2fbf2f4de9138df19665eec4d59\byte-buddy-agent-1.6.14.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.jetbrains.spek\spek-api\1.1.2\879088ec3b4fc9c636920a60aaac642bfb4d4824\spek-api-1.1.2.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-library\1.3\4785a3c21320980282f9f33d0d1264a69040538f\hamcrest-library-1.3.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-reflect\1.1.2-4\d003c37a5be39d8e5b0c6894378f75a7bd8482a1\kotlin-reflect-1.1.2-4.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\com.sksamuel.koors\koors\0.90.0\cbeea25c36a797332d74e7a95d3f1788c458a07b\koors-0.90.0.jar;C:\Program Files\Android\Android Studio\gradle\m2repository\com\google\guava\guava\18.0\guava-18.0.jar;C:\Users\igor.wojda\.android\build-cache\c799cd4a1b3c6d1dc16068f9b228e4f56cf7533e\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\c799cd4a1b3c6d1dc16068f9b228e4f56cf7533e\output\res;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\javax.annotation\javax.annotation-api\1.2\479c1e06db31c432330183f5cae684163f186146\javax.annotation-api-1.2.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\com.google.code.findbugs\annotations\2.0.1\9ef6656259841cebfb9fb0697bb122ada4485498\annotations-2.0.1.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\com.nhaarman\mockito-kotlin-kt1.1\1.5.0\3cb5419929540bcb25bdcc5e9ff7c8a1cb5831c\mockito-kotlin-kt1.1-1.5.0.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.mockito\mockito-core\2.8.47\48840cfced22ec0c07203a0201c5ae7bc12557b5\mockito-core-2.8.47.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\com.squareup\javawriter\2.1.1\67ff45d9ae02e583d0f9b3432a5ebbe05c30c966\javawriter-2.1.1.jar;C:\Users\igor.wojda\.android\build-cache\dc55a65348157facd2f72e911357e3d6c12b77c9\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\dc55a65348157facd2f72e911357e3d6c12b77c9\output\res;C:\Users\igor.wojda\.android\build-cache\7cfa643aafe3e799523b32ff75d607475cf1a23f\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\7cfa643aafe3e799523b32ff75d607475cf1a23f\output\res;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.javassist\javassist\3.18.2-GA\1180a90fbad732b60fb728a6bbe8fcb3151423d7\javassist-3.18.2-GA.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.reflections\reflections\0.9.10\c7f4ea230aedc757ca1499ebcfb2953c81cf8b67\reflections-0.9.10.jar;C:\Users\igor.wojda\AppData\Local\Android\sdk\extras\android\m2repository\com\android\support\support-annotations\25.3.1\support-annotations-25.3.1.jar;C:\Users\igor.wojda\.android\build-cache\994afbe79691429627aa6367f3c6211b66c96490\output\res;C:\Users\igor.wojda\.android\build-cache\994afbe79691429627aa6367f3c6211b66c96490\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\03fd48bf1be1d42babc33071a173831207824171\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\03fd48bf1be1d42babc33071a173831207824171\output\res;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.jetbrains.spek\spek-junit-platform-engine\1.1.2\49ae6e0e7e724b5b0522a2431a2b562b3ea2b6dd\spek-junit-platform-engine-1.1.2.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-stdlib\1.1.2-4\3f5f1498bdbb9611cb1413f794b3a3a5a6edef34\kotlin-stdlib-1.1.2-4.jar;C:\Users\igor.wojda\.android\build-cache\e7c4edc00773154489b88c3821f7aaaf0f114677\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\e7c4edc00773154489b88c3821f7aaaf0f114677\output\res;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-integration\1.3\5de0c73fef18917cd85d0ab70bb23818685e4dfd\hamcrest-integration-1.3.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.junit.platform\junit-platform-engine\1.0.0-M4\5e65bdd2d623dfc6a405911a6449f2c60808b0af\junit-platform-engine-1.0.0-M4.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.jetbrains.kotlin\kotlin-stdlib-jre7\1.1.2-4\e6a7403299b8f3126e3a1576c02e3f07ae5af6fa\kotlin-stdlib-jre7-1.1.2-4.jar;C:\Users\igor.wojda\.android\build-cache\9015b7af2256023df136701708afd9f1ff4db779\output\res;C:\Users\igor.wojda\.android\build-cache\9015b7af2256023df136701708afd9f1ff4db779\output\jars\classes.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\com.nhaarman\mockito-kotlin\1.5.0\25faa884f76375f76cdbd6651c4cebcde36d4117\mockito-kotlin-1.5.0.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.junit.platform\junit-platform-runner\1.0.0-M4\8e7235e28eda64f8216c0c055bd881f07e46e1b9\junit-platform-runner-1.0.0-M4.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.junit.platform\junit-platform-commons\1.0.0-M4\359eb461814bd7356de541d131d45473d394a168\junit-platform-commons-1.0.0-M4.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.objenesis\objenesis\2.5\612ecb799912ccf77cba9b3ed8c813da086076e9\objenesis-2.5.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\com.google.code.findbugs\jsr305\2.0.1\516c03b21d50a644d538de0f0369c620989cd8f0\jsr305-2.0.1.jar;C:\Users\igor.wojda\.android\build-cache\f03f3a7f1dc85c0b3d351925712f1b3a2245b4ee\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\f03f3a7f1dc85c0b3d351925712f1b3a2245b4ee\output\res;C:\Users\igor.wojda\.android\build-cache\71aa26d76a90dd5adb03d673e94b760d67d60e2c\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\71aa26d76a90dd5adb03d673e94b760d67d60e2c\output\res;C:\Users\igor.wojda\.android\build-cache\c2bc1df9c0039f535fb94d9b74add77a267586b1\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\c2bc1df9c0039f535fb94d9b74add77a267586b1\output\res;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.junit.platform\junit-platform-suite-api\1.0.0-M4\b68868db955435ca2e2900f5590879463ceb2a72\junit-platform-suite-api-1.0.0-M4.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\io.kotlintest\kotlintest\2.0.4\736f1910456eac362f522c9e859232d3899cbf74\kotlintest-2.0.4.jar;C:\Users\igor.wojda\.android\build-cache\e36dd458f362575ea7a997013cf291cfe0a4c11a\output\res;C:\Users\igor.wojda\.android\build-cache\e36dd458f362575ea7a997013cf291cfe0a4c11a\output\jars\classes.jar;C:\Users\igor.wojda\AppData\Local\Android\sdk\extras\m2repository\com\android\support\constraint\constraint-layout-solver\1.0.2\constraint-layout-solver-1.0.2.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.opentest4j\opentest4j\1.0.0-M2\e2caa64702a83336d3dd32c7f69c1f092156df6d\opentest4j-1.0.0-M2.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.jetbrains\annotations\13.0\919f0dfe192fb4e063e7dacadee7f8bb9a2672a9\annotations-13.0.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\javax.inject\javax.inject\1\6975da39a7040257bd51d21a231b76c915872d38\javax.inject-1.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.junit.platform\junit-platform-launcher\1.0.0-M4\d4e3657bc7cf204d6b9b55f0387fedc6eaad55fa\junit-platform-launcher-1.0.0-M4.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.greenrobot\eventbus\3.0.0\ddd99896e9569eaababbe81b35d80e1b91c4ad85\eventbus-3.0.0.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\org.hamcrest\hamcrest-core\1.3\42a25dc3219429f0e5d060061f71acb49bf010a0\hamcrest-core-1.3.jar;C:\Users\igor.wojda\.gradle\caches\modules-2\files-2.1\junit\junit\4.12\2973d150c0dc1fefe998f834810d68f278ea58ec\junit-4.12.jar;C:\Users\igor.wojda\.android\build-cache\2a537acf3455fc2ec3fd43ff60922dc5d41e8457\output\jars\classes.jar;C:\Users\igor.wojda\.android\build-cache\2a537acf3455fc2ec3fd43ff60922dc5d41e8457\output\res;C:\Users\igor.wojda\.android\build-cache\6cb0e479c9fc392e04c17b974cb4a35753427fe0\output\res;C:\Users\igor.wojda\.android\build-cache\6cb0e479c9fc392e04c17b974cb4a35753427fe0\output\jars\classes.jar;C:\Users\igor.wojda\.AndroidStudio2.3\config\plugins\spek-idea-plugin\lib\tooling-0.4.0.jar;C:\Program Files\Android\Android Studio\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain org.jetbrains.spek.tooling.MainKt --spec igorwojda.com.presentertest.feature.ProfilePresenterTest_Sample1_Kotlin_Spek
Testing started at 13:04 ...
Exception in thread "main" org.junit.platform.commons.util.PreconditionViolationException: Could not load class with name: igorwojda.com.presentertest.feature.ProfilePresenterTest_Sample1_Kotlin_Spek
    at org.junit.platform.engine.discovery.ClassSelector.lambda$getJavaClass$0(ClassSelector.java:71)
    at java.util.Optional.orElseThrow(Optional.java:290)
    at org.junit.platform.engine.discovery.ClassSelector.getJavaClass(ClassSelector.java:70)
    at org.jetbrains.spek.engine.SpekTestEngine.resolveSpecs(SpekTestEngine.kt:79)
    at org.jetbrains.spek.engine.SpekTestEngine.discover(SpekTestEngine.kt:50)
    at org.junit.platform.launcher.core.DefaultLauncher.discoverRoot(DefaultLauncher.java:115)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:87)
    at org.junit.platform.launcher.Launcher.execute(Launcher.java:93)
    at org.jetbrains.spek.tooling.runner.junit.JUnitPlatformSpekRunner.run(JUnitPlatformSpekRunner.kt:107)
    at org.jetbrains.spek.tooling.MainKt.main(Main.kt:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

BTW are you planning to add support for @RunWith(..)?

[raniejade]

BTW are you planning to add support for @RunWith(..)?

No, JUnit 4 is very problematic.

Can you try adding a Before launch gradle step to compile the test sources?

[igorwojda]

So used new plugin 0.4.1 and recreated configurations, but this didn't help me at all.

Exception in thread "main" org.junit.platform.commons.util.PreconditionViolationException: Could not load class with name: igorwojda.com.presentertest.feature.CorePresenterTestSpek
    at org.junit.platform.engine.discovery.ClassSelector.lambda$getJavaClass$0(ClassSelector.java:71)
    at java.util.Optional.orElseThrow(Optional.java:290)
    at org.junit.platform.engine.discovery.ClassSelector.getJavaClass(ClassSelector.java:70)
       ...

[raniejade]

@igorwojda probably related https://issuetracker.google.com/issues/38454212?

[igorwojda]

OK I nailed the reason of the problem. It looks that something becomes cached and test fails latter in the result.
Steps to reproduce:

1. Unisntall Spek Plugin (in my test setup it was actually never installed, but this should be similar)
2. Checkout the project https://github.com/igorwojda/migrate-java-tests-to-kotlin
3. Run tests in ProfilePresenterTest_Kotlin_Spek (using ide arrow next to class name)
4. Error (the one described above)
5. Install Spek
6. Restart AS, run test - error is still there

To fix the issue I needed to remove whole project folder and check out project again. Then having Spek plugin installed (before checking or probably before first test run) makes tests works as expected.

Tested on Android Studio 2.3.3 + SPek 0.4.1