Arrow: Arrow 0.7.x causes builds to fail on Android projects

[jrgonzalezg]

Just adding the Arrow dependencies on an Android project with Java 8 features enabled causes the builds to fail due to a missing traverse related class / lambda during the desugar task.

Sample repo here: https://github.com/jrgonzalezg/arrow-android-failing

[jrgonzalezg]

Did a bit more research and it seems this also affected other projects (not using Arrow) even with fairly standard lambdas like this one: https://stackoverflow.com/questions/49866403/gradle-build-failure-couldnt-find-outer-class-xxxx So it maybe a more broad issue with lambdas that Arrow just triggered but not any issue in Arrow itself.

After that I checked again with Kotlin 1.2.41 and Android Gradle Plugin 3.2.0-alpha12 and the build succeeded. So maybe it was a bug either in Kotlin compiler fixed on 1.2.41 or the desugar process which may be fixed in canary versions of Android Studio / Gradle plugin. So while this means Arrow can't be used on current stable version of Android tools, it may work on the next one, so maybe this issue could be closed although it would be nice to know why it happened and still I haven't figured that out :disappointed:

[jrgonzalezg]

Checked separately and it seems to be the Android Gradle Plugin and not the Kotlin version which makes the build work. Still, it is now failing on generating release versions in the ProGuard stage with similar errors:

Warning: arrow.data.ListK$traverse$1$2$1: can't find referenced class arrow.data.ListK$traverse$1$2

So I guess the problem about the class not being there still occurs, but now it compiles and fails in ProGuard instead of desugar. I am not sure if it will be safe to make ProGuard ignore all that issues (and it may require quite a bit of ProGuard rules or a broad / clever one). Could it be that the Kotlin compiler is omitting things in the class files that it shouldn't so the class files do not fully conform to the specification?. Maybe someone from Kotlin or with good knowledge of the class file requirements could figure out what is wrong here.

[jrgonzalezg]

I also see that there is a PR updating Kotlin: https://github.com/arrow-kt/arrow/pull/820 Maybe that one could be further updated from 1.2.40 to 1.2.41 and we could check this issue again once a new release of Arrow compiled with a more recent version of Kotlin is deployed. If it was a bug on the compiler it may now be fixed and new artifacts could work again. What do you think @raulraja ?

[raulraja]

@jrgonzalezg asked @jereksel to bump up to latest so we can test it in the upcoming release. I still have no idea why this is showing up. I suspect this is some kind of inlining compiler bug by the error message but would be good to javap those files to see if arrow.data.ListK$traverse$1$2 is actually there.

[jrgonzalezg]

@raulraja How would that be done? What I can see is that if you download the arrow-data-0.7.1.jar from http://search.maven.org/remotecontent?filepath=io/arrow-kt/arrow-data/0.7.1/arrow-data-0.7.1.jar and unzip it, you get a arrow/data/ListK$traverse$1$2$1.class but not a arrow/data/ListK$traverse$1$2.class. The full list of ListK classes on that artifact is:

ForListK.class
Higherkind_arrow_data_ListKKt.class
ListK$ap$1.class
ListK.class
ListK$Companion.class
ListK$foldRight$1$1.class
ListK$foldRight$1.class
ListK$foldRight$2.class
ListKKt.class
ListK$map2$1$1.class
ListK$map2$1.class
ListK$mapFilter$1.class
ListK$traverse$1$1.class
ListK$traverse$1$2$1.class
ListK$traverse$$inlined$run$lambda$1.class

Using javap *.class for all the classes on that artifact I don't see any reference to ListK$traverse$1$2 anywhere, not sure why the message mentions a reference or if the $ alone already implies subclass / inner class reference and thats why it tries to find it.

[jrgonzalezg]

I have also tried compiling @jereksel branch to try to see if the artifacts there do not have this problem but I can't get that one to compile, I keep getting:

e: [kapt] An exception occurred: java.lang.IllegalStateException: currentFile.absoluteFile.parentFile must not be null
        at arrow.common.utils.ProcessorUtilsKt.recurseFilesUpwards(ProcessorUtils.kt:174)

[jrgonzalezg]

@jereksel @raulraja Seems builds are also broken in master. Last commit that works is https://github.com/arrow-kt/arrow/commit/32372492cecc72032663b94b007e356bbc5e7959 and first that fails is https://github.com/arrow-kt/arrow/commit/40baca826f26fc433651ef2b3af9f19c75e8a531 by @JorgeCastilloPrz On builds after that one I get an automatic deletion of infographic/arrow-infographic.txt on "clean" task (./gradlew clean assemble) which could be the file that causes the error. But even if I restore that file and just do ./gradlew assemble (removing the clean) it still fails. Maybe Jorge knows what could be wrong with that commit.

[jrgonzalezg]

Also tested against a rebased master removing some of the commits that cause issues but increasing Kotlin version to 1.2.41 and the contents of arrow-data-0.7.2-SNAPSHOT.jar are the same I reported above for the currently published version, so maybe that update will not help for the issue here :disappointed:

[JorgeCastilloPrz]

Deletion of the infographic.txt file is intended on clean. That's the expected behavior. It's interesting, because I've pulled master right now and run ./gradlew clean build on the root arrow dir and it's a successful build. Travis is also green on master branch.

[jereksel]

@jrgonzalezg Arrow source code must be in "arrow" directory: https://github.com/arrow-kt/arrow/commit/40baca826f26fc433651ef2b3af9f19c75e8a531#diff-ea6113863a28ffe3f15442a287e31dadR103

[jrgonzalezg]

Thanks @jereksel I noticed that this afternoon and even commented it on Slack but it still failed when I renamed my repo root directory from arrow-jrgonzalezg to arrow. But I tried again now and it worked, so I'm not sure if I made a typo on the first renaming attempt or what happened.

Anyway, couldn't the root directory be determined from gradle or something similar instead of hardcoding it? git allows you to clone a repo into a directory with any name and it does not seem very reliable that the project does not work unless the directory name is fixed. Also the reason is very difficult to find.

The current behavior also makes it difficult to have several arrow repos cloned as I have had for a while (arrow-arrow-kt, arrow-jrgonzalez, ...) cause now only one named arrow will work. Is this a limitation of the annotation processors that can not be passed the root folder? If it is and they only get a limited context about location of files, could it be possible to make it recurse to a known directory with fixed name inside the repo instead of the containing root folder? For instance something like recurse up to "modules" and use an extra directory parent from that. In that way it could probably work in all cases despite were the repo is cloned. @JorgeCastilloPrz ?

[jereksel]

@jrgonzalezg I'm sure this can be done another way (i like idea with parent of modules). I've created another issue for that problem: #824

[jrgonzalezg]

I have updated the sample repository with another branch to show builds failing also when Java compatibility is set to Java 7. For that new case it fails on the ProGuard task instead of the desugar one. I have also updated the sample repo's README.md with instructions for triggering the build problem in both cases. See https://github.com/jrgonzalezg/arrow-android-failing

[JorgeCastilloPrz]

I've tried to add Arrow to a repo with plain Android and targeting java 8. This is a very standard configuration for Android projects:

android {
  //...
  compileOptions {
      sourceCompatibility JavaVersion.VERSION_1_8
      targetCompatibility JavaVersion.VERSION_1_8
  }
}

And as @jrgonzalezg said many times, you get a compile time error because of desugaring:

Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/data/ListK$traverse$1$2 of arrow/data/ListK$traverse$1$2$1
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1079)
        at com.google.devtools.build.android.desugar.ClassVsInterface.isOuterInterface(ClassVsInterface.java:56)
        at com.google.devtools.build.android.desugar.InterfaceDesugaring.visitOuterClass(InterfaceDesugaring.java:246)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:638)
        at org.objectweb.asm.ClassReader.accept(ClassReader.java:500)
        at com.google.devtools.build.android.desugar.Desugar.desugarClassesInInput(Desugar.java:477)
        at com.google.devtools.build.android.desugar.Desugar.desugarOneInput(Desugar.java:361)
        at com.google.devtools.build.android.desugar.Desugar.desugar(Desugar.java:314)
        at com.google.devtools.build.android.desugar.Desugar.main(Desugar.java:711)

The artifacts used are:

def arrow_version = '0.7.2'

implementation "io.arrow-kt:arrow-core:$arrow_version"
implementation "io.arrow-kt:arrow-syntax:$arrow_version"
implementation "io.arrow-kt:arrow-data:$arrow_version"

So yeah, there's a problem on this, since Arrow can't be used on Android like this. (People does not target jdk7 anymore if they can avoid it, since Android supports desugaring from some given java 8 features out of the box). I find it quite important. cc @raulraja @jereksel @pakoito.

[jereksel]

Google bug report: https://issuetracker.google.com/issues/72750890 points to this bug report: https://youtrack.jetbrains.com/issue/KT-16084 which is "in progress".

Only thing we can do now is to test workaround and, if it works, post it here.

[pakoito]

We can rewrite the classes not to use that pattern, right?

[pakoito]

https://github.com/arrow-kt/arrow/blob/master/modules/core/arrow-data/src/main/kotlin/arrow/data/ListK.kt#L27

fun <G, B> traverse(GA: Applicative<G>, f: (A) -> Kind<G, B>): Kind<G, ListK<B>> =
    foldRight(Eval.always { GA.just(emptyList<B>().k()) }) { a, eval ->
      GA.run { f(a).map2Eval(eval) { (listOf(it.a) + it.b).k() } }
    }.value()
  }

[pakoito]

If that fails we can make a local lambda or local function or capture a function reference to map2Eval, until it works.

[jereksel]

Yeah, I think for now it'll do - we should comment old code out and restore it after KT-16084 is fixed.

[pakoito]

Waiting to see how many more instances of the issue pop up.

[pakoito]

Posted in gitter:

Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.ListK$traverse$1$2$1: can't find referenced class arrow.data.ListK$traverse$1$2
Warning: arrow.data.ListK$traverse$1$2$1: can't find referenced class arrow.data.ListK$traverse$1$2
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2

[jrgonzalezg]

Just found the way to update the sample repo to Arrow 0.7.3-SNAPSHOT: https://github.com/jrgonzalezg/arrow-android-failing

[jrgonzalezg]

This is for Arrow 0.7.3-SNAPSHOT on the repo above java7 branch:

Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.EitherT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.EitherT$Companion$tailRecM$1$1
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.OptionT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.OptionT$Companion$tailRecM$1$1
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.SequenceK$traverse$1$2$1: can't find referenced class arrow.data.SequenceK$traverse$1$2
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$modifyF$1$1$1: can't find referenced class arrow.data.StateT$Companion$modifyF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$setF$1$1$1: can't find referenced class arrow.data.StateT$Companion$setF$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.data.WriterT$Companion$tailRecM$1$1$1: can't find referenced class arrow.data.WriterT$Companion$tailRecM$1$1
Warning: arrow.instances.EithertKt$traverse$1$1$1$1: can't find referenced class arrow.instances.EithertKt$traverse$1$1
Warning: arrow.instances.EithertKt$traverse$1$1$1$1: can't find referenced class arrow.instances.EithertKt$traverse$1$1
Warning: arrow.instances.OptiontKt$traverse$1$1$1$1: can't find referenced class arrow.instances.OptiontKt$traverse$1$1
Warning: arrow.instances.OptiontKt$traverse$1$1$1$1: can't find referenced class arrow.instances.OptiontKt$traverse$1$1
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2
Warning: arrow.typeclasses.Foldable$traverse_$1$2$1: can't find referenced class arrow.typeclasses.Foldable$traverse_$1$2

[jrgonzalezg]

So it seems the fix for ListK may have worked, but we got a few extra occurrences too :sweat_smile:

[jrgonzalezg]

Seems java7 reporting the issues in ProGuard gives more info than the master java 8 branch which fails in desugar with just these reported:

Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/data/StateT$Companion$tailRecM$1$1$1 of arrow/data/StateT$Companion$tailRecM$1$1$1$1
Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/instances/EithertKt$traverse$1$1 of arrow/instances/EithertKt$traverse$1$1$1$1
Exception in thread "main" java.lang.NullPointerException: Couldn't find outer class arrow/typeclasses/Foldable$traverse_$1$2 of arrow/typeclasses/Foldable$traverse_$1$2$1

but I suspect it may be giving up before reporting all the errors like ProGuard.

[pakoito]

Segun mis ultimos tests el fix de #955 lo ha arreglado todo. Puedes comprobarlo, @jrgonzalezg ??

[erickok]

@pakoito This fixed by Proguard-enabled Android project release build (tested using JitPack build of master branch).

[jrgonzalezg]

Seems this is fixed with Arrow 0.7.3, so we can finally close this :tada:

[m0skit0]

Looks like it is happening again. Kotlin 1.3.41, Arrow 0.9.0.

Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$modify$1$1$1: can't find referenced class arrow.data.StateT$Companion$modify$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1
Warning: arrow.data.StateT$Companion$tailRecM$1$1$1$1: can't find referenced class arrow.data.StateT$Companion$tailRecM$1$1$1

[pakoito]

Yeah, that bug has been haunting us for a while. It's fixed in 0.10.1

[aballano]

@m0skit0 @pakoito To be precise this is fixed in the current 0.10.2 version as 0.10.1 is failing in API < 24 atm

[jrgonzalezg]

@aballano there is no 0.10.2, right? is it going to be generated soon?

[JorgeCastilloPrz]

The snapshot is available, we're aiming for a release during next week :)