Fs2: ClassCastException in translate

Created on 10 Jan 2018  路  31Comments  路  Source: typelevel/fs2

Tried building scodec-stream against 0.10.0-RC1 and encountered a ClassCastException:

java.lang.ClassCastException: cats.effect.IO$Suspend cannot be cast to scodec.stream.decode.Cursor
    at scodec.stream.decode.StreamDecoder$$anonfun$decode$1$$anon$1.apply(StreamDecoder.scala:46)
    at fs2.internal.Algebra$$anon$1.apply(Algebra.scala:351)
    at fs2.internal.Algebra$$anon$1.apply(Algebra.scala:347)
    at fs2.internal.FreeC$Eval.translate(FreeC.scala:74)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:60)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:16)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:13)
    at fs2.internal.FreeC$.go$1(FreeC.scala:114)
    at fs2.internal.FreeC$.fs2$internal$FreeC$$mkViewL(FreeC.scala:121)
    at fs2.internal.FreeC.viewL(FreeC.scala:54)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:16)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:13)
    at fs2.internal.FreeC$.go$1(FreeC.scala:114)
    at fs2.internal.FreeC$.fs2$internal$FreeC$$mkViewL(FreeC.scala:121)
    at fs2.internal.FreeC.viewL(FreeC.scala:54)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:16)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:13)
    at fs2.internal.FreeC$.go$1(FreeC.scala:114)
    at fs2.internal.FreeC$.fs2$internal$FreeC$$mkViewL(FreeC.scala:121)
    at fs2.internal.FreeC.viewL(FreeC.scala:54)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:16)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:13)
    at fs2.internal.FreeC$.go$1(FreeC.scala:114)
    at fs2.internal.FreeC$.fs2$internal$FreeC$$mkViewL(FreeC.scala:121)
    at fs2.internal.FreeC.viewL(FreeC.scala:54)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:16)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:13)
    at fs2.internal.FreeC$.go$1(FreeC.scala:114)
    at fs2.internal.FreeC$.fs2$internal$FreeC$$mkViewL(FreeC.scala:121)
    at fs2.internal.FreeC.viewL(FreeC.scala:54)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$translate$1.apply(FreeC.scala:57)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$suspend$1.apply(FreeC.scala:97)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:16)
    at fs2.internal.FreeC$$anonfun$flatMap$1.apply(FreeC.scala:13)
    at fs2.internal.FreeC$.go$1(FreeC.scala:114)
    at fs2.internal.FreeC$.fs2$internal$FreeC$$mkViewL(FreeC.scala:121)
    at fs2.internal.FreeC.viewL(FreeC.scala:54)
    at fs2.internal.Algebra$$anonfun$fs2$internal$Algebra$$uncons$1$1.apply(Algebra.scala:128)
    at fs2.internal.Algebra$$anonfun$fs2$internal$Algebra$$uncons$1$1.apply(Algebra.scala:128)
    at cats.effect.internals.IORunLoop$.step(IORunLoop.scala:154)
    at cats.effect.IO.unsafeRunTimed(IO.scala:220)
    at cats.effect.IO.unsafeRunSync(IO.scala:173)
    at scodec.stream.decode.SpaceLeakTest$$anonfun$1$$anonfun$apply$1.apply$mcZ$sp(SpaceLeakTest.scala:27)
    at scodec.stream.decode.SpaceLeakTest$$anonfun$1$$anonfun$apply$1.apply(SpaceLeakTest.scala:13)
    at scodec.stream.decode.SpaceLeakTest$$anonfun$1$$anonfun$apply$1.apply(SpaceLeakTest.scala:13)
    at org.scalacheck.Prop$.secure(Prop.scala:456)
    at scodec.stream.decode.SpaceLeakTest$$anonfun$1.apply(SpaceLeakTest.scala:13)
    at scodec.stream.decode.SpaceLeakTest$$anonfun$1.apply(SpaceLeakTest.scala:13)
    at org.scalacheck.Prop$$anonfun$delay$1.apply(Prop.scala:461)
    at org.scalacheck.Prop$$anonfun$delay$1.apply(Prop.scala:461)
    at org.scalacheck.Prop$$anonfun$apply$5.apply(Prop.scala:292)
    at org.scalacheck.Prop$$anonfun$apply$5.apply(Prop.scala:291)
    at org.scalacheck.PropFromFun.apply(Prop.scala:22)
    at org.scalacheck.Test$.org$scalacheck$Test$$workerFun$1(Test.scala:294)
    at org.scalacheck.Test$$anonfun$3.apply(Test.scala:323)
    at org.scalacheck.Test$$anonfun$3.apply(Test.scala:323)
    at org.scalacheck.Platform$.runWorkers(Platform.scala:40)
    at org.scalacheck.Test$.check(Test.scala:323)
    at org.scalacheck.ScalaCheckRunner$$anon$2$$anonfun$execute$3$$anonfun$apply$2.apply(ScalaCheckFramework.scala:102)
    at org.scalacheck.ScalaCheckRunner$$anon$2$$anonfun$execute$3$$anonfun$apply$2.apply(ScalaCheckFramework.scala:100)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
    at scala.collection.immutable.List.foreach(List.scala:381)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
    at org.scalacheck.ScalaCheckRunner$$anon$2$$anonfun$execute$3.apply(ScalaCheckFramework.scala:100)
    at org.scalacheck.ScalaCheckRunner$$anon$2$$anonfun$execute$3.apply(ScalaCheckFramework.scala:97)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186)
    at org.scalacheck.ScalaCheckRunner$$anon$2.execute(ScalaCheckFramework.scala:97)
    at sbt.TestRunner.runTest$1(TestFramework.scala:76)
    at sbt.TestRunner.run(TestFramework.scala:85)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
    at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
    at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
    at sbt.TestFunction.apply(TestFramework.scala:207)
    at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
    at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
    at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
    at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
picture mpilquist

Most helpful comment

ah excellent:-)

picture pchlupacek on 24 Jan 2018
🎉3

All 31 comments

not sure if translate`s Uncons case is correctly defined

pchlupacek picture pchlupacek on 10 Jan 2018

The bug is definitely in Algebra.translate but I have no idea how to fix. The culprit is the cast, which is unsound. Here's an expanded and annotated version:

case un: Uncons[F, x, O2] =>
  val x: FreeC[Algebra[G, x, ?], Unit] = FreeC.suspend(un.s.translate(algFtoG))
  val y: Algebra[G, O2, Option[(Segment[x, Unit], FreeC[Algebra[G, x, ?], Unit])]] =
    Uncons[G, x, O2](x, un.chunkSize, un.maxSteps)
  val res: Algebra[G, O2, X] = y
  res

The assignment of y to res is the issue. X expands to Option[(Segment[x, Unit], FreeC[Algebra[F, x, ?], Unit])] but y has a G in Option[(Segment[x, Unit], FreeC[Algebra[G, x, ?], Unit])].

mpilquist picture mpilquist on 10 Jan 2018

I don't know though -- UnconsAsync was just as unsound as far as I can tell (the natural transformation from Algebra[F, O2, ?] to Algebra[G, O2, ?] has the same unsound cast in the old UnconsAsync case and that one didn't cause a problem).

mpilquist picture mpilquist on 10 Jan 2018

is this the only case in which translate is actually called, and it's always faulty, or there's some code that triggers the cast exception and some that doesn't?

SystemFw picture SystemFw on 10 Jan 2018

@SystemFw Great question -- some cases work fine, so maybe the unsoundness argument is flawed.

scala> scodec.stream.decode.tryMany(scodec.codecs.int32).decode[IO](hex"00112233445566778899".bits).compile.toVector.unsafeRunSync
res8: Vector[Int] = Vector(1122867, 1146447479)
mpilquist picture mpilquist on 10 Jan 2018

well, I guess the next step would be minimisation. I'm still slightly out of my depth on the core interpreter unfortunately 馃槥 (can't wait for it to stabilise so I can deep dive)

SystemFw picture SystemFw on 10 Jan 2018

Mm another question is whether we can take scodec out of the picture and make it fail on its own

SystemFw picture SystemFw on 10 Jan 2018

Here's a reproduction without scodec:

    "translate (2)" in forAll { (s: PureStream[Int]) =>
      runLog(
        s.get
          .covary[Function0]
          .flatMap(i => Stream.eval(() => i))
          .flatMap(i => Stream.eval(() => i))
          .translate(new (Function0 ~> IO) {
            def apply[A](thunk: Function0[A]) = IO(thunk())
          })
      ) shouldBe runLog(s.get)
    }

Results in:

[info] - translate (2) *** FAILED *** (40 milliseconds)
[info]   ClassCastException was thrown during property evaluation.
[info]     Message: cats.effect.IO$Delay cannot be cast to scala.Function0
[info]     Occurred when passed generated values (
[info]       arg0 = PureStream(shrunk: 2 from shrunk: 3 from randomly-chunked: 4 Vector(6, 0),Stream(..)) // 2 shrinks
[info]     )
[info]   org.scalatest.exceptions.GeneratorDrivenPropertyCheckFailedException:
...
[info]   Cause: java.lang.ClassCastException: cats.effect.IO$Delay cannot be cast to scala.Function0
[info]   at fs2.StreamSpec$$anon$1.apply(StreamSpec.scala:212)
[info]   at fs2.internal.Algebra$$anon$1.apply(Algebra.scala:351)
[info]   at fs2.internal.Algebra$$anon$1.apply(Algebra.scala:347)
[info]   at fs2.internal.FreeC$Eval.translate(FreeC.scala:74)
[info]   at fs2.internal.FreeC.$anonfun$translate$1(FreeC.scala:60)
[info]   at fs2.internal.FreeC$.$anonfun$suspend$1(FreeC.scala:97)
[info]   at fs2.internal.FreeC.$anonfun$flatMap$1(FreeC.scala:16)
[info]   at fs2.internal.FreeC$.go$1(FreeC.scala:114)
[info]   at fs2.internal.FreeC$.fs2$internal$FreeC$$mkViewL(FreeC.scala:121)
[info]   at fs2.internal.FreeC.viewL(FreeC.scala:54)
[info]   at fs2.internal.Algebra$.$anonfun$compileFoldLoop$1(Algebra.scala:128)
[info]   at cats.effect.internals.IORunLoop$.step(IORunLoop.scala:154)
mpilquist picture mpilquist on 11 Jan 2018

If the second flatMap is elided, everything works, which points back in the direction of the soundness argument again.

mpilquist picture mpilquist on 11 Jan 2018

If you elide the first flatMap and leave the second, things work as well

SystemFw picture SystemFw on 11 Jan 2018

Right -- in order to get the class cast exception, you need an Uncons step to yield a stream which has an eval and is then subsequently flat-mapped. I think we may need to go back to the design where uncons is handled outside the algebra.

mpilquist picture mpilquist on 11 Jan 2018

I've lost track of whether having uncons outside will mess up interruption or not

SystemFw picture SystemFw on 11 Jan 2018

Me too -- hoping @pchlupacek has some ideas. :)

mpilquist picture mpilquist on 11 Jan 2018

@mpilquist @SystemFw excellent you put up minimalized case. I belive indeed the issue is related to NOT transalting the F ~> G in tail poisition after uncons is made.

The reason why Uncons went to algebra is to solve SoE (#1035). Later it turned out that it allows for much more reliable interruption implementation.

I have some ideas on fixing this. I will explore them today, and let you know the progress.

pchlupacek picture pchlupacek on 11 Jan 2018

@mpilquist @SystemFw I have managed to define translate with FreeC flatMap and get rid of effect instance from the algebra (bugfix/translate-typecast). As such, there are no longer any typecasts in translate, unfortunatelly example still fails. The problem is 100% nested uncons, that mean one on tail, after previous uncons. I am not sure what to think now, but will investigate further. Any eyes welcome :-)

pchlupacek picture pchlupacek on 11 Jan 2018

Also this is the minimal stream where this fails:
Stream.emits(List(List(1), List(2))).flatMap(Stream.emits)
supplied to spec above.

pchlupacek picture pchlupacek on 11 Jan 2018

one approach I am thinking of now is to define uncons as 

final case class Uncons[F[_], X, O](
s: FreeC[Algebra[F, X, ?], Unit],
f: Option[(Segment[X, Unit], FreeC[Algebra[F, X, ?], Unit])] => F[FreeC[Algebra[F, O, ?], Unit])], 
                                      chunkSize: Int,
                                      maxSteps: Long)

But not sure if that is possible

pchlupacek picture pchlupacek on 11 Jan 2018

of course that will lead to definition of uncons like 

def uncons[O2, R](f: Option[(Segment[O, Unit], Stream[F, O])] => Pull[F,O2,R]): Pull[F, O2, R] = ???

which is quite a significant change of the API.

pchlupacek picture pchlupacek on 11 Jan 2018

OTOH perhaps deffinition of 

def uncons[F2, O2](f:  Option[(Segment[O, Unit], Stream[F, O])] => Stream[F2, O2]]): Stream[F2, O2]

May allow to get rid of Pull alltogether :-)

pchlupacek picture pchlupacek on 11 Jan 2018

well, I think the advantage of having both Pull and Stream is that having a Monad instance on the return type (Pull) makes it easier to write stateful combinators (return the rest), whereas having it on the output type (Stream) is better for program flow/FRP. In fact this is an advantage that even haskell libraries forego

SystemFw picture SystemFw on 11 Jan 2018

These are very rough ideas. I think the root of the cause is that Uncons are really fused two operations into one, I mean Uncons itself and F ~> G, given F is Algebra[F, X, ? ], G is Algebra[F, O, ?]. Would be interesting if we can sort of make it two independent operations, in that case, we may perhaps define translate safely.

I have guess that root of the cause why translate doesn't work is that the uncons implementation in compile relies on the fact that f from FreeC.Bind always results in Stream[F, O] and nested uncons are build by flatMaping that f on top of each other. However in case of translate that could mean we miss opportunity to inject translate at correct point.

pchlupacek picture pchlupacek on 11 Jan 2018

Also we may still have Pull, available but would be just type alias something like 

type Pull[F,R] = FreeC[Algebra[F, Nothing, ?], R]
type Stream[F,O] = FreeC[Algebra[F, O, ?], Unit]
pchlupacek picture pchlupacek on 11 Jan 2018

which is sort of today anyhow.

pchlupacek picture pchlupacek on 11 Jan 2018

I really don鈥檛 want to change the Pull API at this point (and it is important for pulls to be able to emit values)

mpilquist picture mpilquist on 11 Jan 2018

yes, completely understand just bringing ideas on the table :-)

pchlupacek picture pchlupacek on 11 Jan 2018

@mpilquist Fixed in #1073. Could you perhaps try again to run that branch against the scodec tests? Thanks.

pchlupacek picture pchlupacek on 24 Jan 2018

Awesome!

[info] + scodec.stream.encode.emit: OK, passed 100 tests.
[info] + scodec.stream.tryMany-example: OK, proved property.
[info] + scodec.stream.sepBy: OK, passed 100 tests.
[info] + scodec.stream.many1: OK, passed 100 tests.
[info] + scodec.stream.decodeResource: OK, passed 100 tests.
[info] + scodec.stream.process.fixed size.ints: OK, passed 100 tests.
[info] + scodec.stream.or: OK, passed 100 tests.
[info] + scodec.stream.many/tryMany: OK, passed 100 tests.
[info] + scodec.stream.toLazyBitVector: OK, passed 100 tests.
[info] + scodec.stream.isolate: OK, passed 100 tests.
[info] + scodec.stream.peek: OK, passed 100 tests.
[info] + scodec.stream.process.fixed size.strings: OK, passed 100 tests.
[info] + space-leak.head of stream not retained: OK, proved property.
[info] Passed: Total 13, Failed 0, Errors 0, Passed 13
mpilquist picture mpilquist on 24 Jan 2018

ah excellent:-)

pchlupacek picture pchlupacek on 24 Jan 2018
🎉3

that's great! :)

SystemFw picture SystemFw on 24 Jan 2018

@ChristopherDavenport tried it on doobie and the tests pass.

tpolecat picture tpolecat on 24 Jan 2018
🎉1

closed by #1073

pchlupacek picture pchlupacek on 24 Jan 2018
🎉1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

andrelfpinto picture andrelfpinto  路  6Comments
ScalaWilliam picture ScalaWilliam  路  3Comments
kubukoz picture kubukoz  路  5Comments
mpilquist picture mpilquist  路  13Comments
Taig picture Taig  路  7Comments