Redex: RuntimeError: redex-all crashed with exit code -6!

[MaTriXy]

Running redex on my app encounters this issue.

redex app.apk -o tiko.apk

libredex/IRTypeChecker.cpp:388: void irtc_impl::TypeInference::analyze_instruction(IRInstruction *, TypeEnvironment *) const: assertion `!is_object(type)' failed.
Unexpected instruction 'FILLED_NEW_ARRAY v0, v1, v2, [Ljava/lang/String;'.

0   redex-all                           0x0000000109fb2c9c _Z11assert_failPKcS0_jS0_S0_z + 188
1   redex-all                           0x000000010a026b50 _ZNK9irtc_impl13TypeInference19analyze_instructionEP13IRInstructionP34PatriciaTreeMapAbstractEnvironmentIj20FiniteAbstractDomainI6IRType16BitVectorLatticeIS5_Lm16ENSt3__14hashIiEENS7_8equal_toIS5_EEEN8fad_impl17BitVectorEncodingILm16EEEXadL_ZNS_12type_latticeEEEEE + 14064
2   redex-all                           0x000000010a03604f _ZNK9irtc_impl13TypeInference12analyze_nodeERKP5BlockP34PatriciaTreeMapAbstractEnvironmentIj20FiniteAbstractDomainI6IRType16BitVectorLatticeIS7_Lm16ENSt3__14hashIiEENS9_8equal_toIS7_EEEN8fad_impl17BitVectorEncodingILm16EEEXadL_ZNS_12type_latticeEEEEE + 111
3   redex-all                           0x000000010a027a8e _ZN25MonotonicFixpointIteratorIN3cfg14GraphInterfaceE34PatriciaTreeMapAbstractEnvironmentIj20FiniteAbstractDomainI6IRType16BitVectorLatticeIS4_Lm16ENSt3__14hashIiEENS6_8equal_toIS4_EEEN8fad_impl17BitVectorEncodingILm16EEEXadL_ZN9irtc_impl12type_latticeEEEEENS7_IP5BlockEEE3runERKSH_ + 174
4   redex-all                           0x000000010a023127 _ZN9irtc_impl13TypeInference3runEP9DexMethod + 983
5   redex-all                           0x000000010a02221e _ZN13IRTypeChecker3runEv + 574
6   redex-all                           0x000000010a0436a8 _ZNSt3__110__function6__funcIZN11PassManager16run_type_checkerERKNS_6vectorIP8DexClassNS_9allocatorIS5_EEEEbbE3$_1NS6_ISB_EEFvP9DexMethodEEclEOSE_ + 72
7   redex-all                           0x0000000109deab25 _ZN4walk15iterate_methodsEPK8DexClassRKNSt3__18functionIFvP9DexMethodEEE + 165
8   redex-all                           0x0000000109debac2 _ZNSt3__110__function6__funcIZ17workqueue_foreachIP9DexMethodE9WorkQueueIT_DnDnERKNS_8functionIFvS6_EEEjEUlRDnS4_E_NS_9allocatorISE_EEFDnSD_S4_EEclESD_OS4_ + 34
9   redex-all                           0x0000000109deff42 _ZN9WorkQueueIP8DexClassDnmE7consumeEP11WorkerStateIS1_DnmES1_ + 50
10  redex-all                           0x0000000109defa91 _ZZN9WorkQueueIP8DexClassDnDnE7run_allERKDnENKUlP11WorkerStateIS1_DnDnEmE_clES7_m + 129
11  libboost_thread-mt.dylib            0x000000010a88b2ac _ZN5boost12_GLOBAL__N_1L12thread_proxyEPv + 156
12  libsystem_pthread.dylib             0x00007fff798036c1 _pthread_body + 340
13  libsystem_pthread.dylib             0x00007fff7980356d _pthread_body + 0
14  libsystem_pthread.dylib             0x00007fff79802c5d thread_start + 13
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Redex assertion failure
0   redex-all                           0x0000000109fb2b92 _Z23crash_backtrace_handleri + 50
1   libsystem_platform.dylib            0x00007fff797f9f5a _sigtramp + 26
2   ???                                 0x00007fb800020008 0x0 + 140428250841096
3   libsystem_c.dylib                   0x00007fff79624312 abort + 127
4   libc++abi.dylib                     0x00007fff775fff8f __cxa_bad_cast + 0
5   libc++abi.dylib                     0x00007fff77600113 _ZL25default_terminate_handlerv + 241
6   libobjc.A.dylib                     0x00007fff7898aeab _ZL15_objc_terminatev + 105
7   libc++abi.dylib                     0x00007fff7761b7c9 _ZSt11__terminatePFvvE + 8
8   libc++abi.dylib                     0x00007fff7761b26d _ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception + 0
9   redex-all                           0x0000000109fb2cdd _Z11assert_failPKcS0_jS0_S0_z + 253
10  redex-all                           0x000000010a026b50 _ZNK9irtc_impl13TypeInference19analyze_instructionEP13IRInstructionP34PatriciaTreeMapAbstractEnvironmentIj20FiniteAbstractDomainI6IRType16BitVectorLatticeIS5_Lm16ENSt3__14hashIiEENS7_8equal_toIS5_EEEN8fad_impl17BitVectorEncodingILm16EEEXadL_ZNS_12type_latticeEEEEE + 14064
11  redex-all                           0x000000010a03604f _ZNK9irtc_impl13TypeInference12analyze_nodeERKP5BlockP34PatriciaTreeMapAbstractEnvironmentIj20FiniteAbstractDomainI6IRType16BitVectorLatticeIS7_Lm16ENSt3__14hashIiEENS9_8equal_toIS7_EEEN8fad_impl17BitVectorEncodingILm16EEEXadL_ZNS_12type_latticeEEEEE + 111
12  redex-all                           0x000000010a027a8e _ZN25MonotonicFixpointIteratorIN3cfg14GraphInterfaceE34PatriciaTreeMapAbstractEnvironmentIj20FiniteAbstractDomainI6IRType16BitVectorLatticeIS4_Lm16ENSt3__14hashIiEENS6_8equal_toIS4_EEEN8fad_impl17BitVectorEncodingILm16EEEXadL_ZN9irtc_impl12type_latticeEEEEENS7_IP5BlockEEE3runERKSH_ + 174
13  redex-all                           0x000000010a023127 _ZN9irtc_impl13TypeInference3runEP9DexMethod + 983
14  redex-all                           0x000000010a02221e _ZN13IRTypeChecker3runEv + 574
15  redex-all                           0x000000010a0436a8 _ZNSt3__110__function6__funcIZN11PassManager16run_type_checkerERKNS_6vectorIP8DexClassNS_9allocatorIS5_EEEEbbE3$_1NS6_ISB_EEFvP9DexMethodEEclEOSE_ + 72
16  redex-all                           0x0000000109deab25 _ZN4walk15iterate_methodsEPK8DexClassRKNSt3__18functionIFvP9DexMethodEEE + 165
17  redex-all                           0x0000000109debac2 _ZNSt3__110__function6__funcIZ17workqueue_foreachIP9DexMethodE9WorkQueueIT_DnDnERKNS_8functionIFvS6_EEEjEUlRDnS4_E_NS_9allocatorISE_EEFDnSD_S4_EEclESD_OS4_ + 34
18  redex-all                           0x0000000109deff42 _ZN9WorkQueueIP8DexClassDnmE7consumeEP11WorkerStateIS1_DnmES1_ + 50
19  redex-all                           0x0000000109defa91 _ZZN9WorkQueueIP8DexClassDnDnE7run_allERKDnENKUlP11WorkerStateIS1_DnDnEmE_clES7_m + 129
20  libboost_thread-mt.dylib            0x000000010a88b2ac _ZN5boost12_GLOBAL__N_1L12thread_proxyEPv + 156
21  libsystem_pthread.dylib             0x00007fff798036c1 _pthread_body + 340
22  libsystem_pthread.dylib             0x00007fff7980356d _pthread_body + 0
23  libsystem_pthread.dylib             0x00007fff79802c5d thread_start + 13
Traceback (most recent call last):
  File "/tmp/redex.8eBT2t/redex.py", line 173, in run_pass
    subprocess.check_call(args, env=env)
  File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/redex.8eBT2t/redex-all', '--apkdir', '/tmp/redex.8eBT2t/tmp_xj_43iv.redex_extracted_apk', '--outdir', '/tmp/redex.8eBT2t/tmpxnv1s53q.redex_dexen', '/tmp/redex.8eBT2t/tmpxnv1s53q.redex_dexen/dex0/classes.dex', '/tmp/redex.8eBT2t/tmpxnv1s53q.redex_dexen/dex1/classes2.dex']' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/redex.8eBT2t/redex.py", line 687, in <module>
    run_redex(args)
  File "/tmp/redex.8eBT2t/redex.py", line 605, in run_redex
    debugger)
  File "/tmp/redex.8eBT2t/redex.py", line 186, in run_pass
    'by running %(lldb_script_name)s') % script_filenames)
RuntimeError: redex-all crashed with exit code -6!
 You can re-run it under gdb by running /tmp/redex.8eBT2t/redex-gdb-eoxv9t6l.sh or under lldb by running /tmp/redex.8eBT2t/redex-lldb-zp3xjsbi.sh

[justinjhendrick]

Interesting. There's a comment right above that assertion saying this:

// Although the Dalvik bytecode specification states that a
// filled-new-array operation could be used with an array of references,
// the Dex compiler seems to never generate that case. The assert is used
// here as a safeguard.

Which java bytecode to dex bytcode compiler did you use to create app.apk? If it's dx, which version? Or was it d8?

Some details about d8 here

ccing the author of that file, @arnaudvenet

[arnaudvenet]

dx seems to never generate a filled-new-array with a reference type, which is why I put the assert there (it also somewhat simplifies the construction of the points-to semantics). It looks like this does happen in the wild. I can modify the type inference algorithm to take care of this situation.

I'm also curious as to whether this occurs in bytecode generated by dx or d8.

[MaTriXy]

@justinjhendrick dx was used with Android studio 3.0.1
Project includes Kotlin and Native code.
Didn't try d8 as i thought it might not work with redex.

I can try if this is needed.

[justinjhendrick]

Interesting. Maybe it was the Kotlin code that generated different dex bytecode than usual.

No need to change your build settings, Arnaud is working on a fix now. We're using dx internally as well and haven't tried d8 yet either, so we're not sure what would happen.

[arnaudvenet]

This should have been fixed by commit ead3f9aab5b274c1bbc1671f3bf40b9faeb7232.

[MaTriXy]

@arnaudvenet I will test now. Thanks.

[MaTriXy]

Tested and returned an apk back without the issue.
Thank you.