Redex: Trying to encode too many field refs in dex

Created on 11 Jul 2020  路  14Comments  路  Source: facebook/redex

when I run command redex input.apk -o out.apk ,some apps could success output a new apk file , but My company's app couldn't . we use the multidex. The method size can't be more than 65535,
here is the error log :

Running IRTypeChecker...
        IRTypeChecker completed in 9.6 seconds
    Running optimization passes completed in 139.4 seconds
        Instruction lowering completed in 3.6 seconds
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::error_info_injector<RedexException>: libredex/DexOutput.cpp:846: void DexOutput::generate_method_data(): assertion `dodx->field_to_idx().size() <= kMaxFieldRefs' failed.
Trying to encode too many field refs in dex 22: 65542 (limit: 65536)
0   redex-all                           0x000000010ac58332 _Z23crash_backtrace_handleri + 50
1   libsystem_platform.dylib            0x00007fff73aae5fd _sigtramp + 29
2   ???                                 0x0000000000000400 0x0 + 1024
3   libsystem_c.dylib                   0x00007fff73984808 abort + 120
4   libc++abi.dylib                     0x00007fff70be4458 abort_message + 231
5   libc++abi.dylib                     0x00007fff70bd58a7 _ZL28demangling_terminate_handlerv + 238
6   libobjc.A.dylib                     0x00007fff727105b1 _ZL15_objc_terminatev + 104
7   libc++abi.dylib                     0x00007fff70be3887 _ZSt11__terminatePFvvE + 8
8   libc++abi.dylib                     0x00007fff70be61a2 __cxa_get_exception_ptr + 0
9   libc++abi.dylib                     0x00007fff70be6169 _ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception + 0
10  redex-all                           0x000000010ac58929 _Z11assert_failPKcS0_jS0_10RedexErrorS0_z + 905
11  redex-all                           0x000000010aca7455 _ZN9DexOutput20generate_method_dataEv + 453
12  redex-all                           0x000000010acb04f8 _ZN9DexOutput7prepareE8SortModeRKNSt3__16vectorIS0_NS1_9allocatorIS0_EEEERK11ConfigFilesRKNS1_12basic_stringIcNS1_11char_traitsIcEENS3_IcEEEE + 920
13  redex-all                           0x000000010acb2a77 _Z20write_classes_to_dexRK12RedexOptionsRKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEPNS2_6vectorIP8DexClassNS6_ISD_EEEEPNS2_13unordered_mapIP9DexStringN8facebook7LocatorENS2_4hashISJ_EENS2_8equal_toISJ_EENS6_INS2_4pairIKSJ_SL_EEEEEEmmRK11ConfigFilesP14PositionMapperPNSH_IP9DexMethodyNSM_IS12_EENSO_IS12_EENS6_INSQ_IKS12_yEEEEEEPNSH_IP7DexCodeNSB_I13DebugLineItemNS6_IS1C_EEEENSM_IS1B_EENSO_IS1B_EENS6_INSQ_IKS1B_S1E_EEEEEEP12IODIMetadataSA_PK12PostLoweringi + 2359
14  redex-all                           0x000000010ac02af0 _ZN12_GLOBAL__N_113redex_backendERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEERK11ConfigFilesR11PassManagerRNS0_6vectorI8DexStoreNS4_ISF_EEEERN4Json5ValueE + 2144
15  redex-all                           0x000000010abfb8c3 main + 6275
16  libdyld.dylib                       0x00007fff738b5cc9 start + 1
Traceback (most recent call last):
  File "/tmp/redex.ABGlxd/redex.py", line 1049, in <module>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/private/tmp/redex.ABGlxd/pyredex/utils.py", line 56, in with_temp_cleanup
    fn()
  File "/tmp/redex.ABGlxd/redex.py", line 1049, in <lambda>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/tmp/redex.ABGlxd/redex.py", line 1028, in run_redex
    run_redex_binary(state, term_handler)
  File "/tmp/redex.ABGlxd/redex.py", line 494, in run_redex_binary
    if run():
  File "/tmp/redex.ABGlxd/redex.py", line 479, in run
    ).format(returncode, gdb_script_name, lldb_script_name)
RuntimeError: redex-all crashed with exit code -6! You can re-run it under gdb by running /tmp/redex.ABGlxd/redex-gdb-d1esx3kh.sh or under lldb by running /tmp/redex.ABGlxd/redex-lldb-5ldyoy95.sh

same as #441

picture DuanTzXavier

Most helpful comment

when i run ./redex-all --show-passes | grep ReBind ... and get nothing

This means your redex binary doesn't have ReBindRefsPass included in it. Which is surprising. These lines of the Makefiles should include ReBindRefs:

https://github.com/facebook/redex/blob/master/Makefile.inc#L46
https://github.com/facebook/redex/blob/master/Makefile.am#L196

Did you edit any of the makefiles before you compiled? Or make any other edits?

but when i remove all passes, it's all passed.

Without any passes, Redex doesn't do any optimizations. What's your goal here?

picture justinjhendrick on 21 Jul 2020
👍2

All 14 comments

Please post the configuration file that you're running redex with. Or are you not using a configuration file at all?

newobj picture newobj on 13 Jul 2020

Please post the configuration file that you're running redex with. Or are you not using a configuration file at all?

I didn't use a configuration file

DuanTzXavier picture DuanTzXavier on 14 Jul 2020

Please post the configuration file that you're running redex with. Or are you not using a configuration file at all?

Do you need I provide my apk file?

DuanTzXavier picture DuanTzXavier on 17 Jul 2020

Try this json config file.

{
  "redex" : {
    "passes" : [
      "ReBindRefsPass",
      "BridgePass",
      "FinalInlinePassV2",
      "DelSuperPass",
      "SingleImplPass",
      "MethodInlinePass",
      "StaticReloPassV2",
      "RemoveEmptyClassesPass",
      "InterDexPass",
      "ShortenSrcStringsPass",
      "RegAllocPass"
    ]
  }
}

I copied the list of passes that is used when no config file is passed and added InterDexPass. InterDex is supposed to handle dex ref limit issues. Try that and post here if it works or not, thanks.

justinjhendrick picture justinjhendrick on 17 Jul 2020

Try this json config file.

{
  "redex" : {
    "passes" : [
      "ReBindRefsPass",
      "BridgePass",
      "FinalInlinePassV2",
      "DelSuperPass",
      "SingleImplPass",
      "MethodInlinePass",
      "StaticReloPassV2",
      "RemoveEmptyClassesPass",
      "InterDexPass",
      "ShortenSrcStringsPass",
      "RegAllocPass"
    ]
  }
}

I copied the list of passes that is used when no config file is passed and added InterDexPass. InterDex is supposed to handle dex ref limit issues. Try that and post here if it works or not, thanks.

Use your config file to redex still get error :

libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::error_info_injector<RedexException>: libredex/DexOutput.cpp:847: void DexOutput::generate_method_data(): assertion `dodx->field_to_idx().size() <= kMaxFieldRefs' failed.
Trying to encode too many field refs in dex 22: 65542 (limit: 65536)
0   redex-all                           0x000000011045ef42 _Z23crash_backtrace_handleri + 50
1   libsystem_platform.dylib            0x00007fff69cc05fd _sigtramp + 29
2   ???                                 0x0000000000000400 0x0 + 1024
3   libsystem_c.dylib                   0x00007fff69b96808 abort + 120
4   libc++abi.dylib                     0x00007fff66df6458 abort_message + 231
5   libc++abi.dylib                     0x00007fff66de78a7 _ZL28demangling_terminate_handlerv + 238
6   libobjc.A.dylib                     0x00007fff689225b1 _ZL15_objc_terminatev + 104
7   libc++abi.dylib                     0x00007fff66df5887 _ZSt11__terminatePFvvE + 8
8   libc++abi.dylib                     0x00007fff66df81a2 __cxa_get_exception_ptr + 0
9   libc++abi.dylib                     0x00007fff66df8169 _ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception + 0
10  redex-all                           0x000000011045f539 _Z11assert_failPKcS0_jS0_10RedexErrorS0_z + 905
11  redex-all                           0x00000001104aded5 _ZN9DexOutput20generate_method_dataEv + 453
12  redex-all                           0x00000001104b7838 _ZN9DexOutput7prepareE8SortModeRKNSt3__16vectorIS0_NS1_9allocatorIS0_EEEER11ConfigFilesRKNS1_12basic_stringIcNS1_11char_traitsIcEENS3_IcEEEE + 856
13  redex-all                           0x00000001104b9db7 _Z20write_classes_to_dexRK12RedexOptionsRKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEPNS2_6vectorIP8DexClassNS6_ISD_EEEEPNS2_13unordered_mapIP9DexStringN8facebook7LocatorENS2_4hashISJ_EENS2_8equal_toISJ_EENS6_INS2_4pairIKSJ_SL_EEEEEEmmR11ConfigFilesP14PositionMapperPNSH_IP9DexMethodyNSM_IS11_EENSO_IS11_EENS6_INSQ_IKS11_yEEEEEEPNSH_IP7DexCodeNSB_I13DebugLineItemNS6_IS1B_EEEENSM_IS1A_EENSO_IS1A_EENS6_INSQ_IKS1A_S1D_EEEEEEP12IODIMetadataSA_PK12PostLoweringi + 2359
14  redex-all                           0x0000000110409bf0 _ZN12_GLOBAL__N_113redex_backendERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEER11ConfigFilesR11PassManagerRNS0_6vectorI8DexStoreNS4_ISE_EEEERN4Json5ValueE + 2144
15  redex-all                           0x00000001104029c3 main + 6275
16  libdyld.dylib                       0x00007fff69ac7cc9 start + 1
17  ???                                 0x0000000000000028 0x0 + 40
Traceback (most recent call last):
  File "/tmp/redex.sfkfC0/redex.py", line 1049, in <module>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/private/tmp/redex.sfkfC0/pyredex/utils.py", line 56, in with_temp_cleanup
    fn()
  File "/tmp/redex.sfkfC0/redex.py", line 1049, in <lambda>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/tmp/redex.sfkfC0/redex.py", line 1028, in run_redex
    run_redex_binary(state, term_handler)
  File "/tmp/redex.sfkfC0/redex.py", line 494, in run_redex_binary
    if run():
  File "/tmp/redex.sfkfC0/redex.py", line 479, in run
    ).format(returncode, gdb_script_name, lldb_script_name)
RuntimeError: redex-all crashed with exit code -6! You can re-run it under gdb by running /tmp/redex.sfkfC0/redex-gdb-row9yh1p.sh or under lldb by running /tmp/redex.sfkfC0/redex-lldb-gljmrgjl.sh
DuanTzXavier picture DuanTzXavier on 18 Jul 2020

What commit are you based on? Please make sure you have this fix in: https://github.com/facebook/redex/commit/84fbd2b94e35cffb2b2455d919ac3d9992cfd6af. Thanks!

emma0303 picture emma0303 on 18 Jul 2020

What commit are you based on? Please make sure you have this fix in: 84fbd2b. Thanks!

pull at latest commit 60760c8168d231e43d27508a5ab90d45713a6a40 and got init error :

    Redex_frontend completed in 5.8 seconds
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::error_info_injector<RedexException>: libredex/PassManager.cpp:589: void PassManager::activate_pass(const std::string &, const Json::Value &): assertion `!debug || false' failed.
No pass named ReBindRefsPass!
0   redex-all                           0x00000001048d65c2 _Z23crash_backtrace_handleri + 50
1   libsystem_platform.dylib            0x00007fff681865fd _sigtramp + 29
2   ???                                 0x0000000000000400 0x0 + 1024
3   libsystem_c.dylib                   0x00007fff6805c808 abort + 120
4   libc++abi.dylib                     0x00007fff652bc458 abort_message + 231
5   libc++abi.dylib                     0x00007fff652ad8a7 _ZL28demangling_terminate_handlerv + 238
6   libobjc.A.dylib                     0x00007fff66de85b1 _ZL15_objc_terminatev + 104
7   libc++abi.dylib                     0x00007fff652bb887 _ZSt11__terminatePFvvE + 8
8   libc++abi.dylib                     0x00007fff652be1a2 __cxa_get_exception_ptr + 0
9   libc++abi.dylib                     0x00007fff652be169 _ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception + 0
10  redex-all                           0x00000001048d6bb9 _Z11assert_failPKcS0_jS0_10RedexErrorS0_z + 905
11  redex-all                           0x00000001049dd269 _ZN11PassManager13activate_passERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEERKN4Json5ValueE + 793
12  redex-all                           0x00000001049dc57f _ZN11PassManager4initERKN4Json5ValueE + 335
13  redex-all                           0x00000001049db431 _ZN11PassManagerC2ERKNSt3__16vectorIP4PassNS0_9allocatorIS3_EEEENS0_10unique_ptrIN10keep_rules21ProguardConfigurationENS0_14default_deleteISB_EEEERKN4Json5ValueERK12RedexOptions + 737
14  redex-all                           0x0000000104883cdb main + 5947
15  libdyld.dylib                       0x00007fff67f8dcc9 start + 1
16  ???                                 0x0000000000000028 0x0 + 40
Traceback (most recent call last):
  File "/tmp/redex.1dvjhb/redex.py", line 1049, in <module>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/private/tmp/redex.1dvjhb/pyredex/utils.py", line 56, in with_temp_cleanup
    fn()
  File "/tmp/redex.1dvjhb/redex.py", line 1049, in <lambda>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/tmp/redex.1dvjhb/redex.py", line 1028, in run_redex
    run_redex_binary(state, term_handler)
  File "/tmp/redex.1dvjhb/redex.py", line 494, in run_redex_binary
    if run():
  File "/tmp/redex.1dvjhb/redex.py", line 479, in run
    ).format(returncode, gdb_script_name, lldb_script_name)
RuntimeError: redex-all crashed with exit code -6! You can re-run it under gdb by running /tmp/redex.1dvjhb/redex-gdb-k98xhc_r.sh or under lldb by running /tmp/redex.1dvjhb/redex-lldb-5hd4bnma.sh
DuanTzXavier picture DuanTzXavier on 20 Jul 2020

The error is pretty self explanatory: "No pass named ReBindRefsPass!".
Although, I am confused why it doesn't find the pass ... mind trying redex --show-passes | grep ReBind?

To unblock yourself, it should be enough to remove the pass from the config.

emma0303 picture emma0303 on 20 Jul 2020

The error is pretty self explanatory: "No pass named ReBindRefsPass!".
Although, I am confused why it doesn't find the pass ... mind trying redex --show-passes | grep ReBind?

To unblock yourself, it should be enough to remove the pass from the config.

when i run ./redex-all --show-passes | grep ReBind

$ ./redex-all --show-passes | grep ReBind

and get nothing,
but when i remove all passes, it's all passed. And don't have Trying to encode too many field refs in dex error.

it seems the bug has resolved.

Thank all you guys!!!

DuanTzXavier picture DuanTzXavier on 21 Jul 2020

when i run ./redex-all --show-passes | grep ReBind ... and get nothing

This means your redex binary doesn't have ReBindRefsPass included in it. Which is surprising. These lines of the Makefiles should include ReBindRefs:

https://github.com/facebook/redex/blob/master/Makefile.inc#L46
https://github.com/facebook/redex/blob/master/Makefile.am#L196

Did you edit any of the makefiles before you compiled? Or make any other edits?

but when i remove all passes, it's all passed.

Without any passes, Redex doesn't do any optimizations. What's your goal here?

justinjhendrick picture justinjhendrick on 21 Jul 2020
👍2

when i run ./redex-all --show-passes | grep ReBind ... and get nothing

This means your redex binary doesn't have ReBindRefsPass included in it. Which is surprising. These lines of the Makefiles should include ReBindRefs:

https://github.com/facebook/redex/blob/master/Makefile.inc#L46
https://github.com/facebook/redex/blob/master/Makefile.am#L196

Did you edit any of the makefiles before you compiled? Or make any other edits?

but when i remove all passes, it's all passed.

Without any passes, Redex doesn't do any optimizations. What's your goal here?

Sorry, I misunderstanding the passes mean.
I thought it was a black list of feature to use.

And I find the commit https://github.com/facebook/redex/commit/7e5a360d62d0e51638b5e21f1de31f3190e2e21b make all opt gone

At last , reset to before commit, still have this error 

Running IRTypeChecker...
        IRTypeChecker completed in 15.8 seconds
    Running optimization passes completed in 231.5 seconds
        Instruction lowering completed in 5.8 seconds
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::error_info_injector<RedexException>: libredex/DexOutput.cpp:849: void DexOutput::generate_method_data(): assertion `dodx->field_to_idx().size() <= kMaxFieldRefs' failed.
Trying to encode too many field refs in dex 22: 65542 (limit: 65536)
0   redex-all                           0x0000000103800192 _Z23crash_backtrace_handleri + 50
1   libsystem_platform.dylib            0x00007fff6ddbe5fd _sigtramp + 29
2   ???                                 0x0000000000000400 0x0 + 1024
3   libsystem_c.dylib                   0x00007fff6dc94808 abort + 120
4   libc++abi.dylib                     0x00007fff6aef4458 abort_message + 231
5   libc++abi.dylib                     0x00007fff6aee58a7 _ZL28demangling_terminate_handlerv + 238
6   libobjc.A.dylib                     0x00007fff6ca205b1 _ZL15_objc_terminatev + 104
7   libc++abi.dylib                     0x00007fff6aef3887 _ZSt11__terminatePFvvE + 8
8   libc++abi.dylib                     0x00007fff6aef61a2 __cxa_get_exception_ptr + 0
9   libc++abi.dylib                     0x00007fff6aef6169 _ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception + 0
10  redex-all                           0x0000000103800789 _Z11assert_failPKcS0_jS0_10RedexErrorS0_z + 905
11  redex-all                           0x000000010384f335 _ZN9DexOutput20generate_method_dataEv + 453
12  redex-all                           0x0000000103858cc8 _ZN9DexOutput7prepareE8SortModeRKNSt3__16vectorIS0_NS1_9allocatorIS0_EEEER11ConfigFilesRKNS1_12basic_stringIcNS1_11char_traitsIcEENS3_IcEEEE + 888
13  redex-all                           0x000000010385b247 _Z20write_classes_to_dexRK12RedexOptionsRKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEEPNS2_6vectorIP8DexClassNS6_ISD_EEEEPNS2_13unordered_mapIP9DexStringN8facebook7LocatorENS2_4hashISJ_EENS2_8equal_toISJ_EENS6_INS2_4pairIKSJ_SL_EEEEEEmmR11ConfigFilesP14PositionMapperPNSH_IP9DexMethodyNSM_IS11_EENSO_IS11_EENS6_INSQ_IKS11_yEEEEEEPNSH_IP7DexCodeNSB_I13DebugLineItemNS6_IS1B_EEEENSM_IS1A_EENSO_IS1A_EENS6_INSQ_IKS1A_S1D_EEEEEEP12IODIMetadataSA_PK12PostLoweringi + 2359
14  redex-all                           0x00000001037a8100 _ZN12_GLOBAL__N_113redex_backendERKNSt3__112basic_stringIcNS0_11char_traitsIcEENS0_9allocatorIcEEEER11ConfigFilesR11PassManagerRNS0_6vectorI8DexStoreNS4_ISE_EEEERN4Json5ValueE + 2144
15  redex-all                           0x00000001037a0ed3 main + 6275
16  libdyld.dylib                       0x00007fff6dbc5cc9 start + 1
17  ???                                 0x0000000000000028 0x0 + 40
Traceback (most recent call last):
  File "/tmp/redex.YSQyyx/redex.py", line 1049, in <module>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/private/tmp/redex.YSQyyx/pyredex/utils.py", line 56, in with_temp_cleanup
    fn()
  File "/tmp/redex.YSQyyx/redex.py", line 1049, in <lambda>
    with_temp_cleanup(lambda: run_redex(args), args.always_clean_up)
  File "/tmp/redex.YSQyyx/redex.py", line 1028, in run_redex
    run_redex_binary(state, term_handler)
  File "/tmp/redex.YSQyyx/redex.py", line 494, in run_redex_binary
    if run():
  File "/tmp/redex.YSQyyx/redex.py", line 479, in run
    ).format(returncode, gdb_script_name, lldb_script_name)
RuntimeError: redex-all crashed with exit code -6! You can re-run it under gdb by running /tmp/redex.YSQyyx/redex-gdb-njhbw8v6.sh or under lldb by running /tmp/redex.YSQyyx/redex-lldb-0anjcpdd.sh
DuanTzXavier picture DuanTzXavier on 22 Jul 2020

And I find the commit 7e5a360 make all opt gone

Ugh, OK, that was supposed to be just a refactor, I'll look into it.

agampe picture agampe on 22 Jul 2020

Try with commit 62586a9e or later.

agampe picture agampe on 23 Jul 2020
🎉1

Finally, I find which different with other app.

In gradle file, we set wrong config:

    dexOptions {
        javaMaxHeapSize "6g"
    }

Change 6g to 4g , successfully get output.apk file

DuanTzXavier picture DuanTzXavier on 23 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pawelkw picture pawelkw  路  6Comments
coolegos picture coolegos  路  4Comments
awelc picture awelc  路  3Comments
lukef picture lukef  路  3Comments
hljwang3874149 picture hljwang3874149  路  4Comments