Arduino: IRQ vectors not being set in PIO toolchain, causing bootloop w/2.5.x releases

[Jason2866]

I open the issue here since here https://github.com/earlephilhower/esp-quick-toolchain it is not possible.
It seems how -Wl,option is done is not the usual way

See GCC docs https://gcc.gnu.org/onlinedocs/gcc/Link-Options.html
-Wl,option
Pass option as an option to the linker. If option contains commas, it is split into multiple options at the commas. You can use this syntax to pass an argument to the option. For example, -Wl,-Map,output.map passes -Map output.map to the linker. When using the GNU linker, you can also get the same effect with -Wl,-Map=output.map

Issue is discussed
https://github.com/platformio/platform-espressif8266/issues/147#issuecomment-491926073

[earlephilhower]

As mentioned in the original chain @platformio, the 2.5.0 and 2.5.1 toolchain are identical. If 2.5.0 worked, then there is no changes to the esp-quick-toolchain stuff in 2.5.1. This is also a GNU linker option, which we absolutely do not modify or even look at in out own patches.

Also, I did just enable issues on the EQT repo. But, again, there was no change in g++ or ld in this release.

[Jason2866]

Yes, the bug with new build chain was and is still in 2.5.0 so as in 2.5.1. It was never corrected
The change from @mcspr does correct this for 2.5.0 and 2.5.1
It is okay if you no longer work on the new build chain and do no more bug fixes

[earlephilhower]

We use the -Wl, stuff to generate .map artifacts and pass in linker control options in the main Arduino, so it being used and works fine for us:

https://github.com/esp8266/Arduino/blob/98125f88605cd7e46e9be4e1b3ad0600dd5d2b51/platform.txt#L105

[Jason2866]

Yes. I know, and it does work fine for Arduino IDE. Platformio does it in a different way for all platforms as @ivankravets did note. He will not change

[mcspr]

As mentioned in the original issue:
https://github.com/esp8266/Arduino/blob/98125f88605cd7e46e9be4e1b3ad0600dd5d2b51/platform.txt#L54
Would doing the same thing as PIO does with this line cause the same problem, make resulting elf different in some way? (use -Wl,-T... at the end instead of -T... in the middle).

[earlephilhower]

I was never able to get PIO working properly, but the output of this line in question is an ELF file and ROM, yes?

It's unlikely that G++/LD arg parsing is busted in the 2.5.1 toolchain, but there may be some ordering requirements or something else in the parsing of the script that changes the output when completed.

Can you build and attach 2 versions of the same Blink.ino (i.e. simplest one you've got) one with 2.5.1 tools and one with whatever toolchain you've been using? I'd like to see the ELF, MAPs, etc. to say see what's different. Right now, I don't even have that...

Thx

[earlephilhower]

Oh, and as verbose a compile and link output as you can get, too, please.

[earlephilhower]

Also, are you saying that using everything the same and suimply changing the executables in xtensa-lx106-elf/... (and no other files) makes the latest Arduino core work in your app?

[earlephilhower]

Just did a quick test and both g++ -Wl,-T... and g++ -T... work and take the script, so there is no problem with -Wl or the toolchain:

earle@server:~/Arduino/hardware/esp8266com/esp8266$ cat comp /home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/xtensa-lx106-elf-gcc -fno-exceptions --verbose -Wl,--verbose -Wl,-Map -Wl,/tmp/arduino_build_799039/sketch_may12a.ino.map -g -Wall -Wextra -Os -nostdlib -Wl,--no-check-sections -u app_entry -u _printf_float -u _scanf_float -Wl,-static -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221 -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ld -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/lib -Wl,-Teagle.flash.4m1m.ld -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read -o /tmp/arduino_build_799039/sketch_may12a.ino.elf -Wl,--start-group /tmp/arduino_build_799039/sketch/sketch_may12a.ino.cpp.o /tmp/arduino_build_799039/core/core.a -lhal -lphy -lpp -lnet80211 -llwip2-536-feat -lwpa -lcrypto -lmain -lwps -lbearssl -laxtls -lespnow -lsmartconfig -lairkiss -lwpa2 -lstdc++ -lm -lc -lgcc -Wl,--end-group -L/tmp/arduino_build_799039

And replacing the -Wl,-T with -T works, too.

The -Wl-T fails to run, but not because it isn't parsing the script, but because it's not including the hardware IRQ vectors. This is a scripting problem and not toolchain one.

````
earle@server:~/Arduino/hardware/esp8266com/esp8266$ diff working failing
10,11c10,11
< COLLECT_GCC_OPTIONS='-fno-exceptions' '-v' '-g' '-Wall' '-Wextra' '-Os' '-nostdlib' '-u' 'app_entry' '-u' '_printf_float' '-u' '_scanf_float' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ld' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/lib' '-T' 'eagle.flash.4m1m.ld' '-o' '/tmp/arduino_build_799039/sketch_may12a.ino.elf' '-L/tmp/arduino_build_799039'

< /home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../libexec/gcc/xtensa-lx106-elf/4.8.2/collect2 -o /tmp/arduino_build_799039/sketch_may12a.ino.elf -u app_entry -u _printf_float -u _scanf_float -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221 -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ld -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/lib -L/tmp/arduino_build_799039 -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../lib/gcc/xtensa-lx106-elf/4.8.2 -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../lib/gcc -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/lib --verbose -Map /tmp/arduino_build_799039/sketch_may12a.ino.map --no-check-sections -static --gc-sections -wrap system_restart_local -wrap spi_flash_read --start-group /tmp/arduino_build_799039/sketch/sketch_may12a.ino.cpp.o /tmp/arduino_build_799039/core/core.a -lhal -lphy -lpp -lnet80211 -llwip2-536-feat -lwpa -lcrypto -lmain -lwps -lbearssl -laxtls -lespnow -lsmartconfig -lairkiss -lwpa2 -lstdc++ -lm -lc -lgcc --end-group -T eagle.flash.4m1m.ld

COLLECT_GCC_OPTIONS='-fno-exceptions' '-v' '-g' '-Wall' '-Wextra' '-Os' '-nostdlib' '-u' 'app_entry' '-u' '_printf_float' '-u' '_scanf_float' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ld' '-L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/lib' '-o' '/tmp/arduino_build_799039/sketch_may12a.ino.elf' '-L/tmp/arduino_build_799039'
/home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../libexec/gcc/xtensa-lx106-elf/4.8.2/collect2 -o /tmp/arduino_build_799039/sketch_may12a.ino.elf -u app_entry -u _printf_float -u _scanf_float -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221 -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/ld -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/libc/xtensa-lx106-elf/lib -L/tmp/arduino_build_799039 -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../lib/gcc/xtensa-lx106-elf/4.8.2 -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../lib/gcc -L/home/earle/Arduino/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/lib --verbose -Map /tmp/arduino_build_799039/sketch_may12a.ino.map --no-check-sections -static -Teagle.flash.4m1m.ld --gc-sections -wrap system_restart_local -wrap spi_flash_read --start-group /tmp/arduino_build_799039/sketch/sketch_may12a.ino.cpp.o /tmp/arduino_build_799039/core/core.a -lhal -lphy -lpp -lnet80211 -llwip2-536-feat -lwpa -lcrypto -lmain -lwps -lbearssl -laxtls -lespnow -lsmartconfig -lairkiss -lwpa2 -lstdc++ -lm -lc -lgcc --end-group
424,428d423
< (/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a)debug-vector.o
< (/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a)double-vector.o
< (/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a)kernel-vector.o
< (/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a)nmi-vector.o
< (/home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a)user-vector.o
````

[ivankravets]

The -Wl-T fails to run, but not because it isn't parsing the script, but because it's not including the hardware IRQ vectors. This is a scripting problem and not toolchain one.

Can we make a fix on our side?

[earlephilhower]

I'm about to put in a PR for it. Basically need to force GCC not to optimize away IRQ vectors before the linker gets to it. Give me a few mins and I'll have it in. Just tested and it's working w/minimal impact on Arduino core.

[earlephilhower]

So I did some more digging and the symbols are there, but LD's not actually putting the vectors in the file when you delay the link file with -Wl-T as opposed to the gcc -T.

First part of passing MAP. See how the linker actually stuffs in the code to call the vector:
.text 0x0000000040100000 0x90 0x0000000040100000 _stext = . 0x0000000040100000 _text_start = ABSOLUTE (.) *(.UserEnter.text) .UserEnter.text 0x0000000040100000 0xd /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a(vector.o) 0x14 (size before relaxing) 0x0000000040100004 call_user_start 0x0000000040100010 . = ALIGN (0x10) *fill* 0x000000004010000d 0x3 *(.DebugExceptionVector.text) .DebugExceptionVector.text 0x0000000040100010 0x6 /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a(debug-vector.o) 0x0000000040100010 _DebugExceptionVector 0x0000000040100020 . = ALIGN (0x10) *fill* 0x0000000040100016 0xa *(.NMIExceptionVector.text) .NMIExceptionVector.text 0x0000000040100020 0x6 /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a(nmi-vector.o) 0xd (size before relaxing) 0x0000000040100020 _NMIExceptionVector 0x0000000040100030 . = ALIGN (0x10) *fill* 0x0000000040100026 0xa *(.KernelExceptionVector.text) .KernelExceptionVector.text 0x0000000040100030 0x6 /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a(kernel-vector.o) 0x0000000040100030 _KernelExceptionVector 0x0000000040100036 0x4 LONG 0x0 0x000000004010003a 0x4 LONG 0x0 0x000000004010003e 0x4 LONG 0x0 0x0000000040100042 0x4 LONG 0x0 0x0000000040100050 . = ALIGN (0x10) *fill* 0x0000000040100046 0xa *(.UserExceptionVector.text) .UserExceptionVector.text 0x0000000040100050 0x6 /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a(user-vector.o) 0xd (size before relaxing) 0x0000000040100050 _UserExceptionVector 0x0000000040100056 0x4 LONG 0x0 0x000000004010005a 0x4 LONG 0x0 0x000000004010005e 0x4 LONG 0x0 0x0000000040100062 0x4 LONG 0x0 0x0000000040100070 . = ALIGN (0x10) *fill* 0x0000000040100066 0xa *(.DoubleExceptionVector.text) .DoubleExceptionVector.text 0x0000000040100070 0x6 /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a(double-vector.o) 0x0000000040100070 _DoubleExceptionVector 0x0000000040100076 0x4 LONG 0x0 0x000000004010007a 0x4 LONG 0x0 0x000000004010007e 0x4 LONG 0x0 0x0000000040100082 0x4 LONG 0x0 0x0000000040100090 . = ALIGN (0x10) *fill* 0x0000000040100086 0xa

Failing (-Wl-T) doesn't seem to set the vector even if it's -u _xxxxVector or whole-archive referenced:

.text 0x0000000040100000 0x40 0x0000000040100000 _stext = . 0x0000000040100000 _text_start = ABSOLUTE (.) *(.UserEnter.text) .UserEnter.text 0x0000000040100000 0xd /home/earle/Arduino/hardware/esp8266com/esp8266/tools/sdk/lib/NONOSDK221/libmain.a(vector.o) 0x14 (size before relaxing) 0x0000000040100004 call_user_start 0x0000000040100010 . = ALIGN (0x10) *fill* 0x000000004010000d 0x3 *(.DebugExceptionVector.text) 0x0000000040100010 . = ALIGN (0x10) *(.NMIExceptionVector.text) 0x0000000040100010 . = ALIGN (0x10) *(.KernelExceptionVector.text) 0x0000000040100010 0x4 LONG 0x0 0x0000000040100014 0x4 LONG 0x0 0x0000000040100018 0x4 LONG 0x0 0x000000004010001c 0x4 LONG 0x0 0x0000000040100020 . = ALIGN (0x10) *(.UserExceptionVector.text) 0x0000000040100020 0x4 LONG 0x0 0x0000000040100024 0x4 LONG 0x0 0x0000000040100028 0x4 LONG 0x0 0x000000004010002c 0x4 LONG 0x0 0x0000000040100030 . = ALIGN (0x10) *(.DoubleExceptionVector.text) 0x0000000040100030 0x4 LONG 0x0 0x0000000040100034 0x4 LONG 0x0 0x0000000040100038 0x4 LONG 0x0 0x000000004010003c 0x4 LONG 0x0 0x0000000040100040 . = ALIGN (0x10) *(.entry.text) *(.init.literal) *(.init)

[ivankravets]

So, the issue is linked with GCC LD?

[earlephilhower]

The LD link scripts more than the exes, I'd guess at this point. There's something in the way eagle.ld is asking for the vectors that is parsed differently (or in a different order of ops) when using gcc vs spawning the ld to do it.

Generated, unique section names, maybe, so the match does not fire...

[alexsomesan]

This is actually happening on Arduino IDE too.
I just uploaded a sketch that was working correctly on 2.5.0 and got into a reboot loop with the following output.

ISR not in IRAM!

Abort called

>>>stack>>>

ctx: cont
sp: 3ffffd90 end: 3fffffc0 offset: 01b0
3fffff40:  0000002b 3ffee694 3ffee888 40201552  
3fffff50:  3ffee678 3ffee730 40209978 3ffee730  
3fffff60:  3ffee678 3ffee734 3ffee888 401005b2  
3fffff70:  3ffee678 3ffee734 3ffee888 4020180c  
3fffff80:  00000000 feefeffe feefeffe feefeffe  
3fffff90:  feefeffe feefeffe feefeffe 3ffee9b4  
3fffffa0:  3fffdad0 00000000 3ffee984 40209a20  
3fffffb0:  feefeffe feefeffe 3ffe8514 4010089d  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
vac02aff5
~ld

[earlephilhower]

That's another problem, as it says. You're calling an ISR which is not in IRAM so it warns you that it will fail at some point (when the ISR is not in cache and there's a background flash operation by the SDK).

Please don't dogpile on this bug as it's a very specific PIO linking thing.

[alexsomesan]

Sorry for dogpiling. It seemed related at first glance. The problem I'm seeing started manifesting after updating to 2.5.1. Previously, in 2.5.0, the sketch was functioning correctly, including the ISR. No codechanges between updates of the SDK.
Should I open another issue?

[earlephilhower]

Sure, @alexsomesan, open another issue. We now allow IRQs during flash operations, so even if it used to work you may still have issues. The warning is in our code and you can either move your ISR to IRAM or we can see if it makes sense to only use that on Debug builds.

[alexsomesan]

OK, confirmed that adding ICACHE_RAM_ATTR to my ISR definition fixes my reported problem. No need to open any new issue.

[mcspr]

After looking again at the check inside platformio-core... If -Wl,-T and -T differ so much, isn't it possible just to use overridable board_build.ldscript for each env instead of it being part of build_flags? That way either board default or user chosen script is used, passed as -T... argument. And Arduino Core does not need to modify anything.
(to avoid issues, maybe issue a warning / error about that + document it "board options")

[earlephilhower]

The problem seems to be a difference in how PROVIDEs are handled. If there is a _PROVIDES, then the -Wl,-T doesn't think it needs to include any code (and this kind of makes sense, PROVIDE is really about listing ROM resources).

Simply adding -us for the IRQ vectors and removing the PROVIDES lines from the eagle.rom.ld file and you get a working build with either -T or -Wl,-T.

I can run that by the other maintainers and see if we can merge it in. It's a no-op (and the ROM isn't changing so mods to that file should not be a problem).

````
earle@server:~/Arduino/hardware/esp8266com/esp8266$ git diff
diff --git a/libraries/ESP8266SdFat b/libraries/ESP8266SdFat
--- a/libraries/ESP8266SdFat
+++ b/libraries/ESP8266SdFat
@@ -1 +1 @@
-Subproject commit f0962921955e2503243f28de0fdc9ac188177ffb
+Subproject commit f0962921955e2503243f28de0fdc9ac188177ffb-dirty
diff --git a/platform.txt b/platform.txt
index 2ba6dfef..588fa882 100644
--- a/platform.txt
+++ b/platform.txt
@@ -51,7 +51,7 @@ compiler.c.flags=-c {compiler.warning_flags} -Os -g -Wpointer-arith -Wno-implici
compiler.S.cmd=xtensa-lx106-elf-gcc
compiler.S.flags=-c -g -x assembler-with-cpp -MMD -mlongcalls

-compiler.c.elf.flags=-g {compiler.warning_flags} -Os -nostdlib -Wl,--no-check-sections -u app_entry {build.float} -Wl,-static "-L{compiler.sdk.path}/lib" "-L{compiler.sdk.path}/lib/{build.sdk}" "-L{compiler.sdk.path}/ld" "-L{compiler.libc.path}/lib" "-T{build.flash_ld}" -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read
+compiler.c.elf.flags=-g {compiler.warning_flags} -Os -nostdlib -Wl,--no-check-sections -u app_entry -u _DebugExceptionVector -u _DoubleExceptionVector -u _KernelExceptionVector -u _NMIExceptionVector -u _UserExceptionVector {build.float} -Wl,-static "-L{compiler.sdk.path}/lib" "-L{compiler.sdk.path}/lib/{build.sdk}" "-L{compiler.sdk.path}/ld" "-L{compiler.libc.path}/lib" "-Wl,-T{build.flash_ld}" -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read

compiler.c.elf.cmd=xtensa-lx106-elf-gcc
compiler.c.elf.libs=-lhal -lphy -lpp -lnet80211 {build.lwip_lib} -lwpa -lcrypto -lmain -lwps -lbearssl -laxtls -lespnow -lsmartconfig -lairkiss -lwpa2 {build.stdcpp_lib} -lm -lc -lgcc
diff --git a/tools/sdk/ld/eagle.rom.addr.v6.ld b/tools/sdk/ld/eagle.rom.addr.v6.ld
index 26287760..df6292c0 100644
--- a/tools/sdk/ld/eagle.rom.addr.v6.ld
+++ b/tools/sdk/ld/eagle.rom.addr.v6.ld
@@ -41,13 +41,13 @@ PROVIDE ( UartRegReadProc = 0x4000381c );
PROVIDE ( UartRegWriteProc = 0x400037ac );
PROVIDE ( UartRxString = 0x40003c30 );
PROVIDE ( Uart_Init = 0x40003a14 );
-PROVIDE ( _DebugExceptionVector = 0x40000010 );
-PROVIDE ( _DoubleExceptionVector = 0x40000070 );
-PROVIDE ( _KernelExceptionVector = 0x40000030 );
-PROVIDE ( _NMIExceptionVector = 0x40000020 );
+/PROVIDE ( _DebugExceptionVector = 0x40000010 );/
+/PROVIDE ( _DoubleExceptionVector = 0x40000070 );/
+/PROVIDE ( _KernelExceptionVector = 0x40000030 );/
+/PROVIDE ( _NMIExceptionVector = 0x40000020 );/
PROVIDE ( _ResetHandler = 0x400000a4 );
PROVIDE ( _ResetVector = 0x40000080 );
-PROVIDE ( _UserExceptionVector = 0x40000050 );
+/PROVIDE ( _UserExceptionVector = 0x40000050 );/
PROVIDE ( __adddf3 = 0x4000c538 );
PROVIDE ( __addsf3 = 0x4000c180 );
PROVIDE ( __divdf3 = 0x4000cb94 );
````

[Jason2866]

Confirmed #6095 fixes this issue

[earlephilhower]

Reopening just to track that until the PR is committed, it's still a problem

[ivankravets]

@earlephilhower You did great research! Thanks!!! Can we apply this patch https://github.com/esp8266/Arduino/pull/6095 to 2.5.1 and release as a part of 2.5.1? So, we will use the same toolchain which is used by Arduino IDE.

[earlephilhower]

@ivankravets Unfortunately, that ship has sailed. V2.5.1 came out a week ago (the impetus for this issue). We can see on the gitter channel if there's a possibility of doing a very early 2.5.2 (now that I hope we've stabilized on the new release process) in a couple weeks with this and a few other fixes that come up as people report back.

[ivankravets]

@earlephilhower we ship own package of Arduino core for ESP8266 and we can do this patch from our side even in 2.5.1. Does it make sense?

[earlephilhower]

Sure, that's all up to you. :) The core itself will get this merged, and we'll sync-up again at the next release.

[ivankravets]

Good! I think we can wait too. There is no critical requirement to use new toolchain. We will switch to it with 2.5.2. Also, I'll switch to the new toolchain in our "stage" branch and see if it works for people.