Esp-idf: Build system error in fnmatch.py (IDFGH-1557)

[raldone01]

Environment

• IDF version v4.0-dev-1287-gd7e659df2
• Build System: CMake
• Compiler version xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0
• Operating System: Windows

Problem Description

If I build try to build the app target. All files are compiled properly but the linking fails with the following exception:

[100%] Generating ld/esp32.project.ld
Traceback (most recent call last):
  File "C:/esp32deps/auto/esp-idf/tools/ldgen/ldgen.py", line 150, in <module>
    main()
  File "C:/esp32deps/auto/esp-idf/tools/ldgen/ldgen.py", line 126, in main
    mapping_rules = generation_model.generate_rules(sections_infos)
  File "C:\esp32deps\auto\esp-idf\tools\ldgen\generation.py", line 354, in generate_rules
    self._create_exclusions(mapping_rules, default_rules, sections_infos)
  File "C:\esp32deps\auto\esp-idf\tools\ldgen\generation.py", line 448, in _create_exclusions
    general_rule.add_exclusion(specific_rule, sections_info)
  File "C:\esp32deps\auto\esp-idf\tools\ldgen\generation.py", line 100, in add_exclusion
    expansions = do_section_expansion(self, section)
  File "C:\esp32deps\auto\esp-idf\tools\ldgen\generation.py", line 84, in do_section_expansion
    expansions = fnmatch.filter(sections_in_obj, section)
  File "C:\Program Files\Python36\lib\fnmatch.py", line 59, in filter
    for name in names:
TypeError: 'NoneType' object is not iterable
mingw32-make.exe[3]: *** [esp-idf\esp32\CMakeFiles\__ldgen_output_esp32.project.ld.dir\build.make:136: esp-idf/esp32/ld/esp32.project.ld] Error 1
mingw32-make.exe[2]: *** [CMakeFiles\Makefile2:4369: esp-idf/esp32/CMakeFiles/__ldgen_output_esp32.project.ld.dir/all] Error 2
mingw32-make.exe[1]: *** [CMakeFiles\Makefile2:2491: CMakeFiles/app.dir/rule] Error 2
mingw32-make.exe: *** [Makefile:1138: app] Error 2

[dobairoland]

Hi @raldone01! Thank you for reporting this issue. I would like to ask you to give more information about it because I could not reproduce it based on the available information. I can fix this failure but it would be great to track down the root cause.

Can you please give us the full output of idf.py -v build after deleting the build directory? It will be a very long output so probably saving and attaching as a file would be the best.

Are you using Windows command line or something else? Are you using the new install.bat and export.bat scripts to set up the environment?

Thank you very much!

[raldone01]

I thought I had deleted the build dir before. That did the trick...

[JakubRakus]

I've just move my project from makefile to cmake and got exactly the same problem, but deleting build directory (either removing it completely or running idf.py fullclean) didn't help. I've tried to run idf.py build in virtual environments for python 2.7 and 3 but it doesn't matter, build always stops at ldgen.py with the same error.

Running idf.py -v build doesn't show any useful additional information, so I've started to look inside python scripts and just print out variables inside do_section_expansion in generation.py (def at line 80) and... for the rule.obj named ethernetif method get_obj_sections return None and then fnmatch crashes... aha!

The cause of the problem is: disabled Ethernet component. I don't need it in my project and want to save some space and/or compilation effort, so I unset both CONFIG_ETH_ENABLED and CONFIG_ETH_USE_SPI_ETHERNET. When these two options are enabled linking and entire build using cmake runs smoothly.

The question is - why I can't disable Ethernet component when bulding app using cmake? And why makefile-based build runs regardless of these settings?

• ESP-IDF: v4.2-dev-1097-g2e14149bf-dirty
• Compiler: esp-2020r1-8.2.0
• OS: Linux

[jeremyherbert]

I had a similar issue to this, the problem was that cmake was using the system ar as opposed to xtensa-esp32-elf-ar and that broke the parser in the ld generator (as it should)

[XaviDCR92]

Having this issue on ESP-IDF v4.1-beta1-317-g84b51781c-dirty. We have checked @jeremyherbert 's solution but xtensa-esp32-elf-ar is already being used. Removing the build directory did not solve it, either.

[jeremyherbert]

Can you do a make with VERBOSE=1 and post here?

[XaviDCR92]

Note, since the build system is based on ninja, the command was idf.py -v build:

$ idf.py -v build
Checking Python dependencies...
Python requirements from /home/xavier/esp-idf/requirements.txt are satisfied.
Executing action: all (aliases: build)
Running ninja in directory /home/xavier/oeapp-common/build
Executing "ninja -v all"...
[1/8] cd /home/xavier/oeapp-common/build/esp-idf/partition_table && /usr/bin/cmake -E echo "Partition table binary generated. Contents:" && /usr/bin/cmake -E echo "*******************************************************************************" && /usr/bin/python /home/xavier/esp-idf/components/partition_table/gen_esp32part.py -q --offset 0x8000 --flash-size 4MB /home/xavier/oeapp-common/build/partition_table/partition-table.bin && /usr/bin/cmake -E echo "*******************************************************************************"
Partition table binary generated. Contents:
*******************************************************************************
# Espressif ESP32 Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,24K,
phy_init,data,phy,0xf000,4K,
hcs,app,ota_0,0x10000,1088K,
ota_buf,app,ota_1,0x120000,1088K,
ota_data,data,ota,0x230000,8K,
*******************************************************************************
[2/8] cd /home/xavier/oeapp-common/build/bootloader && /usr/bin/cmake --build .
ninja: no work to do.
[3/6] cd /home/xavier/oeapp-common/build/esp-idf/esp32 && /usr/bin/python /home/xavier/esp-idf/tools/ldgen/ldgen.py --config /home/xavier/oeapp-common/sdkconfig --fragments /home/xavier/esp-idf/components/xtensa/linker.lf   /home/xavier/esp-idf/components/esp_ringbuf/linker.lf  /home/xavier/esp-idf/components/spi_flash/linker.lf     /home/xavier/esp-idf/components/espcoredump/linker.lf   /home/xavier/esp-idf/components/esp32/linker.lf /home/xavier/esp-idf/components/esp32/ld/esp32_fragments.lf     /home/xavier/esp-idf/components/soc/linker.lf  /home/xavier/esp-idf/components/esp_event/linker.lf     /home/xavier/esp-idf/components/esp_wifi/linker.lf      /home/xavier/esp-idf/components/lwip/linker.lf  /home/xavier/esp-idf/components/log/linker.lf   /home/xavier/esp-idf/components/heap/linker.lf  /home/xavier/esp-idf/components/freertos/linker.lf     /home/xavier/esp-idf/components/newlib/newlib.lf        /home/xavier/esp-idf/components/app_trace/linker.lf     /home/xavier/esp-idf/components/esp_gdbstub/linker.lf --input /home/xavier/esp-idf/components/esp32/ld/esp32.project.ld.in --output /home/xavier/oeapp-common/build/esp-idf/esp32/ld/esp32.project.ld --kconfig /home/xavier/esp-idf/Kconfig --env-file /home/xavier/oeapp-common/build/config.env --libraries-file /home/xavier/oeapp-common/build/ldgen_libraries --objdump /home/xavier/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-objdump
FAILED: esp-idf/esp32/ld/esp32.project.ld 
cd /home/xavier/oeapp-common/build/esp-idf/esp32 && /usr/bin/python /home/xavier/esp-idf/tools/ldgen/ldgen.py --config /home/xavier/oeapp-common/sdkconfig --fragments /home/xavier/esp-idf/components/xtensa/linker.lf /home/xavier/esp-idf/components/esp_ringbuf/linker.lf /home/xavier/esp-idf/components/spi_flash/linker.lf      /home/xavier/esp-idf/components/espcoredump/linker.lf   /home/xavier/esp-idf/components/esp32/linker.lf /home/xavier/esp-idf/components/esp32/ld/esp32_fragments.lf     /home/xavier/esp-idf/components/soc/linker.lf /home/xavier/esp-idf/components/esp_event/linker.lf      /home/xavier/esp-idf/components/esp_wifi/linker.lf      /home/xavier/esp-idf/components/lwip/linker.lf  /home/xavier/esp-idf/components/log/linker.lf   /home/xavier/esp-idf/components/heap/linker.lf  /home/xavier/esp-idf/components/freertos/linker.lf     /home/xavier/esp-idf/components/newlib/newlib.lf        /home/xavier/esp-idf/components/app_trace/linker.lf     /home/xavier/esp-idf/components/esp_gdbstub/linker.lf --input /home/xavier/esp-idf/components/esp32/ld/esp32.project.ld.in --output /home/xavier/oeapp-common/build/esp-idf/esp32/ld/esp32.project.ld --kconfig /home/xavier/esp-idf/Kconfig --env-file /home/xavier/oeapp-common/build/config.env --libraries-file /home/xavier/oeapp-common/build/ldgen_libraries --objdump /home/xavier/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-objdump
Traceback (most recent call last):
  File "/home/xavier/esp-idf/tools/ldgen/ldgen.py", line 158, in <module>
    main()
  File "/home/xavier/esp-idf/tools/ldgen/ldgen.py", line 134, in main
    mapping_rules = generation_model.generate_rules(sections_infos)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 352, in generate_rules
    self._create_exclusions(mapping_rules, default_rules, sections_infos)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 451, in _create_exclusions
    general_rule.add_exclusion(specific_rule, sections_info)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 100, in add_exclusion
    expansions = do_section_expansion(self, section)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 84, in do_section_expansion
    expansions = fnmatch.filter(sections_in_obj, section)
  File "/usr/lib/python2.7/fnmatch.py", line 60, in filter
    for name in names:
TypeError: 'NoneType' object is not iterable
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

[dobairoland]

@renzbagaporo PTAL at the previous post. Haven't this been already fixed? It looks familiar to me.

[XaviDCR92]

Added a couple of debug messages inside do_section_expansion:

# Utility functions for this method
        def do_section_expansion(rule, section):
            if section in rule.get_section_names():
                print("RULE.OBJ: " + str(rule.obj))
                print("RULE.ARCHIVE: " + str(rule.archive))
                sections_in_obj = sections_infos.get_obj_sections(rule.archive, rule.obj)
                expansions = fnmatch.filter(sections_in_obj, section)
                return expansions

Which returned the following results when calling idf.py -v build:

$ idf.py -v build
Checking Python dependencies...
Python requirements from /home/xavier/esp-idf/requirements.txt are satisfied.
Executing action: all (aliases: build)
Running ninja in directory /home/xavier/oeapp-common/build
Executing "ninja -v all"...
[1/8] cd /home/xavier/oeapp-common/build/esp-idf/partition_table && /usr/bin/cmake -E echo "Partition table binary generated. Contents:" && /usr/bin/cmake -E echo "*******************************************************************************" && /usr/bin/python /home/xavier/esp-idf/components/partition_table/gen_esp32part.py -q --offset 0x8000 --flash-size 4MB /home/xavier/oeapp-common/build/partition_table/partition-table.bin && /usr/bin/cmake -E echo "*******************************************************************************"
Partition table binary generated. Contents:
*******************************************************************************
# Espressif ESP32 Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,24K,
phy_init,data,phy,0xf000,4K,
hcs,app,ota_0,0x10000,1088K,
ota_buf,app,ota_1,0x120000,1088K,
ota_data,data,ota,0x230000,8K,
*******************************************************************************
[2/8] cd /home/xavier/oeapp-common/build/bootloader && /usr/bin/cmake --build .                                                                                                                                                                                                
ninja: no work to do.                                                                                                                                                                                                                                                          
[3/6] cd /home/xavier/oeapp-common/build/esp-idf/esp32 && /usr/bin/python /home/xavier/esp-idf/tools/ldgen/ldgen.py --config /home/xavier/oeapp-common/sdkconfig --fragments /home/xavier/esp-idf/components/xtensa/linker.lf   /home/xavier/esp-idf/components/esp_ringbuf/linker.lf  /home/xavier/esp-idf/components/spi_flash/linker.lf     /home/xavier/esp-idf/components/espcoredump/linker.lf   /home/xavier/esp-idf/components/esp32/linker.lf /home/xavier/esp-idf/components/esp32/ld/esp32_fragments.lf     /home/xavier/esp-idf/components/soc/linker.lf  /home/xavier/esp-idf/components/esp_event/linker.lf     /home/xavier/esp-idf/components/esp_wifi/linker.lf      /home/xavier/esp-idf/components/lwip/linker.lf  /home/xavier/esp-idf/components/log/linker.lf   /home/xavier/esp-idf/components/heap/linker.lf  /home/xavier/esp-idf/components/freertos/linker.lf     /home/xavier/esp-idf/components/newlib/newlib.lf        /home/xavier/esp-idf/components/app_trace/linker.lf     /home/xavier/esp-idf/components/esp_gdbstub/linker.lf --input /home/xavier/esp-idf/components/esp32/ld/esp32.project.ld.in --output /home/xavier/oeapp-common/build/esp-idf/esp32/ld/esp32.project.ld --kconfig /home/xavier/esp-idf/Kconfig --env-file /home/xavier/oeapp-common/build/config.env --libraries-file /home/xavier/oeapp-common/build/ldgen_libraries --objdump /home/xavier/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-objdump                                                                                                                                                                                  
FAILED: esp-idf/esp32/ld/esp32.project.ld                                                                                                                                                                                                                                      
cd /home/xavier/oeapp-common/build/esp-idf/esp32 && /usr/bin/python /home/xavier/esp-idf/tools/ldgen/ldgen.py --config /home/xavier/oeapp-common/sdkconfig --fragments /home/xavier/esp-idf/components/xtensa/linker.lf /home/xavier/esp-idf/components/esp_ringbuf/linker.lf /home/xavier/esp-idf/components/spi_flash/linker.lf      /home/xavier/esp-idf/components/espcoredump/linker.lf   /home/xavier/esp-idf/components/esp32/linker.lf /home/xavier/esp-idf/components/esp32/ld/esp32_fragments.lf     /home/xavier/esp-idf/components/soc/linker.lf /home/xavier/esp-idf/components/esp_event/linker.lf      /home/xavier/esp-idf/components/esp_wifi/linker.lf      /home/xavier/esp-idf/components/lwip/linker.lf  /home/xavier/esp-idf/components/log/linker.lf   /home/xavier/esp-idf/components/heap/linker.lf  /home/xavier/esp-idf/components/freertos/linker.lf     /home/xavier/esp-idf/components/newlib/newlib.lf        /home/xavier/esp-idf/components/app_trace/linker.lf     /home/xavier/esp-idf/components/esp_gdbstub/linker.lf --input /home/xavier/esp-idf/components/esp32/ld/esp32.project.ld.in --output /home/xavier/oeapp-common/build/esp-idf/esp32/ld/esp32.project.ld --kconfig /home/xavier/esp-idf/Kconfig --env-file /home/xavier/oeapp-common/build/config.env --libraries-file /home/xavier/oeapp-common/build/ldgen_libraries --objdump /home/xavier/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-objdump                                                                                                                                                                                          
RULE.OBJ: log                                                                                                                                                                                                                                                                  
RULE.ARCHIVE: liblog.a                                                                                                                                                                                                                                                         
RULE.OBJ: log_freertos                                                                                                                                                                                                                                                         
RULE.ARCHIVE: liblog.a                                                                                                                                                                                                                                                         
Traceback (most recent call last):                                                                                                                                                                                                                                             
  File "/home/xavier/esp-idf/tools/ldgen/ldgen.py", line 158, in <module>                                                                                                                                                                                                      
    main()                                                                                                                                                                                                                                                                     
  File "/home/xavier/esp-idf/tools/ldgen/ldgen.py", line 134, in main                                                                                                                                                                                                          
    mapping_rules = generation_model.generate_rules(sections_infos)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 353, in generate_rules
    self._create_exclusions(mapping_rules, default_rules, sections_infos)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 452, in _create_exclusions
    general_rule.add_exclusion(specific_rule, sections_info)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 101, in add_exclusion
    expansions = do_section_expansion(self, section)
  File "/home/xavier/esp-idf/tools/ldgen/generation.py", line 85, in do_section_expansion
    expansions = fnmatch.filter(sections_in_obj, section)
  File "/usr/lib/python2.7/fnmatch.py", line 60, in filter
    for name in names:
TypeError: 'NoneType' object is not iterable
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

Apparently it fails when dealing with components freertos and log for some reason.

[XaviDCR92]

Found the reason why the script is failing: in order to have macro debugging info, our project had the following line on root CMakeLists.txt:
add_definitions("-g3")
This generates archives with different section information (new sections .group and .debug_macro appear) which generation.pydoes not seem to parse well, thus returning None from sections_infos.get_obj_sections(rule.archive, rule.obj) and making the script fail.
This issue does not depend on any user code and is reproducible under the following minimal setup I have attached. The root CMakeLists.txt contains a commented line that triggers the issue when uncommented.
3818.zip