Arduino: Flashing fails with Arduino IDE 1.8.9 / Linux Mint

Created on 11 Nov 2019 · 20Comments · Source: esp8266/Arduino

Basic Infos

[x] This issue complies with the issue POLICY doc.
[x] I have read the documentation at readthedocs and the issue is not addressed there.
[x] I have tested that the issue is present in current master branch (aka latest git).
[x] I have searched the issue tracker for a similar issue.
[x] If there is a stack dump, I have decoded it.
[x] I have filled out all fields below.

Platform

Hardware: [ESP-12]
Core Version: 2.6.0
Development Env: [Arduino IDE|Platformio|Make|other]
Operating System: [Windows|Ubuntu|MacOS]

Settings in IDE

Module: Wemos D1 mini
Flash Mode: serial
Flash Size: 4MB
lwip Variant: v2
Reset Method:
Flash Frequency: [40Mhz]
CPU Frequency: 80
Upload Using: [OTA|SERIAL]
Upload Speed: 57600 (serial upload only)

Problem Description

flashing fails with serial cable for core 2.6.0 toolchain, while it is working for 2.4.2.

This is core 2.4.2 flashing output

Uploading 680992 bytes from /tmp/arduino_build_270612/ESPEasy.ino.bin to flash at 0x00000000
    erasing flash
    size: 0a6420 address: 000000
    first_sector_index: 0
    total_sector_count: 167
    head_sector_count: 16
    adjusted_sector_count: 151
    erase_size: 097000
    espcomm_send_command: sending command header
    espcomm_send_command: sending command payload
    setting timeout 15000
    setting timeout 1000
    espcomm_send_command: receiving 2 bytes of data
    writing flash
................................................................................ [ 12% ]
................................................................................ [ 24% ]
................................................................................ [ 36% ]
................................................................................ [ 48% ]
................................................................................ [ 60% ]
................................................................................ [ 72% ]
................................................................................ [ 84% ]
................................................................................ [ 96% ]
..........................                                                       [ 100% ]
starting app without reboot
    espcomm_send_command: sending command header
    espcomm_send_command: sending command payload
    espcomm_send_command: receiving 2 bytes of data
closing bootloader

After successful flash do:

go to board manager, scroll downn (twice)
pick core 2.6.0, hit install,
restart IDE
power cycle Wemos D1
upload (fails), output is:

Der Sketch verwendet 714452 Bytes (68%) des Programmspeicherplatzes. Das Maximum sind 1044464 Bytes.
Globale Variablen verwenden 43224 Bytes (52%) des dynamischen Speichers, 38696 Bytes für lokale Variablen verbleiben. Das Maximum sind 81920 Bytes.
/home/john/.arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3 /home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/upload.py --chip esp8266 --port /dev/ttyUSB0 --baud 57600 flash_id --end --chip esp8266 --port /dev/ttyUSB0 --baud 57600 --before default_reset --after hard_reset write_flash 0x0 /tmp/arduino_build_906128/ESPEasy.ino.bin --end 
esptool.py v2.7
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 5c:cf:7f:39:b7:b0
Uploading stub...
Running stub...
Stub running...
Manufacturer: e0
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...
esptool.py v2.7
Serial port /dev/ttyUSB0
Connecting........_____....._____....._____....._____....._____....._____....._____
Traceback (most recent call last):
  File "/home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/upload.py", line 25, in <module>
    esptool.main(fakeargs)
  File "/home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/esptool/esptool.py", line 2841, in main
    esp.connect(args.before)
  File "/home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/esptool/esptool.py", line 483, in connect
    raise FatalError('Failed to connect to %s: %s' % (self.CHIP_NAME, last_error))
esptool.FatalError: Failed to connect to ESP8266: Timed out waiting for packet header
esptool.FatalError: Failed to connect to ESP8266: Timed out waiting for packet header

additional note: during upload attempt the internal LED flashes occasionally.
timeout appears much faster / LED flashes faster when high upload speeds are selected.

Source

s0170071

Most helpful comment

@earlephilhower Got it, sorry for not paying full attention earlier.

Suggest you might be able to make this work with current esptool in one pass:

For erase_region, is it possible to keep a file which is 16KB of 0xFFs around and append it to the file list if needed, ie write_flash ... empty_rfcal.bin {build.rfcal_addr} where empty_rfcal.bin is the 16KB of 0xFF bytes. This is a bit hacky, would accept a feature request to allow specifying erase regions in the write_flash command. However it will work right now and won't add a lot of flashing time.
For full erase, I clean forgot that a write_flash --erase-all option was added in https://github.com/espressif/esptool/pull/344 and is part of esptool v2.6 or newer. So you can use this.

This means you can hopefully remove upload.py and call esptool.py directly, one invocation per flash. Which should get rid of most of the weird edge cases with boards that don't reset twice, etc.

projectgus on 13 Nov 2019

👍3

All 20 comments

This may be reset related. After I have seen the message resetting via RTS pin I did some manual resets whenever the host tried to connect. Eventually it worked.
I have to reset it at two points during the programming cycle.
Other than that the progress output is stuck at 3% until programming has finished.

edit: the output is not stuck, for some reason it stops scrolling !!: I wonder if scroll stop can be triggerd by the output...

s0170071 on 11 Nov 2019

I have tried another unit (also Wemos D1 mini) and its working with 2.4.2 and 2.6.0. Seems to be a timing problem...

s0170071 on 12 Nov 2019

@projectgus Have you already had such report ?

d-a-v on 12 Nov 2019

@d-a-v @s0170071 Not exactly. The root cause seems to be that the chip needs to be reset into bootloader mode multiple times, once per esptool command, and this is error-prone for some boards.

Given this is all happening from upload.py, it should be possible to rewrite the code in upload.py so that it only syncs and uploads the stub one time, and then reuses the same environment for all subsequent commands. This will be both faster and less error prone.

I'll try to find some time to take a look at this myself, but I'm happy to give feedback if someone else has a go at it.

projectgus on 12 Nov 2019

If this helps I saw the same problem. No settings change except the version:

This happens with Version 2.6.0 but not Version 2.5.2
Device: Adafruit HUZZAH ESP8266
IDE: Arduino 1.8.9 Windows 10
BAUD: 115200
Just changing back to Version 2.5.2 and recompiling and doing the upload works fine, but not Version 2.6.0

ChazTuna on 12 Nov 2019

@projectgus ,
In our esp8266/Arduino core v2.5.2, we use esptool-v2.6.
In core-2.6.0 (latest one), we use esptool-v2.7.

d-a-v on 12 Nov 2019

@d-a-v I understand, but the behaviour I'm talking about is in the upload.py script and this script is part of Arduino, not part of esptool.

projectgus on 13 Nov 2019

@projectgus do you have suggested esptool options or pointers?

I'm not seeing anything obvious, unfortunately. Worst case we can special-case the upload.py not to use the identify-flash (presently used as a noop when the Arduino IDE isn't asked to fully erase the device), I guess, but if there's some "don't reset on end" that would help things we can easily add it to platform.txt...

earlephilhower on 13 Nov 2019

@earlephilhower

presently used as a noop when the Arduino IDE isn't asked to fully erase the device), I guess, but

Oh! If that's what this is for then I think the best would be the suggestion I made previously to add a --full-erase option to write_flash. This is very simple to implement inefficiently, and still pretty simple to implement efficiently (meaning: we can speed up the subsequent flashing pass a bit, by not re-erasing sectors that we know are already erased).

I'll try and implement this ASAP.

if there's some "don't reset on end" that would help things we can easily add it to

There is but it doesn't work properly with the flasher stub, because the re-Connecting doesn't have a way (yet) of detecting that it's already talking to the flasher stub, and loading the stub over the stub can get weird...

projectgus on 13 Nov 2019

Unfortunately, I don't think it's not quite that simple. We now support 3 modes:
. Erase nothing before upload (flash_id aka noop)
. Erase WiFi settings (erase_region "{build.rfcal_addr}" 0x4000)
. Erase all (erase_flash)

The 2 erase options are leftover from the esptool-ck.exe days. I suppose the reason is the "Erase WiFi settings" does not erase an existing FS in the extra flash vs. the erase_flash command.

earlephilhower on 13 Nov 2019

@earlephilhower Got it, sorry for not paying full attention earlier.

Suggest you might be able to make this work with current esptool in one pass:

For erase_region, is it possible to keep a file which is 16KB of 0xFFs around and append it to the file list if needed, ie write_flash ... empty_rfcal.bin {build.rfcal_addr} where empty_rfcal.bin is the 16KB of 0xFF bytes. This is a bit hacky, would accept a feature request to allow specifying erase regions in the write_flash command. However it will work right now and won't add a lot of flashing time.
For full erase, I clean forgot that a write_flash --erase-all option was added in https://github.com/espressif/esptool/pull/344 and is part of esptool v2.6 or newer. So you can use this.

This means you can hopefully remove upload.py and call esptool.py directly, one invocation per flash. Which should get rid of most of the weird edge cases with boards that don't reset twice, etc.

projectgus on 13 Nov 2019

👍3

@projectgus ,
In our esp8266/Arduino core v2.5.2, we use esptool-v2.6.
In core-2.6.0 (latest one), we use esptool-v2.7.

So it must be the esptool-v2.7 changes from esptool-v2.6 that is causing this?

ChazTuna on 13 Nov 2019

@ChazTuna it's the way we use it rather than the changes from 2.6 to 2.7 according to the discussion above.

d-a-v on 13 Nov 2019

👍1

@ChazTuna , @d-a-v it is a change in board txt
generic.upload.erase_cmd=version
to
generic.upload.erase_cmd=flash_id

jandrassy on 13 Nov 2019

👍1

Hi,
same problem here with lolin nodemcu v3. Chip is recognized well but nothing is happening after hard resetting. I pressed the reset manually again during "connecting..." - then: upload continued.

janzieg on 13 Nov 2019

@jandrassy I suspect that too. version was originally here for a no-op.
Ref: https://github.com/esp8266/Arduino/pull/6429#pullrequestreview-294037180

d-a-v on 13 Nov 2019

@ChazTuna @janzieg
changing in board.txt <your board>.upload.erase_cmd to
<your board>.upload.erase_cmd=version
should be a workaround

jandrassy on 13 Nov 2019

Strangely some boards do not work but here a old nodemcu v0.9 works fine :

esptool.py v2.7
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 18:fe:34:a2:64:c6
Uploading stub...
Running stub...
Stub running...
Manufacturer: ef
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...
esptool.py v2.7
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 18:fe:34:a2:64:c6
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 405616 bytes to 265524...

Writing at 0x00000000... (5 %)
Writing at 0x00004000... (11 %)
Writing at 0x00008000... (17 %)
Writing at 0x0000c000... (23 %)
Writing at 0x00010000... (29 %)
Writing at 0x00014000... (35 %)
Writing at 0x00018000... (41 %)
Writing at 0x0001c000... (47 %)
Writing at 0x00020000... (52 %)
Writing at 0x00024000... (58 %)
Writing at 0x00028000... (64 %)
Writing at 0x0002c000... (70 %)
Writing at 0x00030000... (76 %)
Writing at 0x00034000... (82 %)
Writing at 0x00038000... (88 %)
Writing at 0x0003c000... (94 %)
Writing at 0x00040000... (100 %)
Wrote 405616 bytes (265524 compressed) at 0x00000000 in 23.5 seconds (effective 138.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...