Prusa-firmware: [BUG]Flashing 3.9.3 to MK3S fails
I have a MK3S (bought in September, no MMU), currently at 3.9.2.
Today I tried to flash the new 3.9.3 firmware, but it fails at the end of upload. For the avrdude log see below. When this happens, the mainboard seems to be stuck in a boot loop - I looks as if the LCD is refreshed several times in a row with thew boot message, and then there is a short pause (like a reset / reboot), then it starts again.
I also tried 3.9.3RC1, which also fails. Instead of the constant LCD refresh I just see an empty LCD.
Flashing was done via first via Octoprint (using the Firmware updater). When this failed, I used PrusaSlicer connected directly to go back to 3.9.2 (which worked), the used PrusaSlicer again to try 3.9.3 and 3.9.3RC1.
(I did the update to 3.9.2 successfully before so FW updates as such work).
So it seems I'm stuck with 3.9.2 (falling back to it worked fine both times) and cannot upgrade to 3.9.3. Any other things I can try?
avrdude output (from PrusaSlicer):
avrdude-slic3r -v -p atmega2560 -c wiring -P /dev/ttyACM0 -b 115200 -D -U flash:w:0:/home/hli/Downloads/prusa3d_fw_MK3S_MK3S+_3_9_3_3556.hex:i
avrdude-slic3r: Version 6.3-20160220-prusa3d, compiled on Dec 22 2020 at 16:48:28
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
Using Port : /dev/ttyACM0
Using Programmer : wiring
Overriding Baud Rate : 115200
AVR Part : ATmega2560
Chip Erase delay : 9000 us
PAGEL : PD7
BS2 : PA0
RESET disposition : dedicated
RETRY pulse : SCK
serial program mode : yes
parallel program mode : yes
Timeout : 200
StabDelay : 100
CmdexeDelay : 25
SyncLoops : 32
ByteDelay : 0
PollIndex : 3
PollValue : 0x53
Memory Detail :
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom 65 10 8 0 no 4096 8 0 9000 9000 0x00 0x00
flash 65 10 256 0 yes 262144 256 1024 4500 4500 0x00 0x00
lfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
hfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
efuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
lock 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
calibration 0 0 0 0 no 1 0 0 0 0 0x00 0x00
signature 0 0 0 0 no 3 0 0 0 0 0x00 0x00
Programmer Type : Wiring
Description : Wiring
Programmer Model: AVRISP
Hardware Version: 15
Firmware Version Master : 2.10
Vtarget : 0.0 V
SCK period : 0.1 us
avrdude-slic3r: AVR device initialized and ready to accept instructions
Reading | ################################################## | 100% 0.01s
avrdude-slic3r: Device signature = 0x1e9801 (probably m2560)
avrdude-slic3r: reading input file "/home/hli/Downloads/prusa3d_fw_MK3S_MK3S+_3_9_3_3556.hex"
avrdude-slic3r: writing flash (250818 bytes):
Writing | ################################################## | 100% 48.58s
avrdude-slic3r: 250818 bytes of flash written
avrdude-slic3r: verifying flash memory against /home/hli/Downloads/prusa3d_fw_MK3S_MK3S+_3_9_3_3556.hex:
avrdude-slic3r: load data flash data from input file /home/hli/Downloads/prusa3d_fw_MK3S_MK3S+_3_9_3_3556.hex:
avrdude-slic3r: input file /home/hli/Downloads/prusa3d_fw_MK3S_MK3S+_3_9_3_3556.hex contains 250818 bytes
avrdude-slic3r: reading on-chip flash data:
Reading | ################################################## | 100% 32.12s
avrdude-slic3r: verifying ...
avrdude-slic3r: 250818 bytes of flash verified
avrdude-slic3r done. Thank you.
avrdude-slic3r -v -p atmega2560 -c arduino -P /dev/ttyACM0 -b 115200 -D -u -U flash:w:1:/home/hli/Downloads/prusa3d_fw_MK3S_MK3S+_3_9_3_3556.hex:i
avrdude-slic3r: Version 6.3-20160220-prusa3d, compiled on Dec 22 2020 at 16:48:28
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
Using Port : /dev/ttyACM0
Using Programmer : arduino
Overriding Baud Rate : 115200
avrdude-slic3r: prusa_init_external_flash(): MK3 printer did not boot up on time or serial communication failed
avrdude-slic3r: arduino_open(): Failed to initialize MK3 external flash programming mode
avrdude-slic3r: Could not open port: /dev/ttyACM0
avrdude-slic3r done. Thank you.
Here also the update from the avrdude on my OctoPrint (which resulted in the boot loop)
2021-01-08 18:19:26,605
2021-01-08 18:19:26,611 /usr/bin/avrdude -v -q -p m2560 -c wiring -P /dev/ttyACM0 -D -U flash:w:/tmp/tmp5uaDyn:i
2021-01-08 18:19:27,824 >
2021-01-08 18:19:27,825 > avrdude: Version 6.3-20171130
2021-01-08 18:19:27,826 > Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
2021-01-08 18:19:27,827 > Copyright (c) 2007-2014 Joerg Wunsch
2021-01-08 18:19:27,827 >
2021-01-08 18:19:27,828 > System wide configuration file is "/etc/avrdude.conf"
2021-01-08 18:19:27,829 > User configuration file is "/home/pi/.avrduderc"
2021-01-08 18:19:27,829 > User configuration file does not exist or is not a regular file, skipping
2021-01-08 18:19:27,830 >
2021-01-08 18:19:27,831 > Using Port : /dev/ttyACM0
2021-01-08 18:19:27,831 > Using Programmer : wiring
2021-01-08 18:19:27,832 > AVR Part : ATmega2560
2021-01-08 18:19:27,833 > Chip Erase delay : 9000 us
2021-01-08 18:19:27,833 > PAGEL : PD7
2021-01-08 18:19:27,834 > BS2 : PA0
2021-01-08 18:19:27,835 > RESET disposition : dedicated
2021-01-08 18:19:27,835 > RETRY pulse : SCK
2021-01-08 18:19:27,836 > serial program mode : yes
2021-01-08 18:19:27,837 > parallel program mode : yes
2021-01-08 18:19:27,837 > Timeout : 200
2021-01-08 18:19:27,838 > StabDelay : 100
2021-01-08 18:19:27,838 > CmdexeDelay : 25
2021-01-08 18:19:27,839 > SyncLoops : 32
2021-01-08 18:19:27,840 > ByteDelay : 0
2021-01-08 18:19:27,840 > PollIndex : 3
2021-01-08 18:19:27,841 > PollValue : 0x53
2021-01-08 18:19:27,858 > Memory Detail :
2021-01-08 18:19:27,859 >
2021-01-08 18:19:27,860 > Block Poll Page Polled
2021-01-08 18:19:27,861 > Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
2021-01-08 18:19:27,861 > ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
2021-01-08 18:19:27,863 > eeprom 65 10 8 0 no 4096 8 0 9000 9000 0x00 0x00
2021-01-08 18:19:27,864 > flash 65 10 256 0 yes 262144 256 1024 4500 4500 0x00 0x00
2021-01-08 18:19:27,864 > lfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
2021-01-08 18:19:27,865 > hfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
2021-01-08 18:19:27,866 > efuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
2021-01-08 18:19:27,866 > lock 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
2021-01-08 18:19:27,867 > calibration 0 0 0 0 no 1 0 0 0 0 0x00 0x00
2021-01-08 18:19:27,867 > signature 0 0 0 0 no 3 0 0 0 0 0x00 0x00
2021-01-08 18:19:27,868 >
2021-01-08 18:19:27,869 > Programmer Type : Wiring
2021-01-08 18:19:27,869 > Description : Wiring
2021-01-08 18:19:27,870 > Programmer Model: AVRISP
2021-01-08 18:19:27,871 > Hardware Version: 15
2021-01-08 18:19:27,871 > Firmware Version Master : 2.10
2021-01-08 18:19:27,872 > Vtarget : 0.0 V
2021-01-08 18:19:27,873 > SCK period : 0.1 us
2021-01-08 18:19:27,873 >
2021-01-08 18:19:27,874 > avrdude: AVR device initialized and ready to accept instructions
2021-01-08 18:19:27,874 > avrdude: Device signature = 0x1e9801 (probably m2560)
2021-01-08 18:19:27,875 > avrdude: reading input file "/tmp/tmp5uaDyn"
2021-01-08 18:19:27,876 > avrdude: writing flash (250818 bytes):
2021-01-08 18:19:27,876 >
2021-01-08 18:20:12,022 > avrdude: 250818 bytes of flash written
2021-01-08 18:20:12,024 > avrdude: verifying flash memory against /tmp/tmp5uaDyn:
2021-01-08 18:20:12,024 > avrdude: load data flash data from input file /tmp/tmp5uaDyn:
2021-01-08 18:20:12,025 > avrdude: input file /tmp/tmp5uaDyn contains 250818 bytes
2021-01-08 18:20:12,026 > avrdude: reading on-chip flash data:
2021-01-08 18:20:12,027 >
2021-01-08 18:20:44,372 > avrdude: verifying ...
2021-01-08 18:20:44,378 > avrdude: 250818 bytes of flash verified
2021-01-08 18:20:44,379 >
2021-01-08 18:20:44,380 > avrdude done. Thank you.
2021-01-08 18:20:44,382 >
2021-01-08 18:20:44,383 >
2021-01-08 18:20:44,387 Flashing successful.
hlipka
All 83 comments
One additional thing I noticed (just trying it again): PricaSlicer shows my printer (when starting the FW update) as 'Original Prusa i3 MK3 (/dev/ttyACM1)', not as a MK3S. And the 'Support' screen on the MK3S does the same (saying '1_75mm_MK3' with 'EINSy_10a').
Further update: on trying again, flashing itself succeeded this time, but the MK3S is still in a reboot loop (shows the boot message for several seconds, then seems to update the display many times in a row, with the display flickering, and the showing the boot message again).
The avrdude log this time:
avrdude-slic3r -v -p atmega2560 -c wiring -P /dev/ttyACM1 -b 115200 -D -U flash:w:0:/home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex:i
avrdude-slic3r: Version 6.3-20160220-prusa3d, compiled on Dec 22 2020 at 16:48:28
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
Using Port : /dev/ttyACM1
Using Programmer : wiring
Overriding Baud Rate : 115200
AVR Part : ATmega2560
Chip Erase delay : 9000 us
PAGEL : PD7
BS2 : PA0
RESET disposition : dedicated
RETRY pulse : SCK
serial program mode : yes
parallel program mode : yes
Timeout : 200
StabDelay : 100
CmdexeDelay : 25
SyncLoops : 32
ByteDelay : 0
PollIndex : 3
PollValue : 0x53
Memory Detail :
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom 65 10 8 0 no 4096 8 0 9000 9000 0x00 0x00
flash 65 10 256 0 yes 262144 256 1024 4500 4500 0x00 0x00
lfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
hfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
efuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
lock 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
calibration 0 0 0 0 no 1 0 0 0 0 0x00 0x00
signature 0 0 0 0 no 3 0 0 0 0 0x00 0x00
Programmer Type : Wiring
Description : Wiring
Programmer Model: AVRISP
Hardware Version: 15
Firmware Version Master : 2.10
Vtarget : 0.0 V
SCK period : 0.1 us
avrdude-slic3r: AVR device initialized and ready to accept instructions
Reading | ################################################## | 100% 0.01s
avrdude-slic3r: Device signature = 0x1e9801 (probably m2560)
avrdude-slic3r: reading input file "/home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex"
avrdude-slic3r: writing flash (250818 bytes):
Writing | ################################################## | 100% 48.59s
avrdude-slic3r: 250818 bytes of flash written
avrdude-slic3r: verifying flash memory against /home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex:
avrdude-slic3r: load data flash data from input file /home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex:
avrdude-slic3r: input file /home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex contains 250818 bytes
avrdude-slic3r: reading on-chip flash data:
Reading | ################################################## | 100% 32.12s
avrdude-slic3r: verifying ...
avrdude-slic3r: 250818 bytes of flash verified
avrdude-slic3r done. Thank you.
avrdude-slic3r -v -p atmega2560 -c arduino -P /dev/ttyACM1 -b 115200 -D -u -U flash:w:1:/home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex:i
avrdude-slic3r: Version 6.3-20160220-prusa3d, compiled on Dec 22 2020 at 16:48:28
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
Using Port : /dev/ttyACM1
Using Programmer : arduino
Overriding Baud Rate : 115200
AVR Part : ATmega2560
Chip Erase delay : 9000 us
PAGEL : PD7
BS2 : PA0
RESET disposition : dedicated
RETRY pulse : SCK
serial program mode : yes
parallel program mode : yes
Timeout : 200
StabDelay : 100
CmdexeDelay : 25
SyncLoops : 32
ByteDelay : 0
PollIndex : 3
PollValue : 0x53
Memory Detail :
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom 65 10 8 0 no 4096 8 0 9000 9000 0x00 0x00
flash 65 10 256 0 yes 262144 256 1024 4500 4500 0x00 0x00
lfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
hfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
efuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
lock 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
calibration 0 0 0 0 no 1 0 0 0 0 0x00 0x00
signature 0 0 0 0 no 3 0 0 0 0 0x00 0x00
Programmer Type : Arduino
Description : Arduino
Hardware Version: 3
Firmware Version: 6.2
Vtarget : 0.3 V
Varef : 0.3 V
Oscillator : 28.800 kHz
SCK period : 3.3 us
avrdude-slic3r: AVR device initialized and ready to accept instructions
Reading | ################################################## | 100% 0.00s
avrdude-slic3r: Device signature = 0x1e9801 (probably m2560)
avrdude-slic3r: reading input file "/home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex"
avrdude-slic3r: writing flash (65132 bytes):
Writing | ################################################## | 100% 8.60s
avrdude-slic3r: 65132 bytes of flash written
avrdude-slic3r: verifying flash memory against /home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex:
avrdude-slic3r: load data flash data from input file /home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex:
avrdude-slic3r: input file /home/hli/Downloads/prusa3d_fw_MK3S_3_9_3_3556.hex contains 65132 bytes
avrdude-slic3r: reading on-chip flash data:
Reading | ################################################## | 100% 7.50s
avrdude-slic3r: verifying ...
avrdude-slic3r: 65132 bytes of flash verified
avrdude-slic3r done. Thank you.
Oh. I should have looked at the printer display while writing the previous comment. After 2 or 3 minutes of constantly rebooting, it finally finished (and is on FW 3.9.3). I did not try to print yet.
But after turning the printer off, and after 10 seconds on again, it shows the same behavior of hanging in a reboot loop :( It takes about 2 minutes or so to finally boot up properly. A normal reset shows similar behavior (an sometimes even longer than 2 minutes - it seems to be much worse with an SD card inserted).
hlipka
on 8 Jan 2021
Some more observations (maybe they are caused by the update problem, maybe not):
- in the main menu the selection for the sheet (between 'print frm SD' and 'autoload filament') is an empty line. When I select it even the '>' symbol disappears, although clicking on this empty line changes the sheet selection. When scrolling further done, the empty line is replaced by a duplicated line of 'autoload filament'.
- when going to the sheet setup, the some of the names for the sheets are not shown, or shown garbled (such as '@yx'). When scrolling the list up and down, the names appear and disappear again.
- similar when being in the HW setup menu, the nozzle diameter is shown wrong (e.g. as '0.0X' or '0.0I'). again, this changes when moving up and down in the menu.
So I assumed that the first update try (the one via OctoPrint) did mess with the EEPROM setting for the configuration. I did a factory reset, which took a while. Pressing on rotary knob after a reset was not recognized immediately, the Einsy board kept rebooting instead. I just held the knob, and after several minutes the menu finally appeared.
- I deleted all data.
- the boot process seems to be much faster, but it seems that there is still one moment of the display being refreshed constantly before the language selection finally appears.
- Unfortunately after the language selection (I selected English) there is a loop where the LCD is constantly changing between garbled characters and a screen filled with (IIRC) what looks like an inverted 'H' (white square with a blue 'H' in it - see the first image below, bottom right corner)
So I'm finally going back to 3.9.2, and currently re-doing the calibration (due to deleting all data). I hope that helps in finding out where my problems are coming from...
hlipka
on 9 Jan 2021
Hello!
I have about the same problem, a slightly different constellation:
At the first attempt I also got the error "MK3 printer did not boot up...".
Then I flashed again, that worked, but then the printer behaved very strange.
- strange characters on the display
- after a fresh reboot the display was blank
- ...
Then the support asked me again to flash the new version again => Done + Complete factory reset ==> Again the same errors!
Tested with prusa3d_fw_MK3S_MK3S+_3_9_3_3556.hex and also with prusa3d_fw_MK3S_3_9_3_3556.hex
chrizzo84
on 9 Jan 2021
Same here, blank display and "Failed to initialize MK3 external flash programming mode" with 3.9.3 and MK3S. Going back to 3.9.2 worked.
numo68
on 11 Jan 2021
Same here on 3.9.3 jumbled text on initial calibration and is not always happening at the same spot in the config; rolling back to 3.9.2
mathias93
on 11 Jan 2021
OK, 3 issues here, we haven't seen any of these during a rather thorough testing:
Long/repeated restarts - can you try another PC? This sounds like a misconfigured modem manager or similar process influencing the flashing and booting procedure.
Cannot flash FW, or better to say - cannot flash translations into the external FLASH. This issue has been recurring for at least a year, mostly on Windows 10. We suspect some kind of third party software is messing with the COM port. Please note - we haven't been able to reproduce these issues reasonably consistently so far, so if you can help "demystifying" the upload problems, you are welcome.
Garbled screen - that can be either caused by some errors in communication with the LCD (cable) or the FW was not flashed correctly.
Generally - the one and only change from 3.9.2 is the new XYZ algorithm, so these reports suggesting downgrading to 3.9.2 are highly suspicious - the root cause may be somewhere else.
DRracer
on 12 Jan 2021
I was able to upgrade to 3.9.3 on an MK3S+ without any issue through avrdude and octoprint, maybe it can help pinpoint the issue on specific models.
Tahx
on 12 Jan 2021
regarding the restarts: I cannot try another PC (don't have one), but as described I have seen the issue when flashing from Octoprint (using a RPi3) and via my PC (running Ubuntu 20.10). The restarts also takes very long when the PC is not connected to either of the two.
The problem with flashing the translations is not with the flash process itself, but (in my case) because the Einsy board takes too long to restart after flashing the firmware (and then avrdude just times out). Note that when flashing from octoprint, it did _not_ try to flash the translations, yet the problem happened.
Regarding the LCD problems: FW 3.9.2 work without any problems, so I would rule out a problem with the LCD or the cabling (except when talking to the LCD board was changed it now more prone to any issues in that regard). But since I could use the printer OK (apart from the other issues I found) when it finally booted up, it seems that there is some other issue.
Flashing and verifying the firmware itself was fine (see the logs above), so its not an issue with flashing, but with the FW itself. (I used two different FW files, but both are identical in their contents, so I assume that they are correct).
hlipka
on 12 Jan 2021
One additional change done in 3.9.3 is the (additional) support for the SuperPINDA probe (see https://github.com/prusa3d/Prusa-Firmware/compare/v3.9.2...v3.9.3 ). When my current print finishes, I could try updating after disabling the temperature compensation setting (which I have on currently). Or maybe you could build a FW version without SuperPINDA support and I can try that one.
hlipka
on 12 Jan 2021
Are there any debug messages written on a serial port which I could look at during one of the boot loops?
hlipka
on 12 Jan 2021
In my case I flashed with MacOs and also with Windows, also tried my MacBook (it was clean) - but nothing got better. Same like described before... strange signs and stuff...
chrizzo84
on 12 Jan 2021
- Cannot flash FW, or better to say - cannot flash translations into the external FLASH. This issue has been recurring for at least a year, mostly on Windows 10. We suspect some kind of third party software is messing with the COM port. Please note - we haven't been able to reproduce these issues reasonably consistently so far, so if you can help "demystifying" the upload problems, you are welcome.
In my case Windows 10 or Linux made absolutely no difference and I tried several times with both and even with the avrdude trying the failed command. I admit that I am running both in a VmWare virtual machine but there was no such problem until now. Going back to 3.9.2 worked the first time I tried, as did _all_ the previous updates on both MK2S some time ago and MK3S now. I actually tried 3.9.1 as that was what I had previously, then 3.9.2. Both went fine.
If the external flash needs a working firmware flashed in the first step then I am 99% sure that the problem lies there and that it is _not_ the random failure I was also able to google up in the last months or years. I have no idea what the SuperPINDA is but if it is not present in printers bought half a year ago I strongly suggest checking the firmware with a printer _not_ having it.
numo68
on 12 Jan 2021
The SuperPINDA is the new bed leveling sensor introduced with the MK3S+ (the MK3S has a PINDA V2, the older ones a V1). My MK3S has a PINDA V2 (AFAIK, it was bought in September last year).
hlipka
on 12 Jan 2021
What´s a little bit strange to me is that this initial timout failure only appeared one time... after that everything was flashing fine. But yes... the strange signs :(
Today I reflashed again with factory reset before.... but didn´t help anything, same like before.
chrizzo84
on 12 Jan 2021
All your reports are serious and I'm not saying you are not telling the truth - just thinking loud where could the problems be, especially when you say, that 3.9.2 is ok.
Especially the long boot time is interesting - does it behave so weird if:
- without the USB cable plugged into EINSY?
- without the SD card?
- Raspberry Pi Zero on the EINSY?
DRracer
on 12 Jan 2021
Yes, this happened with a 'bare' Einsy - no USB cable, no SD card, no RPi zero.
hlipka
on 12 Jan 2021
I had everything plugged in - USB and card.
Is there anything how we could help you? Should I record a video?
chrizzo84
on 12 Jan 2021
So I did a test with temperature calibration turned off. (Due to the factory reset I did on the weekend there seems to be no calibration data stored). Flashing succeeded on the first run, but afterwards the display showed strange data (the ones I described after doing the factory reset under 3.9.3 as above). After a reset its stuck in a loop. See the attached video.
https://user-images.githubusercontent.com/68884/104381792-530a3800-552d-11eb-93b8-9fb4c970a393.mp4
I then did a temperature calibration in 3.9.2 (worked fine, and enabled it). Flashed 3.9.3 again. Flashing worked, but afterwards the LCD just stayed empty. After a reset it was back in the reboot loop. Waited until it was finished, and the started temperature calibration on 3.9.3. Interestingly the messages on the LDC were mostly nonsense (and only parts were from the original messages). Started it anyway (knew how it worked from 3.9.2). It ended at the 6th point with the complaint that it could not find the calibration point ("bed calibration point not found"). After a reset, it was back into its reboot loop :(
(and its showing the same strange data for the sheet names and the nozzle size as described above). Btw: I'm using English as locale.
It did not try again to see whether it was more happy, I just went back to 3.9.2 for now.
hlipka
on 12 Jan 2021
If you know how to connect to the printer via serial, can you please send a log with the startup messages? Maybe we can pinpoint where it gets stuck and boot loops
leptun
on 14 Jan 2021
@leptun Which serial connection would that be? Do I need to connect to the header pins (where the RPi Zero would be connected), or the regular USB connection? (Is the latter available during the whole boot process, or does it need to be activated by the Einsy board during startup?)
hlipka
on 14 Jan 2021
@leptun That´s the output of pronterface connected:
Connecting...
start
Printer is now online.
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
Command not found!
On the display there is only text "Original Prusa i3" (nothing more) and the printer got stuck.
### And that´s what I get after using reset-button:
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
start
echo: 3.9.3-3556
echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
Compiled: Jan 7 2021
echo: Free Memory: 1746 PlannerBufferBytes: 1760
echo:Stored settings retrieved
adc_init
Extruder fan type:
NOCTUA
CrashDetect ENABLED!
FSensor ENABLED (sensor board revision: 0.4 or newer)
Sending 0xFF
echo:SD card ok
chrizzo84
on 14 Jan 2021
I see the same sequence. Using GTKTerm I can also provide timestamps - the time between the 'adc_init' message and 'start' differs.
[18641.20h.42m.28s.146] start
[18641.20h.42m.37s.264] echo: 3.9.3-3556
[18641.20h.42m.37s.264] echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
[18641.20h.42m.37s.268] Compiled: Jan 7 2021
[18641.20h.42m.37s.272] echo: Free Memory: 1746 PlannerBufferBytes: 1760
[18641.20h.42m.37s.276] echo:Stored settings retrieved
[18641.20h.42m.37s.280] adc_init
[18641.20h.42m.37s.862] start
[18641.20h.42m.41s.970] echo: 3.9.3-3556
[18641.20h.42m.41s.970] echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
[18641.20h.42m.41s.979] Compiled: Jan 7 2021
[18641.20h.42m.41s.983] echo: Free Memory: 1746 PlannerBufferBytes: 1760
[18641.20h.42m.41s.987] echo:Stored settings retrieved
[18641.20h.42m.41s.987] adc_init
[18641.20h.42m.42s.875] start
[18641.20h.42m.46s.984] echo: 3.9.3-3556
[18641.20h.42m.46s.988] echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
[18641.20h.42m.46s.992] Compiled: Jan 7 2021
[18641.20h.42m.46s.998] echo: Free Memory: 1746 PlannerBufferBytes: 1760
[18641.20h.42m.47s.001] echo:Stored settings retrieved
[18641.20h.42m.47s.005] adc_init
[18641.20h.42m.48s.409] start
[18641.20h.42m.52s.518] echo: 3.9.3-3556
[18641.20h.42m.52s.518] echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
[18641.20h.42m.52s.526] Compiled: Jan 7 2021
[18641.20h.42m.52s.526] echo: Free Memory: 1746 PlannerBufferBytes: 1760
[18641.20h.42m.52s.534] echo:Stored settings retrieved
[18641.20h.42m.52s.534] adc_init
[18641.20h.42m.53s.738] start
[18641.20h.42m.57s.847] echo: 3.9.3-3556
[18641.20h.42m.57s.847] echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
[18641.20h.42m.57s.855] Compiled: Jan 7 2021
[18641.20h.42m.57s.855] echo: Free Memory: 1746 PlannerBufferBytes: 1760
[18641.20h.42m.57s.860] echo:Stored settings retrieved
[18641.20h.42m.57s.863] adc_init
[18641.20h.43m.00s.694] start
[18641.20h.43m.10s.226] echo: 3.9.3-3556
[18641.20h.43m.10s.226] echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
[18641.20h.43m.10s.234] Compiled: Jan 7 2021
[18641.20h.43m.10s.234] echo: Free Memory: 1746 PlannerBufferBytes: 1760
[18641.20h.43m.10s.239] echo:Stored settings retrieved
[18641.20h.43m.10s.242] adc_init
[18641.20h.43m.11s.843] start
[18641.20h.43m.15s.952] echo: 3.9.3-3556
[18641.20h.43m.15s.952] echo: Last Updated: Jan 7 2021 12:57:03 | Author: (none, default config)
[18641.20h.43m.15s.960] Compiled: Jan 7 2021
[18641.20h.43m.15s.960] echo: Free Memory: 1746 PlannerBufferBytes: 1760
[18641.20h.43m.15s.968] echo:Stored settings retrieved
[18641.20h.43m.15s.968] adc_init
[18641.20h.43m.16s.349] Extruder fan type: NOCTUA
[18641.20h.43m.17s.353] CrashDetect ENABLED!
[18641.20h.43m.18s.451] FSensor ENABLED (sensor board revision: 0.4 or newer)
[18641.20h.43m.20s.740] echo:SD init fail
@PrusaDevs: Today I tried some stuff with a support guy of you (Dominik).
- we tried flashing with different USB cables, tried flashing with Prusa Slicer an also OctoPrint Firmware updater. Didn‘t get any better.
- Also tried the 3.9.3 RC hex, also did not get any better.
- Then I cloned the 3.9.3 Repo and ran the build.sh, flashed it, worked (kind of) => Did get the message that printer type changed from MK3S to MK3, okay might be not correct version... but the thing is that the printer really worked.... now flashed back to 3.9.2...
- If you have questions, contact your colleague from support (Dominik). I sent pictures and recorded Vid of complete flash of your builded hex.
- Also wrote him some dev infos (what I did)...
chrizzo84
on 14 Jan 2021
What I already learned from trying to understand the firmware (between the 'adc_init' and the 'fan type' outputs), when the printer type gets changed the EEPROM should be filled with all default settings. (btw: when I did the factory reset as described above, I still had the correct Z-distance for my second steel sheet stored. So I'm not sure whether this worked as it should).
In the build script I can see that the MK3 version is build by default, not the MK3S version.
I tried to build from the official sources too (the MK3S version), but it did not work because the build environment did pick up my local avr-gcc (from my Arduino 1.8.12, which is an avr-gcc 7.3.0). This raises the question whether maybe something like this happened for the official build (the build env provides a avr-gcc 4.9.2, but when another one wins over the provided one...)?
(Edit: got it working by moving my ~/.arduino15 folder away. Compiled for MK3S - and the HEX file got was slightly larger than the official one. Will test it tomorrow.)
hlipka
on 15 Jan 2021
Today I had time to setup the real build => with copying the Configuration_prusa.h stuff, setting language stuff to 0 and so on. Build was done on Linux with ./build.sh.
Hmmm, what should I say... for me it´s not better (assumed that I did everything right and everything should correspond to how you build the firmware)
chrizzo84
on 15 Jan 2021
Same here - found out that I did copy the config file to the wrong location, so I still had a MK3 version (which got me the prompt at startup). But the MK3 version seemed to work fine. When I build the MK3S version, it has the correct size, but different content than the original firmware, and it still hangs in the boot loop (and has all the problem already outlined here). Let my try adding some more debug code. maybe I can see where it really goes into the reboot.
hlipka
on 15 Jan 2021
OK, _now_ its getting interesting. I took the 3.9.3 code (git tag 3.9.3) which was not working, and just added some additional debug output to the serial port (some _SERIAL_ECHORPGM(_n("debug1"));_ in _Marlin_main.cpp_, and some _SERIAL_ECHOLN("in tp_init 1");_ in _temperature.cpp_). And lo and behold - now it works! (No boot loop, and also not the strange display issues described like the sheet names).
The only explanation I have for that is that due to the new messages some code / data is in a different place now, which does not trigger the bug I have seen. Whether this is due to a compiler bug, or something in the code, I have no idea.
Here the firmware I ended up with:
firmware_393_debug_working.zip
For now I'm fine with what I have, I will now start re-calibration (since the MK3 FW version probably has messed with the settings) and see what happens.
hlipka
on 15 Jan 2021
@hlipka Just flashed my printer with your version! Can approve, it´s working!
So that´s really strange.
chrizzo84
on 15 Jan 2021
Okay, now getting more interesting - there are only left some little problems in the menu.
In my case for example menu entry does not say "Fail stats" it´s "F´il stats" => If I´m scrolling over the item, it´s "Fail stats" but only a little part of a second.
Tested now:
XYZ Calibration => Working
Printing => Working
Checked Menu items => does not say "Fail stats" it´s "F´il stats"
So for the moment it´s looking good....
chrizzo84
on 15 Jan 2021
I have a brand new Mk3s+ received on the 30 of December 2020. It was a Kit. Has firmware 3.9.2. When I tried to flash to 3.9.3 I get the display with all the squares. Tried everything I could find including tech support. Ended up going back to 3.9.2, works flawless. no issues.
ricardocab32
on 18 Jan 2021
In my case it´s also a MK3S+ (Kit), received on 23rd of Dec. (Think I forgot to mention the model in the other posts)
chrizzo84
on 18 Jan 2021
We are looking into this issue, still we haven't been able to reproduce it except for one occasion (on a very non-standard printer). The comment from @hlipka saying "after adding some debug output, the FW works" points to some timing issues.
DRracer
on 18 Jan 2021
@DRracer is there any possibility of Remote-Debugging or sth? If yes, you can try your stuff on my printer.
chrizzo84
on 18 Jan 2021
We have done some serious retest on our testing farm and production farm as well.
So far we have been completely unsuccessful in replication of this issue, including upgrades from 3.9.2 to 3.9.3 with and without factory reset.
@chrizzo84 the printer itself doesn't support remote debugging, however if you had the possibility of connecting your machine to e.g. RPi2 or similar and enabling one of our devs to ssh into it, that may help.
We are also thinking about exchanging some of the problematic machines as they are for brand new ones (completed). The only condition must be, that the machine to be exchanged causes this issue every time - i.e. the reproducibility is high.
Anybody experiencing this issue located in CZ directly? You can write to my email on github if you do not wish to reveal your location publicly.
I can see the author of this issue is from DE, which is also "close".
DRracer
on 20 Jan 2021
@DRracer What exactly do you need on the RPI? Have a RPI here and can also setup possibility that you guys can have access to the PI over SSH. Actually there is running Octopi on it... but it´s no problem for me to setup sth. new.
If it would be interesting for you, feel free to contact me!!
We are also thinking about exchanging some of the problematic machines as they are for brand new ones (completed). The only condition must be, that the machine to be exchanged causes this issue every time - i.e. the reproducibility is high.
=> The complete machine? oO My one does produce the error every time with 3.9.3... but wouldn´t it be easier to just exchange the mainboard for testing purpose?
chrizzo84
on 20 Jan 2021
Not really. I just had a scenario where one board was having the issue, but after the contents of the flash and eeprom were moved to a different printer, all the issues were gone
leptun
on 20 Jan 2021
@chrizzo84 if you can manage the RPi to be accessible from the outside Internet, then it is a great possibility of trying stuff. Please write me an email so that we can sort the possible access into your MK3S+.
The complete machine: we still think it is something electrical, therefore shipping just the EINSY may no produce the issue again.
DRracer
on 20 Jan 2021
@leptun Okay, that really seems legit... well, I can offer myself for the exchange if you really want it => It´s a MK3S+ Kit => Bought on BlackFriday, delivered on 23rd of Dec.
chrizzo84
on 20 Jan 2021
@DRracer Please check your E-Mails! I just setup PI and SSH... also tested it, is working now....
chrizzo84
on 20 Jan 2021
@chrizzo84 thank you, you should have an answer from me.
DRracer
on 20 Jan 2021
@DRracer if you need a second machine for remote debugging, I think I can set up a tunnel to the outside world into the OctoPi I'm using for the MK3S.
Regarding the replacement: I can reproduce it every time (though I did no re-test after flashing that 'debug output' version which works fine so far). Maybe its easier to first replace Einsy+LDC board first, as these might the main culprits if its really an electrical / electronic problem? (I have a MK3S kit, bought in September, shipped mid-October. Unfortunately I don't have any of the packaging material...)
hlipka
on 20 Jan 2021
@hlipka thank you, so far we are looking at the machine of @chrizzo84 remotely. He is having the exactly the same problem.
#ifndef W25X20CL
SERIAL_PROTOCOLLNPGM("start");
#else
if ((optiboot_status != 0) || (selectedSerialPort != 0))
SERIAL_PROTOCOLLNPGM("start");
#endif
///// 4s delay /////////////////
SERIAL_ECHO_START;
printf_P(PSTR(" " FW_VERSION_FULL "\n"));
Then obviously the WD kills the boot up process.
DRracer
on 20 Jan 2021
So as a data point: I just re-flashed official 3.9.3, and I'm again in the boot loop. So having 3.9.3 running fine for a week does not cure the problem.
To rule out power supply issues I disconnected everything from the PSU (apart from the Einsy board - just left one of the 24V rails on the board and disconnected the heatbed power input). Does not change anything.
hlipka
on 20 Jan 2021
@hlipka Can you provide your modified source code, which is working?
3d-gussner
on 20 Jan 2021
To rule out power supply issues I disconnected everything from the PSU (apart from the Einsy board - just left one of the 24V rails on the board and disconnected the heatbed power input). Does not change anything.
What else do you have connected to the PSU?
3d-gussner
on 20 Jan 2021
@3d-gussner Usually I have the RPi3B+ connected (via a step-down-converter, so it should need about 0.5A), and some 24 LED stripes (which need about 1A). From my measurements the PSU has enough headroom to supply them even with the bed heating.
For this test I disconnected everything, and just left the one connector on the EInsy board which powers the board itself (including the motors and the hot end, it seems). I disconnected the one connector supplying the bed. The power-loss-detection cable was also still connected.
Regarding my changes - here the patch file I created from my changes:
diff --git a/Firmware/Marlin_main.cpp b/Firmware/Marlin_main.cpp
index f6509cdc..679f5537 100755
--- a/Firmware/Marlin_main.cpp
+++ b/Firmware/Marlin_main.cpp
@@ -1247,9 +1247,12 @@ void setup()
else { //printer version was changed so use default settings
Config_ResetDefault();
}
+ SERIAL_ECHORPGM(_n("debug1"));
SdFatUtil::set_stack_guard(); //writes magic number at the end of static variables to protect against overwriting static memory by stack
+ SERIAL_ECHORPGM(_n("debug2 "));
tp_init(); // Initialize temperature loop
+ SERIAL_ECHORPGM(_n("debug3 "));
if (w25x20cl_success) lcd_splash(); // we need to do this again, because tp_init() kills lcd
else
diff --git a/Firmware/temperature.cpp b/Firmware/temperature.cpp
index d429bb88..abc84384 100755
--- a/Firmware/temperature.cpp
+++ b/Firmware/temperature.cpp
@@ -1060,12 +1060,14 @@ static void updateTemperaturesFromRawValues()
void tp_init()
{
+ SERIAL_ECHOLN("in tp_init 1");
#if MB(RUMBA) && ((TEMP_SENSOR_0==-1)||(TEMP_SENSOR_1==-1)||(TEMP_SENSOR_2==-1)||(TEMP_SENSOR_BED==-1))
//disable RUMBA JTAG in case the thermocouple extension is plugged on top of JTAG connector
MCUCR=(1<<JTD);
MCUCR=(1<<JTD);
#endif
+ SERIAL_ECHOLN("in tp_init 2");
// Finish init of mult extruder arrays
for(int e = 0; e < EXTRUDERS; e++) {
// populate with the first value
@@ -1080,6 +1082,7 @@ void tp_init()
#endif //PIDTEMPBED
}
+ SERIAL_ECHOLN("in tp_init 3");
#if defined(HEATER_0_PIN) && (HEATER_0_PIN > -1)
SET_OUTPUT(HEATER_0_PIN);
#endif
@@ -1233,6 +1236,7 @@ void tp_init()
#endif
}
#endif //AMBIENT_MAXTEMP
+ SERIAL_ECHOLN("in tp_init 3");
}
#if (defined (TEMP_RUNAWAY_BED_HYSTERESIS) && TEMP_RUNAWAY_BED_TIMEOUT > 0) || (defined (TEMP_RUNAWAY_EXTRUDER_HYSTERESIS) && TEMP_RUNAWAY_EXTRUDER_TIMEOUT > 0)
I did not do any further experiment which output is the one that makes the difference. Tell me when this would help you (or when you would like me to add something in other places - I have the build environment working so it should not take too long to test something.
hlipka
on 20 Jan 2021
@chrizzo84 we are still playing with your machine - when we disable WDT at the beginning of setup(), your machine now boots. We'll investigate further.
DRracer
on 20 Jan 2021
@DRracer Regarding the "We are also thinking about exchanging some of the problematic machines" - do you have an ETA for that decision?
The reason I'm asking is that I wanted to use the weekend for my printers first real maintenance break (I think I'm through 6 rolls if filament as of now), and wanted to use that to upgrade the Y-axis to MK3S+ style and change the nozzle. And that does not make sense when I'm asked to send it back two days later :)
(I can easily post-pone that maintenance, I just don't want to revert my changes...)
hlipka
on 20 Jan 2021
@hlipka hard to say now, we have to investigate the ASM differences between Linux and Windows builds of the FW 3.9.3. That's our plan for the next hours.
So far we have narrowed the problem into something, which _looks_ like a compiler issue. That's the only lead we have right now, any slight change to the source code makes the FW boot correctly, as you have verified with the serial output texts. We don't want to release a FW with an additional text on the serial line, we need to find the real cause of the problem to be able to avoid it in the future.
To be exact - tag v3.9.3 built on Linux and Windows differs somehow to the extent, that the Windows build runs fine on the machine of @chrizzo84 while the Linux build (same Arduino 1.8.5 and compiler version) ends up in a loop of restarts.
If we manage to isolate and fix the problem remotely on @chrizzo84 's machine, that would be the best case - because the worst case scenario is to collect a printer, start the analysis and find out, that the problem disapeared during transport (which has happened several times in the past in case of electrical problems).
@hlipka so please keep your maintenance planned, we'll post our progress again when we find something significant.
DRracer
on 21 Jan 2021
@DRracer I wuld not so much look for a compiler error, but more for timing differences between the two compiler versions. On my board the boot eventually suceeds (sometimes on the first try, but most times it takes like 5 to 10 iterations). When the WDT is the one doing the reboot, I would guess there is task during boot which does not handle the WDT correctly, and runs at the wrong time (and takes too long). As soon as this task then runs a little bit earlier or later the WDT will not be triggered - and adding serial output takes time, so some tasks will run later.
(Thats why I proposed to add some GPIO toggles as markers, so you can see which parts of the code show this timing variations)
hlipka
on 21 Jan 2021
@hlipka I can confirm the code from Windows is at least 70% identical to the Linux (official build) version:
- some regions / functions are swapped (these were not taken into consideration - thus ~70% comparison line-to-line)
- startup functions are identical
- the only differences are jmp/rjmp and call/rcall and of course some constants referencing memory areas
- no instructions added/changed/removed in the (simply) comparable ASM outputs. Assuming, the compiler behaved similarly in the unchecked 30% of code.
Therefore I agree with the theory of a timing issue. @chrizzo84 's machine never made it from the boot loop, at least we didn't see that ever happen yesterday.
Anyone experiencing this problem has an additional EINSY board available to swap into your problematic printer (both with FW 3.9.3 of course)? That would rule out or confirm the influence of the HW.
DRracer
on 21 Jan 2021
@DRracer
Anyone experiencing this problem has an additional EINSY board available to swap into your problematic printer (both with FW 3.9.3 of course)? That would rule out or confirm the influence of the HW.
Just a proposal: Actually there is an RMA open for my printer => have a warped printbed and will get a new one. As I can see it is not shipped yet, only announced to UPS (but seems like UPS didn´t get it). Maybe you can send me a test EINSY with that shipment. Think one way or another I need to send the warped Bed back... and then can also send back the EINSY...
chrizzo84
on 21 Jan 2021
@DRracer
@chrizzo84 's machine never made it from the boot loop, at least we didn't see that ever happen yesterday.
That´s correct => Only if I switched it off and on again - then it sometimes came up...
chrizzo84
on 21 Jan 2021
@DRracer I will have a go on the logic analyzer this evening. I started a fork at https://github.com/hlipka/Prusa-Firmware/tree/2954-pin-toggle - the pins I want to use are D18, D53 and D73 (all three are at J19). I would start with adding markers at the same places where I added the serial debug out, and then work my way either to earlier places in the code or dig deeper.
hlipka
on 21 Jan 2021
@hlipka @chrizzo84 Can you please contact me at [email protected]? Use my name in the e-mail, it will get assigned to me. I would like to collect your Einsy boards and further examine them, but first, we will send you new ones.
JakoobCZ
on 21 Jan 2021
@JakoobCZ I just sent the Email!
chrizzo84
on 21 Jan 2021
@JakoobCZ Me too.
hlipka
on 21 Jan 2021
While adding these debug-pin-toggles I came to the point where building the 3.9.3 FW branch now works for me, without any changes (using the official build setup). Its size differs from the original FW, though. I have no idea whats happening here. I have attached the FW:
firmware_hli_393_3556.zip
So this ruins my plan of using the logic analyzer, since I cannot reproduce the problem with my own firmware :(
But one additional thought: even when the WDT would explain the reboot loop, it does not explain why some parts of the menu behave strange afterwards (like the 'sheets' menu).
hlipka
on 21 Jan 2021
But one additional thought: even when the WDT would explain the reboot loop, it does not explain why some parts of the menu behave strange afterwards (like the 'sheets' menu).
Well, that´s also what was really strange for me - in your (hlipka) compiled Firmware there was only one little strange thing... "Fail stats" was "F´il stats" => When scrolling over it it changed to "Fail stats"... nothing more.
Now with the Windows compiled version there are no strange signs... it´s really strange. Okay, I´m a Dev and do know really good that strange things could happen... but strange signs because of a timing? oO
Strange that I´m using strange so often :D :D
chrizzo84
on 21 Jan 2021
OK, found my mistake: I was using the 3.9.3 _branch_ - which is different from the 3.9.3 _firmware_ (the branch has an additional 'merge branch MK3' commit so its more like a 3.9.4 at the moment)
(and now I finally have a build with pin toggles in a reboot loop running...)
hlipka
on 21 Jan 2021
What I have seen so far with the LA:
- when the reboot happens, it seems to reboot multiple times around the first call to _lcd_splash()_ (the one after _spi_init()_)
- when I add one pin toggle to much around that call, boot works fine - and its actually the presence of the toggle, not how much time it takes (having pulse of 500µs length reboots, but two pulses of 50µs each make it booting up)
- it seems as if the real reboot happens around the _lcd_splash()_ after _if (w25x20cl_success)_ (though I'm not totally sure of that)
- when that fast reboot happens, it seems as if the delay function hangs (I see a 15ms pulse where it should be only 50µs long)
- I see periods where the toggle pins stays high for about 4 seconds (which would be in the __delays_us() call)
I also checked for these strange menu display behaviours - and when I ran into a boot loop, after finally starting up the menu acts weird, and is fine when the boot ran normally
hlipka
on 22 Jan 2021
@hlipka that's a very good report, thank you, we'll investigate further today.
DRracer
on 22 Jan 2021
@JakoobCZ @DRracer UPS just picked up Einsy board. Before I packed it up, I installed stock 3.9.3 on it, and checked that even with just the LCD connected and using a lab power supply the boot loop happens (which it does). I hope that helps in finding the root cause.
hlipka
on 28 Jan 2021
No news on this matter? Still have not been able to update my firmware.
ricardocab32
on 2 Feb 2021
I've been having issues flashing firmware versions all weekend, it's been recommended to me from the Prusa forums that I should bump this issue with my experience.
Last night I tried flashing the 3.9.3 firmware using the latest PrusaSlicer. When attempting the first flash, I was given an error like this:
`avrdude-slic3r -v -p atmega2560 -c wiring -P COM4 -b 115200 -D -U flash:w:0:C:\Users\dstar\Downloads\prusa3d_fw_MK3S_3_9_3_3556.hex:i
avrdude-slic3r: Version 6.3-20160220-prusa3d, compiled on Jan 11 2021 at 14:16:13
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
Using Port : COM4
Using Programmer : wiring
Overriding Baud Rate : 115200
AVR Part : ATmega2560
Chip Erase delay : 9000 us
PAGEL : PD7
BS2 : PA0
RESET disposition : dedicated
RETRY pulse : SCK
serial program mode : yes
parallel program mode : yes
Timeout : 200
StabDelay : 100
CmdexeDelay : 25
SyncLoops : 32
ByteDelay : 0
PollIndex : 3
PollValue : 0x53
Memory Detail :
Block Poll Page Polled
Memory Type Mode Delay Size Indx Paged Size Size #Pages MinW MaxW ReadBack
----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
eeprom 65 10 8 0 no 4096 8 0 9000 9000 0x00 0x00
flash 65 10 256 0 yes 262144 256 1024 4500 4500 0x00 0x00
lfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
hfuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
efuse 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
lock 0 0 0 0 no 1 0 0 9000 9000 0x00 0x00
calibration 0 0 0 0 no 1 0 0 0 0 0x00 0x00
signature 0 0 0 0 no 3 0 0 0 0 0x00 0x00
Programmer Type : Wiring
Description : Wiring
Programmer Model: AVRISP
Hardware Version: 15
Firmware Version Master : 2.10
Vtarget : 0.0 V
SCK period : 0.1 us
avrdude-slic3r: AVR device initialized and ready to accept instructions
Reading | ################################################## | 100% 0.00s
avrdude-slic3r: Device signature = 0x1e9801 (probably m2560)
avrdude-slic3r: safemode: hfuse reads as D0
avrdude-slic3r: safemode: efuse reads as FD
avrdude-slic3r: reading input file "C:\Users\dstar\Downloads\prusa3d_fw_MK3S_3_9_3_3556.hex"
avrdude-slic3r: writing flash (250818 bytes):
Writing | ################################################## | 100% 48.57s
avrdude-slic3r: 250818 bytes of flash written
avrdude-slic3r: verifying flash memory against C:\Users\dstar\Downloads\prusa3d_fw_MK3S_3_9_3_3556.hex:
avrdude-slic3r: load data flash data from input file C:\Users\dstar\Downloads\prusa3d_fw_MK3S_3_9_3_3556.hex:
avrdude-slic3r: input file C:\Users\dstar\Downloads\prusa3d_fw_MK3S_3_9_3_3556.hex contains 250818 bytes
avrdude-slic3r: reading on-chip flash data:
Reading | ################################################## | 100% 32.12s
avrdude-slic3r: verifying ...
avrdude-slic3r: 250818 bytes of flash verified
avrdude-slic3r: safemode: hfuse reads as D0
avrdude-slic3r: safemode: efuse reads as FD
avrdude-slic3r: safemode: Fuses OK (E:FD, H:D0, L:FF)
avrdude-slic3r done. Thank you.
avrdude-slic3r -v -p atmega2560 -c arduino -P COM4 -b 115200 -D -u -U flash:w:1:C:\Users\dstar\Downloads\prusa3d_fw_MK3S_3_9_3_3556.hex:i
avrdude-slic3r: Version 6.3-20160220-prusa3d, compiled on Jan 11 2021 at 14:16:13
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2014 Joerg Wunsch
Using Port : COM4
Using Programmer : arduino
Overriding Baud Rate : 115200
avrdude-slic3r: prusa_init_external_flash(): MK3 printer did not boot up on time or serial communication failed
avrdude-slic3r: arduino_open(): Failed to initialize MK3 external flash programming mode
avrdude-slic3r: Could not open port: COM4
avrdude-slic3r done. Thank you.`
My printer then started bootlooping. After a couple minutes of bootlooping, it would finally boot. I tried flashing 3.9.3 again and this time was told it was successful, however it still bootlooped for a few minutes on every power on and crashed frequently while navigating menus.
I then tried a factory reset, despite the bootlooping making that pretty difficult. This then made the bootlooping more severe. I had to leave the printer on and bootlooping for nearly two hours before it finally fully booted again. I was greeted with the language select menu because of the factory reset I performed. Rather than selecting a language, I immediately tried flashing firmware 3.9.1 (last known working for me). Flashed without error, first-time-calibrated just fine.
Now that the printer was working properly again, I tried once more to flash 3.9.3, and the flash failed with the above error again and started the bootlooping over again. I flashed 3.9.1, PrusaSlicer once again told me it failed and the printer still bootlooped, however upon fully booting, the menu reads 3.9.1 and I haven't had any crashes. It's currently printing fine on 3.9.1, but likely still has the bootlooping problem. When this print finishes, I'm going to check for bootlooping and if it's present doing another factory reset.
Suffice it to say, I don't think any of this is working as intended.
dstarr5d
on 8 Feb 2021
There are no news on this issue yet besides we got the EINSY from @hlipka . So hopefully the board still does the issue. I'll be looking into is this week.
Still, we haven't been able to reproduce the issue at Prusa HQ at all, even though we tried really hard. I'm really curious what the issue is.
DRracer
on 8 Feb 2021
On Friday the board was picked up by UPS, I think it should arrive today at Prusa HQ. I am very curious to know if you also still have the error, and if you can then reproduce the error.
chrizzo84
on 8 Feb 2021
@hlipka @chrizzo84 do your new boards work correctly even with FW 3.9.3?
DRracer
on 8 Feb 2021
@DRracer cannot say anything yet, hopefully the new one will arrive tomorrow.
chrizzo84
on 8 Feb 2021
@DRracer Since I was still doing my printer maintenance it took a while - I'm currently doing the calibration wizard. But so far its OK, no boot loops at least. I'm curious how the bed levelling / first layer looks like, since there were some reports around that.
hlipka
on 8 Feb 2021
So the new board runs fine so far - first run wizard did run through, first layer calibration was fine (even with a new Nozzle-X I had nearly the same Z-offset as before), first prints were also OK.
hlipka
on 9 Feb 2021
@DRracer got the board today. installed it, calibrated everything and could not find any errors so far. i will now make 1-2 test prints, but do not assume that there will still be something.
One question: Should I replace the 3.9.3, which is currently on it? - So just reflash? Simply to test.
chrizzo84
on 10 Feb 2021
@hlipka good news - your board is still doing the boot loop on my table, investigating now why.
@chrizzo84 great to hear that - there should be FW 3.9.3 on your board already (the same release HEX file as on our github page), but you can try flashing it again to see if it changes anything.
update: @hlipka and yes, the board start successfully after 10 consecutive reboots
DRracer
on 11 Feb 2021
Print time with new board about 12-15h => No errors or anything else so far.
@DRracer Have you been able to test it with the boards we sent you?
chrizzo84
on 13 Feb 2021
Still having this problem myself. For good measure, I did a factory reset first, and then I flashed 3.9.1, flashed without fail. Then I flashed 3.9.2, flashed without fail. Then I flashed 3.9.3, started bootlooping, gave me the failure message above, and I had to flash 3.9.2 to get the bootlooping to stop.
I'm using the latest version of PrusaSlicer and drivers provided on the website, tried it on multiple machines plugged into the printer directly via USB. There's just nothing I can do to get 3.9.3 to flash successfully.
dstarr5d
on 13 Feb 2021
We are still waiting for the board from @chrizzo84 to arrive. We only have one board which we don't want to destroy (it is the only specimen we have so far).
Anyway - if you look at PR #3033 - that's all it took to make @hlipka 's board boot correctly. There must be some timing issue but we still don't know where.
DRracer
on 17 Feb 2021
When you look in the forums there are some more users with a similar problem.
As for the reason - could it be that the EEPROM read / write speed is not deterministic? IIRC in the latest version there is additional storage for the SPINDA status, and maybe this brought some close-to-the-edge timing over the final edge. (looking at the data sheet, it seems reads are always 4 clock cycles, but writes might differ). But then it could also be miniscule differences in the actual clock speed (or the watchdog timer which has its own clock). Maybe its both :(
hlipka
on 17 Feb 2021
I'm writing something to keep this alive. Still on 3.9.2 unable to upgrade firmware on my new i3 Mk3s+
ricardocab32
on 24 Feb 2021
To anyone still experiencing this issue with FW 3.9.3 - could you please try flashing the fresh 3.10.0-RC1 and report back if the issue persists? The board from @hlipka started booting reliably and successfully after some fixes present in the 3.10 branch. Thank you.
DRracer
on 19 Mar 2021
I have flashed Firmware 3.10.0-RC1 successfully. I was able to run the wizard and calibrate the printer without any issues. No issues seen so far. Will run a couple of complicated prints to see if any issues arise. Thank you for your support.
ricardocab32
on 19 Mar 2021
I have flashed 3.10.0-RC1 successfully from Linux, thanks! Will try printing later and report in case of problems.
For some reason I got either timeouts or verification errors while verifying the first (250 k) part when flashing from Windows but that might be my setup, I am running from inside a virtual machine so it is not exactly a standard one. Interestingly it however did not make problems with the previous versions.
numo68
on 19 Mar 2021
I, too, have successfully flashed 3.10.0-RC1 using Windows. I'll do some prints this weekend to verify everything functions as it should, but the problem I used to have is gone, anyway. Thanks much!
dstarr5d
on 19 Mar 2021
Related issues
Patronics
·
4Comments
RacingHell
·
4Comments
3d-gussner
·
4Comments
anilrobo90
·
3Comments
sgoll
·
3Comments
Most helpful comment
What I have seen so far with the LA:
I also checked for these strange menu display behaviours - and when I ran into a boot loop, after finally starting up the menu acts weird, and is fine when the boot ran normally