Klipper: Power loss.

[Hairyloon]

I've just suffered a power loss most of the way through an eight hour print. I'm sure we have all had similar experiences.
Sometimes you can start it back up from where it stopped, but only if you can find out exactly where it stopped... well, maybe not exactly, but you need to be fairly close.
In many instances, you can just look at the log, find the last instruction and run the code from there, but here the log was lost in the power failure (I presume overwritten when the Pi restarted).

Assuming that nothing has moved, it ought to be possible to find the precise position of the tool by counting the steps back to home: is there any way to do that?
If not, could the feature be added?
Or is there a better way to recover that I am not aware of?

klippy.log

[BlackStump]

I believe it is called a UPS.

[Hairyloon]

I believe it is called a UPS.

Oh well done.
Between the pair of you, you might make a whole wit.

[Arksine]

It wasn't intended to be sarcastic. A UPS is the only reliable way to recover from a power loss. There simply isn't enough stored power in a PSU's capacitors to detect power loss, notify the host, pause the print, and record the current position.

[jakep82]

Assuming that nothing has moved, it ought to be possible to find the precise position of the tool by counting the steps back to home: is there any way to do that?

Re-homing a printer with a partially completed print on the bed would be catastrophic most of the time.

I've never understood the desire to try and save a print after power loss. Within a few seconds the hotend will melt a hole into the part, and with many beds the part will start self releasing as the bed cools down. A UPS is the only reliable way to guarantee a print is not destroyed in a power loss event, and only if the power is restored before the battery in the UPS is exhausted.

[jakep82]

Between the pair of you, you might make a whole wit.

I should also note that insulting contributors is not a good strategy for recruiting someone to invest their valuable time implementing a feature you want. I know I'm done helping you.

[Hairyloon]

It wasn't intended to be sarcastic. A UPS is the only reliable way to recover from a power loss. There simply isn't enough stored power in a PSU's capacitors to detect power loss, notify the host, pause the print, and record the current position.

A UPS won't help recover from a power failure, it might help prevent one and it's a very costly solution to the problem as posted.
Position can easily be derived from the log, if you're logging the serial commands, and might be derivable from the klippy.log, if one knew how to read it.

[Hairyloon]

Assuming that nothing has moved, it ought to be possible to find the precise position of the tool by counting the steps back to home: is there any way to do that?

Re-homing a printer with a partially completed print on the bed would be catastrophic most of the time.

Only if you're trying to home through the print, and why would you want to do that?

>

I've never understood the desire to try and save a print after power loss. Within a few seconds the hotend will melt a hole into the part, and with many beds the part will start self releasing as the bed cools down.

And yet the question was prompted after the second time that I successfully saved a print that had stopped.
The first time was a piece of cake, since I knew exactly where it had stopped and the second time much more trouble because I didn't, and had to take a best guess.
As I said in OP, it could be made simple with a fairly easy bit of coding.

[jakep82]

As I said in OP, it could be made simple with a fairly easy bit of coding.

I look forward to seeing your pull request.

[glowtape]

Only if you're trying to home through the print, and why would you want to do that?

Because 3D printers typically don't have encoders to derive position without homing, and recovering position from a log file relies on _successfully_ flushing to disk/SD on the moment of the power loss, but _after_ loss of stepping functionality. Anyone working on filesystems will attest you that it's anything but trivial. If the log file is behind, you'll end up with a massive layer shift. If the system dies before writing out all filesystem metadata, you can end up with a corrupted system. Nor do you want to be flushing like an idiot, due to how solid state storage works.

[UPS is] a very costly solution

A 1500VA UPS from Cyberpower goes for 200 dollars on Amazon. It even has sinewave output. It should probably be able to last you up to 20 mins, if not 30 mins.

If you're intending to implement a state save function for Klipper, that can last across restarts, you'll be required to have an UPS anyway. Because a) you can't rely on the capacitance in the system to be sufficient to deliver enough energy to do everything required to save the state, and b) ideally you park the head after interrupting the print, because for reasons jakep82 elaborated.

[Hairyloon]

Only if you're trying to home through the print, and why would you want to do that?

Because 3D printers typically don't have encoders to derive position without homing, and recovering position from a log file relies on _successfully_ flushing to disk/SD on the moment of the power loss, but _after_ loss of stepping functionality.

Eh?
If your print head is at (say) x50, y60, z70 and you don't know that then if you move it to home then it will move -50x, -60y and -70z.
If klippy simply measures those moves then we would know where it was before it moved: what would you need encoders for?

[DeviousPenguin]

I agree mains UPS is the best and easiest solution, but it may be worth trying a small USB battery power pack type 'UPS' for the Raspberry Pi, it could at least keep the Pi running even if the printer gets powered off.

If the Pi detects a Printer USB disconnect then it may be able to save the GCode pointer/position, and allow for a possible restart somehow. It may be able to just pause in a similar way to a filament sensor.

Homing may be a problem however, it may be possible to keep the nozzle in the air, home X and Y only, and use Z safe homing to home on a part of the bed without a print, even a corner may be OK depending on your setup.

[Hairyloon]

Homing may be a problem however, it may be possible to keep the nozzle in the air, home X and Y only, and use Z safe homing to home on a part of the bed without a print, even a corner may be OK depending on your setup.

Mine homes by default to a corner outside of the print: I'd assumed that was normal. The only problem with that is that the hotend tends to dribble into the Y-chain. I should probably move it to the other end or something...

If the log file is behind, you'll end up with a massive layer shift. If the system dies before writing out all filesystem metadata, you can end up with a corrupted system. Nor do you want to be flushing like an idiot, due to how solid state storage works.

Why would the log file be behind?
It seems that the system sends an instruction and logs it while it is being executed. Unless the comms are very very slow, the execution would be by far the slowest part of that operation.
I can see that the system might queue up a load of instructions so it can slip of for a crafty fag break while they are executed, but I can't see any purpose in it doing that: perhaps you could explain where I have got it wrong?
While we're on the point, Octopi warns that a serial log may adversely affect performance: how likely and how bad is that effect?

[UPS is] a very costly solution

A 1500VA UPS from Cyberpower goes for 200 dollars on Amazon. It even has sinewave output. It should probably be able to last you up to 20 mins, if not 30 mins.

My printer cost £80. Double that to guard against a rare inconvenience comes in as a costly solution. Maybe the "Very" was over-egging it.

[glowtape]

Why would the log file be behind?

You assume that write operations hit the disk immediately. They typically are buffered unless requested otherwise. Something you'd like to avoid on solid state storage, because flushing a few bytes will rewrite the whole block (typical erase block sizes are 128KB and up), wearing the media out faster than necessary. If the buffered data hasn't been flushed before the power goes out, the log file will be behind.

[Hairyloon]

Why would the log file be behind?

You assume that write operations hit the disk immediately. They typically are buffered unless requested otherwise. Something you'd like to avoid on solid state storage, because flushing a few bytes will rewrite the whole block (typical erase block sizes are 128KB and up)....

Sorry, but what are we erasing here? Isn't a log just appended to the file?

[glowtape]

Solid state media is arranged in rather large blocks (way larger than the typical 4KB block size you find in common filesystems), and every time you write something, the whole block gets rewritten. If you write and flush one byte to the SD card, it'll rewrite the whole block (most likely in a different location, tho, due to wear leveling). If the erase block size is 128KB, it'll write 128KB, even tho you flushed only one byte to disk/SD.

That's why today's SSD drives have ample RAM and a low density highly wear resilient reserved area to buffer lots of data, to combine it in as few block writes as possible, to minimize wear. Unlike spinning mechanical disks, NAND cells do wear out.

--edit: I forgot the point I was wanting to make. Operating systems typically buffer writes, more so if they know it's solid state storage. Just because an application wrote something, doesn't mean it actually hit the disk.

[Hairyloon]

OK, fair enough, presumably this is true of SD cards and USB sticks as well as SSD.

But this is actually a sideline off the original point, which was why can't the printer count how far it moves back to home, in order to work out where it started from?
If you know where it was when it stopped then you should have a pretty good idea of where it was in the programme.

[mental405]

Assume you have a stepper motor with 16x microstepping and it loses power midway between two of the detents on the motor. The motor shaft is going to snap to the neareast detent. That is a loss of a half a full step or upwards of 8 microsteps steps of accuracy. On most printers that is a about .1mm.

Worse is if it is in the middle of a particularly fast move and it overshoots because of something newton something mass something motion something.

Or the Z axis that sags.

Or the quad gantry that becomes a potato chip.

Once the motors lose power they are very much out of whatever state they were in before they lost power.

[Hairyloon]

Assume you have a stepper motor with 16x microstepping and it loses power midway between two of the detents on the motor. The motor shaft is going to snap to the neareast detent. That is a loss of a half a full step or upwards of 8 microsteps steps of accuracy. On most printers that is a about .1mm.

Ah, I see. So the idea would only work if it's such a big blocky print that you wouldn't notice a whacking great 0.1mm discrepancy in the middle of it.
But still, there are probably a few of those...

Worse is if it is in the middle of a particularly fast move and it overshoots because of something newton something mass something motion something.

If it's such a fast move, then it probably isn't printing, so the exact position probably isn't so critical.

Or the Z axis that sags.

The Z axis will always be on one layer or another, so unless it sags by more than a whole layer, it should be simple enough to decide which layer its on.
So once again the idea proves useless except on those really coarse prints with over 0.1mm layer height...
Except that the Z axis tends to be a higher resolution: mine is 0.00125 and I do occasionally print with coarser layers than that if I'm in a hurry...

Or the quad gantry that becomes a potato chip.

I've no idea what you're saying there.

[mental405]

Good luck!

[Hairyloon]

Good luck!

Thank you, but I don't know where to start, and I get the feeling that you remain unconvinced...

[jakep82]

I think anyone with the ability to implement this is unconvinced that it's just a few lines of code as you seem to imply. If you want a better understanding of what would really be involved to add this feature, see this duplicate issue from a year ago. Note that it's not strictly a software solution for reasons already explained.

https://github.com/KevinOConnor/klipper/issues/1645

[Hairyloon]

I think anyone with the ability to implement this is unconvinced that it's just a few lines of code as you seem to imply.

Sorry, I don't know the code language well enough, but if I put it in English, perhaps you can explain why it wouldn't do what I think it would do?
Xcount=0
Ycount=0
Zcount=0
While (X_endstop = open){
Move X, increment
Xcount = Xcount - increment
}
While (Y_endstop = open){
Move Y, increment
Ycount = Ycount - increment
}
While (Z_endstop = open){
Move Z, increment
Zcount = Zcount - increment
}
Echo: "Original position ="; Xcount, Ycount, Zcount

OK, the first answer is that Klipper would return the error "Must Home axes first" and I'd lift z up a bit first, but I'm sure that's not too hard to do.
Clearly finding the position is only part of the problem, you still need to find the place in the code to restart from...

If you want a better understanding of what would really be involved to add this feature, see this duplicate issue from a year ago. Note that it's not strictly a software solution for reasons already explained.

Yes, they are addressing the same problem, but seeking a different solution.

[JohnEdwa]

If it could be implemented in Klipper, I would assume it pretty much would have to be the same kind of implementation that Prusa uses, which has an extra board in the power supply that detects the power outage and gives just barely enough time for the main board halt the print and properly save everything (step number one: you have to disable the heaters immediately). And that is assuming the Pi stays awake (which, granted, is easier to UPS than a complete printer). If the Pi goes down as well, there won't be a log file to write to and the brownout has a high chance of corrupting stuff anyway.

And that you are using.. the virtual SD card, I suppose? Power Resume will never work with OctoPrint unless it's implemented there too, and Foosel has explained why it most likely will not be implemented there. A lot of those points fit why implementing it with Klipper is also a bad idea.

[Hairyloon]

And still you are trying to implement a process that was never asked for here...

[JohnEdwa]

Re-read your message, my bad.
If you really are just after "report the steps it takes to go to home from this position", then that does sound like a simple addition to the Manual Stepper Command. So something like MANUAL_STEPPER STEPPER=X SET_POSITION=250 MOVE=0 STOP_ON_ENDSTOP=1 could either have STOP_ON_ENDSTOP have a third option that also prints it, or there could be REPORT_DISTANCE=1 that does it or something similar.

So the result could be "Manual Move: X, distance: 135.653454mm" or something like that?

[mental405]

If you just want to report the number of steps then you would use [get_position] (https://github.com/KevinOConnor/klipper/blob/master/docs/G-Codes.md#extended-g-code-commands) to report the position of the toolhead. If the MCU is reset all of the positions will read zero. After homing, the stepper position will be the number of steps sent to the stepper motor before it contacted the endstop as seen on the first line with the "mcu:" heading.

You can then math yourself into the X / Y coordinates by multiplying the absolute value of those step counts by your step distances for the respective axes. You might have to take the various endstop offsets into account.

If you have a delta or corexy, you are probably in for a bit more mathing.

Before homing

Send: get_position
Recv: // mcu: stepper_x:0 stepper_y:0 stepper_z:0 stepper_z1:0
Recv: // stepper: stepper_x:0.000000 stepper_y:0.000000 stepper_z:0.000000 stepper_z1:0.000000
Recv: // kinematic: X:0.000000 Y:0.000000 Z:0.000000
Recv: // toolhead: X:0.000000 Y:0.000000 Z:0.000000 E:0.000000
Recv: // gcode: X:0.000000 Y:0.000000 Z:0.040000 E:0.000000
Recv: // gcode base: X:0.000000 Y:0.000000 Z:0.000000 E:0.000000
Recv: // gcode homing: X:0.000000 Y:0.000000 Z:0.000000

After homing

Send: get_position
Recv: // mcu: stepper_x:-11263 stepper_y:-18721 stepper_z:0 stepper_z1:0
Recv: // stepper: stepper_x:-0.000000 stepper_y:0.000000 stepper_z:0.000000 stepper_z1:0.000000
Recv: // kinematic: X:-0.000000 Y:0.000000 Z:0.000000
Recv: // toolhead: X:0.000000 Y:0.000000 Z:0.000000 E:0.000000
Recv: // gcode: X:0.000000 Y:0.000000 Z:0.040000 E:0.000000
Recv: // gcode base: X:0.000000 Y:0.000000 Z:0.000000 E:0.000000
Recv: // gcode homing: X:0.000000 Y:0.000000 Z:0.000000

I guess we all assumed you had read the manual and were asking for something that didn't already exist.

[Hairyloon]

I guess we all assumed you had read the manual and were asking for something that didn't already exist.

That's a fairly rash assumption. ;)

Sorry, but there's quite a lot in the manual, much of it looks technical and over my head, and a lot of it looks to be stuff that I can't see a use for so I'd kind of skipped over a lot of it.
Also, I am of the opinion that often the best way to find something out is to ask somebody who knows, and in that policy I am entirely open to a well placed "RTFB", especially if it comes with a hint as to what page.

My sincere apologies for the original question not being more clear, and thank you all for engaging.

[Hairyloon]

I just had a print fail and this trick appears to have rescued it. Thank you folks.