Octoprint: Printer freezes or stutters when loading web interface, results in artefacts

[thedoble]

What were you doing?

Loading the OctoPrint web interface

What did you expect to happen?

The print to continue normally

What happened instead?

My printer stops for a few seconds, stutters ... and then keeps printing normally.

Branch & Commit or Version of OctoPrint

Version: 1.2.9 (master branch)

Printer model & used firmware incl. version

Rostock MAX V2, Repetier 0.91

Browser and Version of Browser, Operating System running Browser

Firefox 43.0.1, Windows 7 Pro x64
Problem also occurs when accessing from Safari on iPhone

Link to octoprint.log

https://dl.dropboxusercontent.com/u/908196/octoprint.log

Link to contents of terminal tab or serial.log

Excerpt from a stutter when loading the web interface:

Recv: ok 884
Send: N885 G1 X8.493 Y9.059 E152.8347_90
Recv: ok 885
Send: N886 G1 X-9.625 Y-9.059 E154.2554_95
Recv: wait
Send: N887 G1 X-10.190 Y-9.059 E154.2868_109
Recv: wait
Send: N888 M105_47
Recv: ok 886
Send: N889 G1 X7.928 Y9.059 E155.7075*94

Link to contents of Javascript console in the browser

http://pastebin.com/FmT6nzST

Screenshot(s) showing the problem:

This picture shows a print artifact (gap in print) which was caused due to a particularly long 'freeze' of the printer (around 5 seconds) - most of the time when the freeze occurs, a blob is left by the printer as the print head stops moving but plastic oozes out.

https://www.dropbox.com/s/o57mphukbpz9a98/Photo%2024-02-2016%2C%205%2043%2052%20PM.jpg?dl=0

I have read the FAQ.
I was directed here by the OctoPi team - https://github.com/guysoft/OctoPi/issues/197

Thanks

[foosel]

I see it's trying (And weirdly enough failing) to delete time lapse data there in the log. Now, this is only a hunch but: did this also happen with 1.2.8? And if not (our you don't know) could you please try deleting/moving the existing time lapse files, so it doesn't have to fetch those from disk on a reload and see if it makes a difference in behaviour?

Easiest way on OctoPi would be to log in via ssh and do this:

sudo service octoprint stop
cd ~./octoprint
mv timelapse timelapse.bck
sudo service octoprint start

That will move the existing time lapse data top a new backup folder, so the data is safe but OctoPrint doesn't care about it. Then try to reproduce the issue and report back.

You can get your timelapse data back to the right location with

sudo service octoprint stop
cd ~./octoprint
mv timelapse timelapse.new
mv timelapse.bck timelapse
mv timelapse.new/*.mpg timelapse 
rm -r timelapse.new
sudo service octoprint start

Or, if you didn't do time lapses during the test or don't need to keep them

sudo service octoprint stop
cd ~./octoprint
rm -r timelapse
mv timelapse.bck timelapse
sudo service octoprint start

[thedoble]

Thanks for your reply.

Yes this did happen with 1.2.8 as well.

I'll try moving the existing timelapse data as you say, I'm halfway through a print now so will do it tomorrow.

[thedoble]

I found that after moving the timelapse data, the printer did not stutter in my testing during the first 30 minutes of the print.

I have just tested again (8 hours into print) and the printer is back to stuttering when loading the interface :(

[ntoff]

Maybe your SD card is too slow so while it's trying to read the UI data and write the timelapse footage at the same time the card is getting bogged down? Try making an image of your SD card and duplicating it to another known to be good quality card.

[thedoble]

Hi

The SD card is a Sandisk Class 10 Ultra 16gb. The timelapse files are around 200kb and are only written every 30 seconds, I doubt this would cause the SD card to bog down?

I have loaded up iotop on the Raspi and will do some testing to confirm this.

[Loup911]

it happened to me the same thing.
During printing if I reload the web page of my print server I notice the processuce octoprint take 100% of the assigned CPU

I'am wits a raspberry pi 2B with 4 core

I think the solution might lie in separating the serial and web in 2 difent process. so they used a diferent process and could be assigned a diferent CPU.

I do not know if linux can assign priority level of the process. If so would require serial communication is in high priority over the web interface.

[markwal]

Since I don't have this difficulty with either RPi 2 nor 3 and the CPU is not heavily loaded in either case, I doubt it is CPU. It's either different add-on hardware (SD card, wi-fi dongle, etc.) or some other software installed or busy doing something that mine is not (like deleting old timelapses, searching through old gcode files updating analytics and stats).

[Loup911]

[Loup911]

printing job runing and updating the web page of octoprint server

may be some plugin!

[gddeen]

Hmmmm...

I stopped doing time lapses, and changed the port speed to 250K to the printer and keep my laptop connected and try to never watch a print.... (I waited for RPI3, but gave it away to daughter. Less stressful than watching prints.)

[markwal]

Huh. It is the CPU.

[Loup911]

I think to use a different application to manage the serial communication and the website could be an advantage.

Using one application that handles serial interface only, it would be easier to manage more than one printer with the same server.

What do you think?

[Loup911]

Currently I have three computer watching the video stream in real time and mjpg_streamer consumes 7 to 20% of the cpu, and it never Is the same cpu which is used by octoprint.

During assembly of the video file I have one of my cpu saturated at 100% and I print in the same time and have my video feed open on 1 computer. And the printing did not freezes

[gddeen]

I don't think that would fix it. (Random guess) If it's a bottleneck problem because of some locked resource, spreading it to two processes won't unlock the resource. It may complicate the discovery of a solution. I traced mine down to a timer and file io, then got rid of time-lapse. I replaced my SD card. I moved to a RPI2B (from dual core 64bit). I upgraded to a better video camera. I got rid of video camera. I shielded all cables and put ground planes beneath the boards. I added ferrite cores around wiring onto the board. I had a lot of fun... (I didn't really spend any time LOOKING for the problem...)

[ntoff]

I just tried on my pi2b and I got a spike of 27% for CPU usage though I don't have timelapse turned on. Once open though it settles down to about 1.6 - 5%

[theturtle32]

I have the same problem. I'm running OctoPrint 1.2.10 on a dedicated RasPi 2 Model B, connected with ethernet. Nothing connected except the 3d printer and the network cable. Nothing else installed and running. I have to be very careful not to load/reload the OctoPrint web interface after a print has started or it will ruin the print.

Every time the Web UI is loaded, it pegs one core at 100% for about 10-15 seconds, during which time the printer's buffer starves out. The flow of G-Code from the pi slows to a crawl, and the print head will be mostly stopped but will stutter a little bit here and there until the web UI finishes loading, at which point the print resumes full speed.

I don't have (and never did have) a webcam connected, and my timelapse directory is empty.
I do have 369MiB of data in the uploads folder, however. Does it scan through everything in that folder when serving the web UI?

[markwal]

@theturtle32 It will do a complete tree on the uploads folder in order to deliver the file list to the web UI. The operation should be on the order of the number of items in the folder not so much the size. It will also read a history yaml file containing information about the duration of the last print or whether it was successful or not.

[chaseacton]

+1

I am not using the time lapse feature, but I do have a camera attached. This is on a Pi 2.

[nallar]

I'm having the same problem on a Pi 2. Time lapse is usually enabled with per-layer, but I tried turning it off and it still happens.

Are there any diagnostics or logs I can provide to help diagnose this?

[rincewind0803]

Im getting the same effect lately since the last update to 1.2.15.
But i never experienced this behavior before.
Steps to Reproduce:
-Start print Job ( with timelapse)
-Close Browser (Firefox/Archlinux)
-Open Octoprint in new Browser
Printer completely stops, Print interface unresponsive.
https://www.dropbox.com/s/9li39b11esgcf76/octoprint.log?dl=0
It does not happen if i leave the Browser open or do not reopen a new Window. I can watch printing progress with the octodroid app that does not make the printer stop.

[thedoble]

I thought I would post here again, I had stopped using Octoprint because of this issue causing blobs in my prints. I've recently started using it again, and have updated to v1.2.15

I have disconnected my USB cameras completely, so there is no longer any additional CPU use, and there is no timelapse being done.

As mentioned by @rincewind0803 the printer will stutter or freeze completely for 3-6 seconds when loading the web interface.

Thanks

[rincewind0803]

After further investigation it may rahter be a raspbian problem.
I switched to RepetierHost Sever for a print Job today and this one also stopped printing after reopening in Browser. SSH Connektion was alive and Pi was responsive but Repetier Server was killed.
I remember updating debian for the first time in months the same day I updated to 1.2.15.
I`ll post al list of packages updated if someone is interested.

[rincewind0803]

Things are getting misterious: I downloaded a new image no plugins and upgraded the hardware to a brand new Pi 3, only made the 1.2.15 update and getting trouble again.

[luke321]

I got the same problem with 100% CPU usage on 3 differernt Pi 2. I think it started with version 1.2.7.
The printer does not get enough Gcode lines an stutters or even stops in place for a few seconds.
Currently I use the telegram plugin to check the status of the printer.

[foosel]

Since I haven't seen this behaviour myself yet but it was already pointed out that OctoPrint will scan your upload folder when loading the file list via the API (which is part of what happens during a reload): how many files are people experiencing this have in there? And the same question for the timekeeper folder.

[luke321]

Actually my problem was caused by the history plugin. Probably because of the long list of prints loading each time.
I never had more than 4 files in the upload folder, no camera and no timelapse videos.
I will reinstall the history plugin and double check if the stuttering comes back and than post an issue in the plugin repo.

[nallar]

Probably encountering the same issue here. I keep the uploaded files clean (currently 2 .gcodes there), but have hundreds of past jobs recorded by the history plugin.

edit: Yep, the request which takes longest on a page load when it freezes is to /history...

[foosel]

Yikes. That definitely looks like a likely culprit then. The login request probably takes so long because the server is blocked by the history request from just before it, that's just a wild guess right now.

Some proper caching headers on the history plugin's resource might already help tremendously here.

Meanwhile I'm also looking into some better cache handling on the API's of OctoPrint itself, to reduce overhead here.

[nallar]

Is it possible to have a separate process for communicating with the printer? Doing it in its own thread should be fine, but python still has the damn global interpreter lock so that doesn't actually work.

I understand it's likely to involve a lot of work, but it's probably the only way to avoid occasional reports from users of serial pauses in the long run.

In terms of just fixing the history plugin: It should set caching headers, as you've pointed out. It could also time.sleep(0) in the large loops it's doing, which would yield and allow serial communication to happen in the middle of it. maybe?

[foosel]

@nallar I've spent a lot of time thinking about splitting it into multiple processes, but that will definitely be something that not only is a lot of work but also means a complete loss of backwards compatibility for some existing plugin extension points. The idea would be to move the whole serial communication into its own process (with high process priority) and have some fancy inter process communication between that and the server component. Problem is that plugins that depend on ANY modification or just tracking of what goes over the line or even replace the serial transport completely would stop functioning if they depend on shared memory - there's no way that I've been able to think of so far to have a plugin exist in both processes and transparently share state without side effects regarding compatibility.

And that's why it's currently pushed far far down the TODO list, since while it indeed is something I need to find a solution for in the long term, it's such a breaking change that it's out of the question to look closer at this currently with everything else that's already in the pipeline.

But yes, I curse the GIL daily ;)

[sfsdfd]

Contributing some additional data about resource caching.

I reloaded OctoPrint three times, at about 30-second intervals (since each load took about 20 seconds), and kept an eye on Chrome's network tab. OctoPrint was printing at the time, so some things obviously changed - but not a ton in the span of 70 seconds.

Results:

http://imgur.com/a/vJzm2

Objects that are cached during successive loads:

• stylesheets, images, and a font
• some packed libraries
• less.min.js and Worker.js

Objects that aren't cached during successive loads:

• info, files, printerprofiles, slicing, timelapse, connection, custom, settings, history, profiles, login, automaticshutdown, logs, check, pluginmanager, channels
• the currently printing G-Code

That's kind of a lot of data to exchange, isn't it? Especially over a short period, where the only things that are _really_ changing are the printer status, Terminal tab, and camera feed.

The files that caused the most delay, and some thoughts about each:

gcode (13.5 seconds)
At first I thought that this file might contain the last (x) G-Code commands, which would explain why a big chunk of data gets exchanged on every refresh. Nope, it's the same file every single time - the static G-Code file that I uploaded. This is easily the biggest source of delay in the OctoPrint page load.

automaticshutdown (3.96 seconds)
login (3.82 seconds)
timelapse?unrendered=true (1.30 seconds)
These are just static scripts, right? No identifiable reason why they're not caching.

settings (1.31 seconds)
This contains all of the settings. Again, they're not changing at all from one instance to the next, so there's no reason it shouldn't be cached.

history (3.19 seconds)
history (2.96 seconds)
This history gets updated whenever a new print gets queued, so it's not changing while the same print is in progress. No reason why it's not cached. But worse than that - the exact same resource is being fully loaded twice, and imposing a significant hit each time.

That's the bulk of it. Everything else loads in less than 1 second, and much of it concurrently. Many of those could be resolved through caching as well - and some of them would probably be easy, if techniques developed to cache the big objects can be equivalently applied to smaller, static ones as well.

Resolving these cache misses could likely reduce the reload time of OctoPrint from 18+ seconds to 3-4 seconds, and maybe even less. Not only would it make OctoPrint _much_ snappier - it would greatly reduce the load on the RPi during page refreshes, maybe reducing printer artifacts introduced by the load spike.

I don't think that I'm having trouble with my printer stalling out due to buffer starvation, but I can definitely report that page loads greatly disrupt my camera feed. Probably 60% of the time I reload the page, the camera feed on the Control tab stops - and doesn't come back. Meanwhile, I can access /webcam/?action=stream and see the camera stream flowing like butter... so the RPi is handily serving the stream - but the big delay between OctoPrint loading all of these resources (much of it redundantly) is interrupting the streaming of video into Control. Certainly not a fatal problem, but maybe a reason to raise its priority?

[foosel]

I've already raised its priority and spent several days a couple weeks back adding caching to most of the resources you mentioned above where it made sense and was feasible, that's what I meant with

Meanwhile I'm also looking into some better cache handling on the API's of OctoPrint itself, to reduce overhead here.

a couple of comments upwards ;)

That stuff will be released with 1.3.0, not the current 1.2.x maintenance releases however, since just adding caching headers is only half the story, the tricky part is getting the cache invalidation to work properly, and if stuff goes wrong there, things WILL break - too risky for stability to put into a regular maintenance release.

Note that it only makes sense to cache GET requests though, everything else IS expected to change. And for some resources (e.g. the file list) reliably figuring out IF something changed or the cache is still valid is already something that depending on the underlying file system and available system resources can take some time. In case of e.g. the list of available printer ports it can become actually outright impossible to figure this out without doing the same work I'd need to do in order to fulfill the uncached request - in which case caching kinda stops making sense.

As a side note, I also learnt that there's either a jquery or a browser bug when it comes to handling consecutive requests to the same resource if ETag headers come into play - cost me a day.

Hope that makes it clear why - while it appears to be a low hanging fruit - it is not being tackled on the maintenance branch but instead rather pushed into the next big release. A better caching policy for the API endpoints in OctoPrint's core (I don't have control about plugins) IS being worked on though, so please, patience :)

[nallar]

I think @imrahil may have stopped working on the print history plugin. No development on it since January, no updating of issues. It's a shame since lots of people are using it and it'll be causing print freezing.

[imrahil]

Hi folks! Hi @nallar!
"The reports of my death are greatly exaggerated" :)
I'll try to sort this out in next few days!

[fiveangle]

I see this is closed, but still very much an issue with 1.3.0; exacerbated by 2.5x jump in print speed now available with LINEAR_ADV in Marlin RC8. Oh well, I see it's on a lot of people's minds beside mine.

[thedoble]

Hi @fiveangle

I have only done a couple of tests but so far it seems that disabling the 'history' plugin has resolved my issue. I suggest you give it a try if you are having the same problem.

Thanks everyone for their discussion and help with this, and thanks @foosel for her great work as always.

[foosel]

@fiveangle also, THIS issue here hasn't been closed (even though so far all reports that actually did came back appear to indicate the history plugin), a duplicate of it has. The ticket in the history plugin's tracker however has been closed It looks like the ticket in the plugin's tracker is actually still open too, so anyone still experiencing issues like this here when this plugin is installed and active in the current version (of both OctoPrint and the plugin) and which vanish when the plugin is disabled or uninstalled should please report this to @imrahil. I can't solve issues with a plugin I do not maintain myself.

In general, OctoPrint in 1.3.0 now supports starting in safe mode, which will disable all third party plugins. To restart OctoPrint in safe mode there are currently four options:

• Starting OctoPrint manually from the command line with the --safe parameter: octoprint --safe
• Editing config.yaml manually and setting server > startOnceInSafeMode to true, then restarting
• Doing the same via OctoPrint's new config command line interface: octoprint config set --bool server.startOnceInSafeMode true + restart
• Doing the same via the YamlPatcher Plugin by applying the patch [["set", "server.startOnceInSafeMode", true]] and then restarting

And yes, I'll add some form of checkbox for that into the settings in the future.

When the issue vanishes after starting OctoPrint in safe mode, it is caused by at least one of the installed third party plugins.

From personal experience, unless I print with very tiny segments, I can refresh as much as I want under 1.3.0, it won't interfere with the print in any noticable way (audibly or visually in the result).

[chaseacton]

I can confirm disabling the print history plugin on both of my printers fixed this issue for me. I can also confirm that this issue is no longer present with the latest version of the plugin.

[fiveangle]

Disabling print history plugin did not resolve for me, but I have too many variables to state with 100% confidence if it is OP yet. Should have some time to test/eliminate in the next few days.

[fiveangle]

In my case, it looks like it is related to octopi installation where octoprint goes to "sleep" or is otherwise not available and when refreshing page, haproxy instances are launched which temporarily starve the octoprint process for resources enough that it can't keep up with serial coms to Marlin, causing printing to stutter.

I'm still investigating but I moved back to 1.2.0-dev-649-gaa0e7ae6 (beta) by loading an old backup image of SD card. With no other changes, I don't get the same behavior, but then I realize that's too far back to probably be of any use.

One thing I noticed is that tweaking Octoprint process to startup with priority "--realtime" (daemonstartstop options) helped reduced the stuttering since it helps prioritize OP resources over the haproxy ones launching (I tried setting haproxy daemon options to "--idle" but haproxy would not adjust priority even though it would launch).

I even upgraded to Pi3 to try and resolve (from Pi2) but didn't fully on the fastest prints with lots of small curves and LIN_ADVANCE option enabled on Marlin.

I do like the plugin features of OP (which are not available in 1.2.0 beta branch I have working) so I will try and figure this out hopefully when I have more time. If it appears ocotopi env related, i'll open a defect on the tracker.

[nophead]

When haproxy uses a lot of CPU time I think it is actually the wifi driver
that is using all the CPU streaming the video. I have to limit the frame
rate to 4 fps when using wifi.

On 16 January 2017 at 21:19, Dave Johnson [e-mail] wrote:

In my case, it looks like it is related to octopi installation where
octoprint goes to "sleep" or is otherwise not available and when refreshing
page, haproxy instances are launched which temporarily starve the octoprint
process for resources enough that it can't keep up with serial coms to
Marlin, causing printing to stutter.

I'm still investigating but I moved back to 1.2.0-dev-649-gaa0e7ae6 (beta)
by loading an old backup image of SD card. With no other changes, I don't
get the same behavior, but then I realize that's too far back to probably
be of any use, so I'll keep plugging away.

Once thing I noticed is that tweaking Octoprint process to startup with
priority "--realtime" (daemonstartstop options) helped reduced the
stuttering since it helps prioritize OP resources over the haproxy ones
launching (I tried setting haproxy daemon options to "--idle" but haproxy
would not adjust priority even though it would launch).

I even upgraded to Pi3 to try and resolve (from Pi2) but didn't fully on
the fastest prints with lots of small curves and LIN_ADVANCE option enabled
on Marlin.

I do like the plugin features of OP (which are not available in 1.2.0 beta
branch I have working) so I will try and figure this out more when I have
time.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/foosel/OctoPrint/issues/1241#issuecomment-272964426,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAijhT934vbtiCSmGYA_abZ_lRssAa95ks5rS97pgaJpZM4HhbWI
.

[fiveangle]

No, that is not it in my case.

Edit: but I will keep that in mind while investigating.

[markwal]

Ah. That makes sense for some other reports I've heard. I wonder if there's a bug such that the gcode viewer is downloading the file on page load (It's supposed to wait until you click on the tab). I've been noticing that it is already ready when I click over. Over wifi that download would probably show up in haproxy's process if the Wi-Fi driver is eating it.

[foosel]

@markwal actually, it never was coded to not load until the viewer is opened, only to stop live rendering when it's not visible. But I changed that in the above commit now since it definitely should help mitigate an initial bandwidth spike, especially if all you want is the current progress or something. So that might also help with bandwidth induced CPU load on page load.

[markwal]

Ah. Oops. I guess I overgeneralized from some other tab switching code. But nice feature, I think it should help some of these load spikes.

[krisrok]

just wanted to chime in, it's still happening with the newest builds.

ctrl + f5 (or loading the website with a new browser) results in octoprint process going up for 5-10 secs.
meanwhile, the print gets paused. after the huge startup delay there is some minorish stuttering while building/loading the website.

RPi 2B, connected via ethernet, pi camera connected

[foosel]

@krisrok have you tried safe mode?

I'm not sure what browser you are talking about here and I might be mistaken, but I think that Ctrl+F5 actually does enforce a cache refresh, which would explain performance issues. You are basically telling OctoPrint to throw out the (expensive to generate) cached copy of the webpage and generate it anew, which on an RPi2b depending on the printed model indeed will cause issues. That doesn't explain why it's also causing issues when you simply load the page in a new browser though. Please provide octoprint.log, that might contain some valuable insights here.

[forkineye]

I just noticed an issue on my machine where the buffers were getting low (Wanhao I3) and it would start stuttering. Logging into my Pi showed haproxy (and octoprint) at over 40% CPU each and a load average of over 5! Closing the tab in my browser left the haproxy usage un-changed. I had completely close the browser for haproxy to release the CPU. Using Chrome 60.0.3112.101

[klcjr89]

Through trial and error I've found out what can cause stutters and lag. The ethernet port is on the same bus as the USB, so it's actually better to use WiFi (I know it's counter-intuitive to use wireless instead of wired, but it's true). Ethernet isn't the problem in itself, but rather when you want to watch a video stream, all of that traffic is actually on the USB bus.

If you have a USB webcam, well that's also on the same bus as the printer, so reduce your resolution and/or framerate and that should help quite a bit as well. This was tested on the Raspi 3 model B. With the Logitech C920, I had to go from 720p 30fps to VGA 30fps. Mjpg streamer isn't the best way of serving a stream due to the nature of having to generate a jpg image x amount of times a second. A better solution would be hardware decoded H.264 HTML5 stream; alot of webcams don't have this though.

I have since upgraded to a 32 bit Smoothieboard and unfortunately don't trust the Raspi hardware to keep the buffer full over USB, so I will have the Raspi dedicated to streaming 1080p video using OctoPrint and I will use the Smoothie's crappy web interface for printing. I wish there was a better all in one solution to have one device for everything. The Duet only supports IP cameras which is a poor solution.

[fiveangle]

Gina- Have you given thought to breaking out just the gcode->serial (usb) process so it could be run as a separate daemon and be given a high (highest) priority level ? I know you said that would be too much work. Do you still feel the same ? Printers aren't getting any slower. I think everyone can accept dealing with the slowness of the UI if we have confidence that the code is being sent as real-time as possible (given the limitations of the platform, in this case Raspberry Pi). Or is this just too much of an undertaking ?

[foosel]

@fiveangle See this. It's not "too much work" that's the issue here, it's full blown backwards incompatibility. Making the comm layer modular is the first step towards this since it would allow coexistence of some backwards compatible layer and something separate/native - and something I keep getting distracted from by the regular "every day", regular maintenance and high prio bugs and such, which is why that is still an ongoing task.

[dragonnn]

Sometimes you need to break backwards compatibility. My octoprint is running on Atom 1.6GHz and I am having this problem too. And the CPU don't hit 100% usage when loading the website, the load is between 50-70% max.

[Diaoul]

I got this issue too running on my RPi 1. No issue when using Repetier Host (so smooth), no issue (or almost?) when I close the web UI. However as soon as it is open, the print stutters in the small holes and curves.

Personally I don't mind backward incompatibility, people that don't want the update can just keep the current version. Besides this issue, OctoPrint always gave me satisfaction for the past years.

Maybe something less radical could be done in the meantime like decrease the load of the webui, maybe in the terminal tab something can be done?

Edit: I've increased the baudrate to 500000, it seems to behave better but I need more testing.

[universam1]

Closing existing issues in favor of an apparently dead thread does not really solve the problem.
Is there any idea what should help?

[marcboivin]

It's not a simple matter. The comm stuff is really core to Octoprint and it's not a trivial thing to modify that heavily. Plus if we get into threading there are a lot more things that can go wrong.

Only @foosel can tackle this with any hope of success. And if it's breaking compatibility, we would then talk about a V2 feature.

We could maybe try to make new page load a bit more gentle on the CPU by deferring loading of some components and cache more aggressively. A more achievable endeavor, but not much guarantee.

[foosel]

OctoPrint is already caching pretty aggressively (unless you keep forcing a full blown refresh, see above). The HTML code itself should only ever be generated once per targeted URL and locale (and that will even happen on server startup unless people have the preliminary cache disabled). The individual REST requests used to get file lists and whatnot from the backend also make heavy use of caching headers and validation checks. There are no really any low hanging fruits left here I fear, but I'd be happy to be proven wrong.

I can't vouch for the caching behaviour and resource handling of any installed third party plugins however (which in the past have been the source of stuttering during page load, see above), so the first step towards debugging any kind of stuttering issues should also always be running in safe mode and checking if the issues vanish in that case - if so, a plugin is putting too much load on things.

Closing existing issues in favor of an apparently dead thread does not really solve the problem.

Opening one ticket after another about the same phenomenon causes tremendous overhead for me, taking away a ton of time for ticket management that could otherwise be used for fixing issues or general improvement for anyone. All I'm asking for is to please keep the discussion and reports of one thing to one ticket so I don't have to spend hours every week trying to navigate the tracker and clean up duplicates.

[universam1]

Thanks for the effort put into this project, OP is for sure a one-of-its-kind that has proven faithful for a long time!!
Hence I am really sad that the current situation forces me to think about alternatives. It's too bad that this i damaging basically every print if I am impatient to wait until the end of the print before going to its web page. But unfortunately this is a serious issue.

I disabled all plugins already because it smelled to me, but I will do a clean safe run as well according to your request.

I've also traced in detail the load on the system and it is clear that this is not the reason at all. So digging deeper to optimize the load will not bring the benefit.
Nevertheless, something is blocking the comm handling routine and probably only @foosel will know what changed compared to older versions.

Threading that off from other workload is definitely key for the future success even if it might break something for a certain time. Timecritical stuff just cannot happen in a cooperative single threaded concept in a safe way - granted that I am not the expert here.

[JimJamUrCode]

I spent countless hours trying to debug my printer to find out that loading the web interface to check on a print was causing defects in my prints. I really do love OctoPrint. I have 3 printers that are all being controlled by OctoPrint via Raspberry Pi 3. Each and every one of them has this issue. Fresh OctoPrint installs don’t change anything. Is the history plugin installed by default? I am going to try to boot into safe mode and see if that helps. Thanks for all of your hard work.

[burner-]

I can confirm that I have same problem. I use latest marlin firmware with printer. When I open marlin web page print will not only lag or freeze but it can skip G-code rows. If I close and open web page at first layer and I have z-lift enabled it can cause skipping for lift row and then it mash head to table.

There has been suggestion that problem is caused by history plugin. I dont have it in use. Also there had been suggestion than this occurs only if there is small moves in command buffer. With tests I notice that it is not a case neither. I have 50x50cm print table i my printer and with large and simple prints there can be ~15-30 seconds in buffer (it take ~15-30 second to stop my print). Even so when I open my web page it can cause wrong moves, skip of g-code lines etc problems. So important question is that what firmware people use in their printer?

This looks that there is 2 problems A) there is something wrong in marlin firmware or octoprit so that it can skip g-code lines if there is lag in sending (or receiving Recv messages). and B) there is performance issue what brings that problem a to visible. That why I think it was mistake from @foosel to close ticket #2777 because it was report of that problem A and this ticket is more like report of problem B. Yes they both are related to each other but not necessarily a same bug.

[burner-]

Actually I just notice that if this is some kind communication problem with printer & octoprint this needs also more information what board people use. I use ultratronic and I notice that there is some odd code at fimware

if ENABLED(ARDUINO_ARCH_SAM)

#undef TX_BUFFER_SIZE
#undef RX_BUFFER_SIZE

So if you have this problem pleas tell what board and firmware you use with it. Lets try find some kind common factor why some of users have this problem and why some users does not have this problem.

[universam1]

I run Marlin 1.1.9 stable but tried older versions without difference.
Also I monitored/ hunted for errors in communication but none were logged, that’s why I see no indication that the firmware’s are to blame.

Btw is there a good alternative to Octoprint someone can share? Unfortunately it becomes unusable for me now.

[foosel]

@universam1 take a look at Repetier Server.

Also anyone encountering this make sure to check if it happens in safe mode. It might be some
third party plugin that causes too much load on the machine while the page is loading, in which case I can't do anything about it.

If it is being encountered, share a fully filled out ticket template here as the contribution guidelines ask you to when reporting a "me too". I can't reproduce this myself and I can't analyse it blind.

[nophead]

You can always exceed the USB comms bandwidth of any version of OctoPrint if the combination of your slicer's output and print speed exceeds a finite number of line segments per second. I think this is why it becomes unusable for some people and not others. It depends on the resolution of your STL file, whether your slicer combines tiny segments or not, and how fast you print.

[universam1]

How would that relate to the page reload issue discussed here?

If so, that would be a bug in OP since it would need to check that the FIFO is not overrun.
Ex.:
```
while serial.outWaiting() > 0 :
pass

[nophead]

When OctoPrint is busy generating web pages the maximum number of segments per second is less. If you stick below what ever that is then you never see a problem.

If I silce yoda with Cura I can't print it over USB at 50mm/s without stuttering. If I slice it with Skeinforge I can.

[burner-]

@universam1 I understand that if there is no enough cpu printer can start stutter because it does not get enough lines. But I dont understand how it can cause skipping lines without firmware or comms lib bug. At least in MK4duo (and probably in marlin) there is some magic in planner what will cause that one linear move can get splitted to two if there is no lines in buffer. You can test it by doing one single move and you get result like this https://www.youtube.com/watch?v=DBFeBk0lPwU This is now just guessing but if that kind feature includes bug it is quite clear way to reproduce these problems when combined to performance issue of octoprint. So that why I am interested also of firmwares. Naturally performance issue is also problem but I like to find also where these printing offsets will come. We can hide that problem by fixing so that there is no performance issues BUT it just hide real problem so that these offsets occurs even more randomly with larger prints.

[nophead]

If a line gets split in two then that is a firmware bug. All that should happen when the buffer runs dry is the print should pause. That will tend to cause a blemish as the filament oozes but not any gross errors unless the print head catches on a blob.

[universam1]

Just to mention @foosel that this is getting really terrible with 1.3.11, the head does not bump but crashes about 5mm -Z into the print. Broke my head actually!

I can confirm from a friend of mine with different setup the same happens now as well!
It was related with enabling bed level for him.

Basically reload the page while printing is all it needs.
Please confirm this as a bug finally

[foosel]

Just to mention @foosel that this is getting really terrible with 1.3.11, the head does not bump but crashes about 5mm -Z into the print. Broke my head actually!

That sounds like a completely different issue. Not being able to feed data fast enough to the printer should never make the printhead drop magically. Mangled GCODE might. In which case I'd need logs to see if there were communication issues going on before.

I still haven't been able to reproduce slowdowns on any version ever since this ticket was opened on reload unless I explicitly nuke the caches and run on lower end Pi models. This ticket however is still open regardless, and I've also just added the labels that weren't yet a thing back when the ticket was initially created. Other than that I cannot do anything unless I get more data and hopefully a reproduction 🤷‍♀️

To quote myself:

Also anyone encountering this make sure to _check if it happens in safe mode_. It might be some
third party plugin that causes too much load on the machine while the page is loading, in which case I can't do anything about it.

If it is being encountered, _share a fully filled out ticket template here_ as the contribution guidelines ask you to when reporting a "me too". I can't reproduce this myself and I can't analyse it blind.

So far no one has reacted to that.

[burner-]

I find one problem with MK4duo. If printer is connected via non native usb port it will do many kind stupid things. My freezing and shuttering problems went away when I move octopi communicate via native usb port.

[xaqaria]

I've noticed this off and on for some time. I am about to move from a Raspberry Pi 3B to 3B+ and thought I would try to resolve. Details are below, but _as I was capturing the data I'm also noticing the UNDER VOLTAGE_ alert I don't remember seeing before.

_Let me address the alert before you do any work with this feedback._

Video link wouldn't work in code section:
Video

#### What were you doing?
1.  Started a print
2.  Waited for print head to begin moving
3.  Logged on to the http://octopi page
4.  Printer started to stutter

#### What did you expect to happen?
Printer wouldn't stutter

#### What happened instead?
Printer stutters

#### Did the same happen when running OctoPrint in safe mode?
* occurred
* uninstalled disabled plugins and it occurred
* ran in safe mode and it occurred


#### Version of OctoPrint
1.3.11

#### Operating System running OctoPrint
Octopi 0.15.1 - octopi 4.14.34-v7+ #1110 SMP Mon Apr 16 15:18:51 BST 2018 armv7l GNU/Linux

#### Printer model & used firmware incl. version
Ultimaker 2+  fw 2.6.2


#### Browser and version of browser, operating system running browser
Windows 10 1903
Chrome 76.0.3809.100
Firefox 68.0.2 

#### Link to octoprint.log
https://gist.github.com/xaqaria/1c5f4eb163be8ed22a4f4ceee3ce3f87

#### Link to contents of terminal tab or serial.log
https://gist.github.com/xaqaria/678893c0cd7b63a7fdda38a9a9655f1e

#### Link to contents of Javascript console in the browser
Not available

#### Screenshot(s)/video(s) showing the problem:

[Video](https://photos.google.com/share/AF1QipNCj58Q4HmHu6N9idjc59pp8QTJdTGDiGqCMGVFePCHmrB9CnkYp1hKW5CQZK6q7Q/photo/AF1QipMu871lwm8h08kPmfw_9asQCGhqcP02isGJeIIq?key=TlJ2VnFBQ2pmUVVSTTBMVW9uT1VnWWVvbUJLX3NB)

update: addressing video link

[xaqaria]

I've addressed the Under Voltage and it didn't address the issue for me. From there I've tried the following to no avail:

• Timelapse disabled
• MJPG Streamer set to:
  
  
  
  • 1920x1080@30fps
  
  
  • 1280x720@30fps
  
  
  • 1280x720@15fps
  
  
• webcamdaemon disabled/camera unplugged

[thedoble]

Have you tried deleting your job history?
Have you tried removing all other files from file list?

On Tue., 20 Aug. 2019, 1:54 pm Jason McSorley, [e-mail]
wrote:

I've addressed the Under Voltage and it didn't address the issue for me.
From there I've tried the following to no avail:

• Timelapse disabled
• MJPG Streamer set to:
  
  
  
  • 1920x1080@30fps
  
  
  • 1280x720@30fps
  
  
  • 1280x720@15fps
  
  
• webcamdaemon disabled/camera unplugged

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/foosel/OctoPrint/issues/1241?email_source=notifications&email_token=ADSB7ARGPODHPSOLO42W3UTQFNTITA5CNFSM4B4FWWEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4U7BTQ#issuecomment-522842318,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADSB7AVSCHHEQJGZS4KSRPLQFNTITANCNFSM4B4FWWEA
.

[xaqaria]

@thedoble I think you may have hit on something. I need to turn everything else back on and re-test.

Adding a single browser does load a lot faster and it doesn't trigger the stutter. I could get it to stutter by quickly logging in two different browsers at the same time, but it was a lot shorter.

[justfalter]

I don't really understand why a hard browser refresh (Ctrl-Shift-R - leading to a request with Cache-Control: no-cache) should be tied to the OctoPrint service refreshing its caches. Typically, a request with Cache-Control: no-cache would cause intermediate proxies to invalidate their caches, but not the application server's caches. Why would OctoPrint ever need to invalidate its internal
caches _outside_ of development?

In my opinion, the refreshing of OctoPrint's internal caches should not be tied to a browser refresh, perhaps it should be be a separate REST API endpoint?
At the very least, priority should always go to an in-progress print-job. Perhaps OctoPrint should refuse to do a refresh if a print is in progress?

[justfalter]

@foosel Do you have any thoughts on the above? It is trivial for a user to cause OctoPrint to refresh its caches with a hard-refresh of their browser. On raspberry pi hardware, this can lead to several seconds of CPU spikes that negatively affect an in-progress print.

[foosel]

Happy to accept the above PR if the backwards compatibility issue is solved (trivial change).

[RobeeeJay]

Hi! Love OP, thanks for all your work Foosel!

I am having random rare problems which are OP CPU related on my Prusa Mini, and after disabling various plugins, thinking this was all sorts of other reasons and eliminating them, I am now suspecting this is the same issue.

When this happens, on my Pi 3 B+, OctoPrint consumes 30% of the CPU, and it is not keeping up with providing Gcode to the printer. At it's worst it leaves all sorts of artifacts on the model I'm printing and can make a print which is due to finish in 10 minutes not finish for over an hour.

I think this might be related when I'm using Safari to monitor the print, and maybe it's quite aggressive at suspending tabs when I task switch.

Does this attached octoprint log look normal? When this happened I only had one browser open talking to OctoPrint, and only one tab. Webcam was off, Octolapse was off, Display Layer Progress was off.

octoprint.log

[Phil1988]

I am having the same issue since always :D

I never knew that this is not normal.
It's surprising to me, that this is actually preventable and I will do further research on this.

@foosel :
You mentioned that disableing the plugins by using safe mode might avoid that.
I will check this for sure in the next days and give you a feedback, but before that, I would like to see/get as much information as possible.
This way I want to make sure, that I identify my problem and give you (all) as much benefit from it as possible.
Hopefully this might help others to and get the source of the problem.
I read that the "history" plugin might be a good candidate for this issue.

To give you a short overview about my setup:
I am using this instance of Octopi for quite a long time with a lot of plugins installed.
I'm printing from octoprint directly (no SD card).
So I am having multiple files (really a lot) uploaded.
There might be a lot in the history as well ;)
I am also using basic auth. This is typically not used in a wide field of users.

How can I check the plugins load and time needed?
"top" is very limited and "hto"p also ony shows this after hitting F5:

@nallar : You have a nice graphic for that.
Can you or anyone else tell me how to get a great view like that?

EDIT:
CPU load during a fast (real 85mm/s) 3D print, with a round shape (=> many commands for the curves):

EDIT 2:
This is my timeline:

(For anyone interested, this is made by the google chrome development tools under network)

Needed 15.02s to finish the request.

EDIT 3:
reloaded with save mode:

13,29s (can be measuring inaccuracy)

EDIT 4:
after deleting all (2.08GB) if timelapses:

13,96s
still not really a change...

EDIT 5:
Ok! now we have something.
After deleting 498 uploaded .gcode-files:

4.43s

EDIT 6:
safe mode now turned off and sorted by response time:

7.16s is definetly an improvement.

I will test tomorrow if the stuttering appears again while printing and report back.

[foosel]

How can I check the plugins load and time needed?

You currently can't. There are now debugging tools or anything. You can use the web dev console as you already noticed, and you can use https://docs.octoprint.org/en/master/development/request-profiling.html but that's about it at the moment (as always, contributions welcome).

What I find weird here is that it loads the file list completely fresh. That is one of the most heavily cached endpoints precisely to avoid issues like this, so if it isn't properly caching here, the question is why. Will take a look.

[RobeeeJay]

Would it be difficult to add a simple timer whenever OctoPrint calls a plugin and if it takes over 100ms it logs the name of the plugin with warning? That might help locate the cause easier. I'm not sure how many places plugins are called regularly during printing, but if it's not that many I could try this.

if it was useful, it could also be extended to the UI alerting users that a plugin is consuming too many resources and give them the option to ignore the warning or disable the plugin.

[foosel]

Would it be difficult to add a simple timer whenever OctoPrint calls a plugin and if it takes over 100ms it logs the name of the plugin with warning?

Not difficult per se, but there's a ton of places where hooks and implementations are used, potentially even in other plugins, so that would become quite a lot of places that would need wrappers, some of which might even be tricky to pull off. I think it might be time to look into something like this however as it certainly would help debugging performance issues.

Meanwhile I've discovered an issue with the caching header evaluation that led to the cache for e.g. the files api endpoint not working as intended. The internal files cache still worked, but the response to the files request always had to be processed instead of just returning 304, which might explain some performance issues. Mind you, that only affects situations where you load the UI again in the same browser, not when you fire up a completely fresh browser with a clean history.

[foosel]

some of which might even be tricky to pull off

Strike this, I just had an idea, will see I can get that to work centrally in the plugin core.

[foosel]

So, OctoPrint 1.5.0 will ship with a new debug option, the plugintimings.log:

Enabling this will create a plugintimings.log in your log folder that contains stuff like this:

2020-09-30 13:50:23,434 - octoprint.plugins.action_command_prompt.ActionCommandPromptPlugin.on_event - 00.00ms
2020-09-30 13:50:23,442 - octoprint.plugins.announcements.AnnouncementPlugin.on_event - 00.00ms
2020-09-30 13:50:23,443 - octoprint_file_check.FileCheckPlugin.on_event - 00.00ms
2020-09-30 13:50:23,443 - octoprint_firmware_check.FirmwareCheckPlugin.on_event - 00.00ms
2020-09-30 13:50:23,443 - octoprint.plugins.pluginmanager.PluginManagerPlugin.on_event - 00.00ms
2020-09-30 13:50:23,443 - octoprint.plugins.softwareupdate.SoftwareUpdatePlugin.on_event - 00.00ms
2020-09-30 13:50:23,444 - octoprint.plugins.tracking.TrackingPlugin.on_event - 00.00ms
2020-09-30 13:50:23,444 - octoprint_firmwareupdater.FirmwareupdaterPlugin.on_event - 00.00ms
2020-09-30 13:50:23,444 - octoprint_z_probe_offset.Z_probe_offset_plugin.on_event - 00.00ms
2020-09-30 13:50:23,446 - octoprint.plugins.backup.BackupPlugin.socket_emit_hook - 00.00ms
2020-09-30 13:50:23,448 - octoprint.plugins.action_command_notification.ActionCommandNotificationPlugin.on_event - 00.00ms
2020-09-30 13:50:23,474 - octoprint.plugins.action_command_prompt.ActionCommandPromptPlugin.on_event - 00.00ms
2020-09-30 13:50:23,492 - octoprint.plugins.backup.BackupPlugin.socket_emit_hook - 00.00ms
2020-09-30 13:50:23,493 - octoprint.plugins.announcements.AnnouncementPlugin.on_event - 00.00ms
2020-09-30 13:50:23,533 - octoprint_file_check.FileCheckPlugin.on_event - 00.00ms
2020-09-30 13:50:23,534 - octoprint_firmware_check.FirmwareCheckPlugin.on_event - 00.00ms
2020-09-30 13:50:23,560 - octoprint.plugins.pluginmanager.PluginManagerPlugin.on_event - 00.00ms
2020-09-30 13:50:23,566 - octoprint.plugins.softwareupdate.SoftwareUpdatePlugin.on_event - 00.00ms
2020-09-30 13:50:23,567 - octoprint.plugins.tracking.TrackingPlugin.on_event - 00.00ms
2020-09-30 13:50:23,578 - octoprint_firmwareupdater.FirmwareupdaterPlugin.on_event - 00.00ms

It will also create a plugintimings.csv file at the same location:

2020-09-30 13:55:06,019;octoprint.plugins.tracking.TrackingPlugin.on_event;0.000000
2020-09-30 13:55:06,019;octoprint_firmwareupdater.FirmwareupdaterPlugin.on_event;0.000000
2020-09-30 13:55:06,019;octoprint_z_probe_offset.Z_probe_offset_plugin.on_event;0.000000
2020-09-30 13:55:07,485;octoprint.plugins.backup.BackupPlugin.socket_emit_hook;0.000000
2020-09-30 13:55:07,488;octoprint.plugins.backup.BackupPlugin.socket_emit_hook;0.000000
2020-09-30 13:55:07,489;octoprint.plugins.action_command_notification.ActionCommandNotificationPlugin.on_event;0.000000
2020-09-30 13:55:07,490;octoprint.plugins.action_command_prompt.ActionCommandPromptPlugin.on_event;0.000000
2020-09-30 13:55:07,490;octoprint.plugins.announcements.AnnouncementPlugin.on_event;0.000000
2020-09-30 13:55:07,491;octoprint_file_check.FileCheckPlugin.on_event;0.000000
2020-09-30 13:55:07,491;octoprint_firmware_check.FirmwareCheckPlugin.on_event;0.000000
2020-09-30 13:55:07,491;octoprint.plugins.pluginmanager.PluginManagerPlugin.on_event;0.000000
2020-09-30 13:55:07,491;octoprint.plugins.softwareupdate.SoftwareUpdatePlugin.on_event;0.000000
2020-09-30 13:55:07,491;octoprint.plugins.tracking.TrackingPlugin.on_event;0.000000
2020-09-30 13:55:07,491;octoprint_firmwareupdater.FirmwareupdaterPlugin.on_event;0.000000
2020-09-30 13:55:07,492;octoprint_z_probe_offset.Z_probe_offset_plugin.on_event;0.000000
2020-09-30 13:55:07,615;octoprint.plugins.backup.BackupPlugin.socket_emit_hook;0.000000
2020-09-30 13:55:07,615;octoprint.plugins.action_command_notification.ActionCommandNotificationPlugin.on_event;0.000000
2020-09-30 13:55:07,616;octoprint.plugins.action_command_prompt.ActionCommandPromptPlugin.on_event;0.000000
2020-09-30 13:55:07,616;octoprint.plugins.announcements.AnnouncementPlugin.on_event;0.000000

Both are rotated on server start. They log A LOT, so this should really only be enabled when actively hunting performance issues. Consequently there's also a navbar icon visible if it's enabled.

Should be available on the maintenance branch as of now.

[RobeeeJay]

Wow that was really quick! Thank you for this!!! You're the best!