Octoprint: [Request] Nanny cam. Stop print or Alarm when the current print doesn't match the expected silhouette.

[dm3942]

Let octoprint warn you or stop your print before you end up with spaghetti all of the place.

How would this work. For each layer it is possible to extract a silhouette of the object. By using the camera to take a picture at each level, ensure that the expected silhouette for that layer matches what has been printed. Given that most home printers only print in one color openCV should be able to extract a silhouette of the object being printed.

To start with the camera would need to be positioned facing the side of the print that is not obstructed by the print head.

The attached image demonstrates the idea proposed.

[GitIssueBot]

Hi @dm3942,

It looks like there is some information missing from your bug report that will be needed in order to solve the problem. Please take a look at the Contribution Guidelines which will provide you with a template to fill out here so that your bug report is ready to be investigated (I promise I'll go away then too!).

If you did not intend to report a bug, please take special note of the title format to use as described in the Contribution Guidelines.

I'm marking this one now as needing some more information. Please understand that if you do not provide that information within the next two weeks (until 2016-03-19 12:10) I'll close this ticket so it doesn't clutter the bug tracker. This is nothing personal, so please just be considerate and help the maintainers solve this problem quickly by following the guidelines linked above. Remember, the less time the devs have to spend running after information on tickets, the more time they have to actually solve problems and add awesome new features. Thank you!

Best regards,
~ Your friendly GitIssueBot

PS: I'm just an automated script, not a human being, so don't expect any replies from me :) Your ticket is read by humans too, I'm just not one of them.

[markwal]

The plugin API has everything you would need to write this as a plugin. It won't be an easy plugin to write such that it is easy to use mind you, but the API is sufficient.

One of the problems you will have is helping the user to calibrate the angle of view so that your plugin will know how to project a matching silhouette. Another problem will be to do the object vs. background separation. There is software for this, but you need it to be fairly accurate (you don't want a false positive to ruin a good print) and fast (you're still trying to run on a pi, right?) You might ask the user to make sure that the plastic and the printer and/or bed are of contrasting materials (green screen style).

[dm3942]

Thanks Mark,

Can't say I'm the guy for the job. But when I get time I'll see what I can do. Happy for others to pursue this.

Maybe spaghetti detection will be an easier first plugin. :)

Regards,
Daniel.

Important:This transmission is intended only for the use of the addressee. It is confidential and may contain privileged information or copyright material. If you are not the intended recipient, any use or further disclosure of this is strictly forbidden. If you have received this transmission in error, please notify me immediately by telephone and delete all copies of this transmission as well as any attachments.

On 6 Mar 2016, at 6:24 AM, Mark Walker <[email protected][e-mail]> wrote:

The plugin API has everything you would need to write this as a plugin. It won't be an easy plugin to write such that it is easy to use mind you, but the API is sufficient.

One of the problems you will have is helping the user to calibrate the angle of view so that your plugin will know how to project a matching silhouette. Another problem will be to do the object vs. background separation. There is software for this, but you need it to be fairly accurate (you don't want a false positive to ruin a good print) and fast (you're still trying to run on a pi, right?) You might ask the user to make sure that the plastic and the printer and/or bed are of contrasting materials (green screen style).

—
Reply to this email directly or view it on GitHubhttps://github.com/foosel/OctoPrint/issues/1256#issuecomment-192712800.

[thinkslynk]

A few notes from someone who maintains an OCR library for a living. This is completely doable, though would be difficult.

1) You first need to determine the perspective of the camera. This could be easily determined by object tracking the extruder and moving it to each of the 4 corners both close to the bed and top height. This would give you 8 points in space to determine the camera perspective.

2) You would need to render the model in that perspective with appropriate scale to a 2d image and binarize it. (Black being "stuff" and white being "something else." This will make it easier to determine where everything should be.

3) When printing begins, determine filament color by watching the base print. Average the colors

4) Every frame, for X padding of pixels around the expected object location, if color is some small distance from the average color, add to counter. When counter exceeds some threshold, cancel print / pause print / send user push notification / self destruct / etc.

There's still the issue of the filament matching the background, which could by fixed by binarizeing each frame with a thresholding algorithm that accentuates edges. But it complicates things. Maybe just disable the plugin if you detect the average base color to be too similar to the environment. _shrug_.

There's also an issue of not counting the extruder in the color check, which could be solved a number of ways. (Tracking via whatever method used in step 1. Tracking by translating gcode positions to camera perspective. Etc)

I may play around with this but I'm still 2 weeks out from receiving my 3d printer. :(

[dm3942]

Hi Foosel,

I'm glad to hear we will have someone with your expertise looking into this. I think this will really change home 3D printing in a big way. Eliminating the fear of a big ball of ABS goop stuck to the end of the extruder or a spaghetti monster growing over night. Not easy, I know, but really practical.

So I like your ideas, below are my thoughts which might help you look at the problem differently as you develop the solution. Given your experience and the fact that you are taking on the code, I'm just here to share some ideas. I'm happy for you to take the lead here, experience trumps untested ideas. [&#X1f60a]

1. Absolutely. And another option would be to inject a bounding box or square into the G-Code for the first layer to capture perspective. This would provide a reference square or rectangle to seek. At least one full edge and 2 corners must be visible to calculate the perspective.

Alternatively the base plate could be used to determine perspective. Once the user has inputted the dimensions of the build plate (not build area) it would be possible to position the camera so that 2 edges are visible.

1. Sounds good. In addition I was thinking that instead of comparing the silhouette at every position, it only needs to be matched at a single Y position (assuming the bed only travels along the Y axis). Say Y position 0 or the average of the G-Code in layer 1 if the print was not started at the center.
   Assuming that a Raspberry Pi camera is being used we might only be able to check every 3-4 layers.
2. Yes, black on black will be a problem. But we might need to recommend that painters tape be placed on the entire bed to provide contrast. Other than that my CocoonCreate from ALDI (a Wanhoa Duplicator i3 V2 replica) has a black mat on the build plate so detecting grey, blue, white, should be possible.
   But I'm thinking the square/rectangle injected into the g-code is probably best as filament color can be found along the lines of the injected shape.
3. Agree. I like the self-destruct option for top secret prints. :)

Your final comments. Totally agree, this is were testing will help. But the counter idea plus maybe just checking every few layers at Y position 0 will reduce the amount of processing required.

I hope your introduction to 3D printing is a good experience. I am now on my second 3D printer and it is 10 times better than my first one.

Regards,
Daniel M.

---

From: thinkslynk [email protected]
Sent: Saturday, 12 March 2016 12:00 AM
To: foosel/OctoPrint
Cc: Daniel M
Subject: Re: [OctoPrint] [Request] Nanny cam. Stop print or Alarm when the current print doesn't match the expected silhouette. (#1256)

A few notes from someone who maintains an OCR library for a living. This is completely doable, though would be difficult.

1) You first need to determine the perspective of the camera. This could be easily determined by object tracking the extruder and moving it to each of the 4 corners both close to the bed and top height. This would give you 8 points in space to determine the camera perspective.

2) You would need to render the model in that perspective with appropriate scale to a 2d image and binarize it. (Black being "stuff" and white being "something else." This will make it easier to determine where everything should be.

3) When printing begins, determine filament color by watching the base print. Average the colors

4) Every frame, for X padding of pixels around the expected object location, if color is not some small distance from the average color, add to counter. When counter exceeds some threshold, cancel print / pause print / send user push notification / self destruct / etc.

There's still the issue of the filament matching the background, which could by fixed by binarizeing each frame with a thresholding algorithm that accentuates edges. But it complicates things. Maybe just disable the plugin if you detect the average base color to be too similar to the environment. shrug.

There's also an issue of not counting the extruder in the color check, which could be solved a number of ways. (Tracking via whatever method used in step 1. Tracking by translating gcode positions to camera perspective. Etc)

I may play around with this but I'm still 2 weeks out from receiving my 3d printer. :(

Reply to this email directly or view it on GitHubhttps://github.com/foosel/OctoPrint/issues/1256#issuecomment-195354674.

[foosel]

@dm3942 The person with the expertise in this is @thinkslynk, not me.

[dm3942]

Thanks for the correction @fooselhttps://github.com/foosel

@thinkslynkhttps://github.com/thinkslynk sorry my reply was misdirected. Thank you for showing interest in this feature / plugin for OctoPrint. Again, it's great to have someone with your expertise on board.

@dm3942https://github.com/dm3942

---

From: Gina Häußge [email protected]
Sent: Tuesday, 15 March 2016 7:41 PM
To: foosel/OctoPrint
Cc: Daniel M
Subject: Re: [OctoPrint] [Request] Nanny cam. Stop print or Alarm when the current print doesn't match the expected silhouette. (#1256)

@dm3942https://github.com/dm3942 The person with the expertise in this is @thinkslynkhttps://github.com/thinkslynk, not me.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/foosel/OctoPrint/issues/1256#issuecomment-196721297

[thinkslynk]

Yeah, no problem. Sounds like a fun project. ^^

The only other problem I thought of was that with some 3D printers the bed physically moves, right? Which means at best, the zoom of the silhouette changes. At worst, if the camera isn't positioned directly on the moving axis, the object would appear to move along 2 axis (relative to the camera.) This would be very difficult to deal with but could be fixed by mounting the camera to the bed so that they are locked.

This could also introduce motion blur if the object is in motion during a capture frame, but maybe that could be fixed by determining via gcode when the object isn't in motion.

[gddeen]

• How do potato chip manufacturers do it?
• Why scan an entire photo and come to an EXTREMELY difficult decision? (I know it won’t work that
  way… But a human would think many times.)
• Why not focus on the optics to produce a stream of a few bits around the nozzle. Now the decision would be
  not how to reject a print but to repair a spot, mark a spot on an object, or pause for intervention.
  LEDs can be placed and blinked to create shadows to detect around a point. Essentially 9 points could be studied in detail.
• The problem then becomes, study the known errors in a print and things to do to correct each one.
  While the print is happening, another expected stream is compared.

I feel the problem has to be attacked to reduce data analysis so it can be performed real time. A 3d print is a VERY complex
thing. Scaffolding may fail, but wasn’t needed anyway. Infill, isn’t important in all places. Perimeters are a big concern.

I’ve been studying the itty bitty issues just around the nozzle. A physical tool needs to capture very narrow things at high speed,
high frame rate with little content. A supervisor needs to go through the print process and generate an expected frame, and the
special process to run at each point. (WHEW.) THEN, one can get around to what to do with the info. That’s another entire story.

I’m not underestimating the scope of the problem for acceptable results. I have been studying this for several months. The amount
of thought a human can perform looking at a print is astounding (or just let it print…)

If an entire picture is used there will be many thousand points that are “wrong.” And it will be OK. I would think image processing
of large detailed media might be very processor intensive.

But, I’d like to see what you come up with… Because, after all, doing nothing is what is done now.

On Mar 15, 2016, at 9:11 AM, thinkslynk [email protected] wrote:

Yeah, no problem. Sounds like a fun project. ^^

The only other problem I thought of was that with some 3D printers the bed physically moves, right? Which means at best, the zoom of the silhouette changes. At worst, if the camera isn't positioned directly on the moving axis, the object would appear to move along 2 axis (relative to the camera.) This would be very difficult to deal with but could be fixed by mounting the camera to the bed so that they are locked.

This could also introduce motion blur if the object is in motion during a capture frame, but maybe that could be fixed by determining via gcode when the object isn't in motion.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/foosel/OctoPrint/issues/1256#issuecomment-196810823 https://github.com/foosel/OctoPrint/issues/1256#issuecomment-196810823

[thinkslynk]

You don't need to scan the entire photo, just a few pixels around where you think the object would be. That would catch print slipping and whatever you call the issue when half the object looks like spaghetti lol (still new to 3d printing.) You know where the object should be, and by proxy where it shouldn't be. You could even do a general height of object search to see if the extruder is jammed.

The library I work on right now scales down an iphone/android full res image to about 356 wide, with the height cropped before processing. We do a lot of processing intensive operations and still pull out 8 fps on an iPod reading 7 segment displays. Though I have no clue how that would compare to a raspberry pi. The problem with only analyzing the nozzle would be that you couldn't miss anything; it would have to be very fast and accurate. Where as the solution I'm proposing works no matter how slow the machine that runs it, and it reacts when ever it notices an issue.

[gddeen]

Well, I ALWAYS over think everything I do. Right at the moment, when I think I’d have to analyze a picture
to determine if it failed or not, I am overwhelmed. But, we didn’t have multiplication and division when I went
to school, only “by” and “goes into.” Times have changed!

[thinkslynk]

If done properly, your solution would be better. Your solution would be able to even grade printing stability / quality / accuracy by judging every step of the process from the point of extrusion. But I think that would be a lot harder to do, especially when the nozzle may not be in view. (In fact there is no perspective you could place a camera to always have full visibility of the nozzle the entire way through the print, so the camera would have to move.)

Analyzing an entire image is very overwhelming. You really have to think like a computer when writing these things. The computer doesn't see edges until you tell it how. It doesn't see depth unless you enable it to. It can't recognize features like text, even without reading it, unless you tell it how. Etc. But it's easier to judge something large with 99% accuracy, than something small with 99% accuracy. More room for error suppression.

School hasn't changed that much. I don't even have a degree. Found school to be boring so I taught myself programming in middle school. They push most kids to stop around algebra anyways, which isn't much further from multiplication and division.

[nophead]

Perhaps just have something that can recognise spaghetti. If looks like
spaghetti then stop as it is very visually different from anything correct
regardless of what that looks like. When it is only subtly wrong it is no
big deal but when it is producing spaghetti it makes a mess of the machine.

On 15 March 2016 at 18:49, thinkslynk [email protected] wrote:

If done properly, your solution would be better. Your solution would be
able to even grade printing stability / quality / accuracy by judging every
step of the process from the point of extrusion. But I think that would be
a lot harder to do, especially when the nozzle may not be in view. (In fact
there is no perspective you could place a camera to always have full
visibility of the nozzle the entire way through the print, so the camera
would have to move.)

Analyzing an entire image is very overwhelming. You really have to think
like a computer when writing these things. The computer doesn't see edges
until you tell it how. It doesn't see depth unless you enable it to. It
can't recognize features like text, even without reading it, unless you
tell it how. Etc. But it's easier to judge something large with 99%
accuracy, than something small with 99% accuracy. More room for error
suppression.

School hasn't changed that much. I don't even have a degree. Found school
to be boring so I taught myself programming in middle school. They push
most kids to stop around algebra anyways, which isn't much further from
multiplication and division.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/foosel/OctoPrint/issues/1256#issuecomment-196970275

[thinkslynk]

You're falling into the trap that everyone does when starting down the path of computer vision. What defines "spaghetti?" How do you differ it from someone's crazy abstract art that they are printing? How would you define it in code such that a computer can see it? Computer vision is very much about recognizing that your computer is stupid and you have to tell it how to see lol. Defining what constitutes spaghetti would be hard. Determining when filament exists to the left or right of where it should be is easier; you simply multiple the binary silhouette template against the image and above some threshold, there is a problem. This one simple concept can be used for print slipping, spaghetti, jamming, etc as in each case the image should not match the expected silhouette.

[nophead]

Spaghetti would be characterised by lots of curved edges only a filament width apart that cross each other and are not in a horizontal plane. Nothing correct, even abstract art, has that characteristic. I would imagine a neural net could be taught to recognise it if it was too hard for conventional CV.

[foosel]

I'll just leave this here: http://www.thingiverse.com/thing:240158

[thinkslynk]

"Spaghetti would be characterized by lots of curved edges only a filament width apart that cross each other and are not in a horizontal plane"

Define lots? How much curvature defines a curved edge? Define "filament width" as there are commonly 2 different filament widths used. How do you compensate for the relative filament width changing by camera perspective warping? Now you're talking about doing intersection checks on every edge we find, which are curved (expensive.)

"Nothing correct, even abstract art, has that characteristic."

^ See above.

"I would imagine a neural net could be taught to recognize it if it was too hard for conventional CV."

If you'd like to use traditional teaching methods to create cv templates, go for it. I have my doubts that you could create a template for something that is seemingly random in shape, size, and distribution. _shrug_

[guysoft]

Hey guys,
So I have an alternative solution to the problem, one that would use machine learning.
I actually have the backend code 95% written in another project.

The Idea:

1. Collect from users, using a plugin, images of failed prints and successful prints.
2. Put them in a neural network I wrote (there is a "train" script)
3. Get a neural network that returns a number between 0 to 1 if the camera sees an image of a failing print (assuming the network will learn how spaghetti looks).

Code:
This is code from a hackathon I wrote with the help of a few others, that detects "doors" and "not doors". Just treat "doors" as successful prints, and "not doors" as failed prints.
https://github.com/guysoft/thedoorks

Advantage:
no need to do complicated silhouette traces that look different in every camera, just have users send images and append them to the cases.
You could add test cases like those flowers above and the network will learn, there are just too many edge cases to code this.

Con:
We will need to collect around tens of thousands of images (given the community size this is doable). Preferably from most users using the plugin.

Note: might detect shift if spaghetti is formed.

Solving CPU/GPU issues:
The training can be done on a strong machine, but its a one-time thing.
Running the network might need a slightly stronger machine, but we should be able to run it on the cloud, I think I can get a machine for that, where you send an image once every 5 mins or so and it gives a status.

I am super busy and never really wrote an OctoPrint plugin, If someone wants to build a plugin, I can help you set up the machine-learning side of things, all I need is someone that could do the web interface part, and initially make a test dataset for only their printer.

[thinkslynk]

You're free to try it. But good luck getting images from users of prints in the act of failing, rather than several minutes after the failure when print is a giant garbled mess. It's useless to train a neural network to detect that a print failed after the fact. Good luck getting images from users all from the same perspective, same image quality, same lighting, etc, and generalizing the model to accommodate any print. I personally wouldn't use a plugin that would cancel my print half way through because it doesn't generalize.

Advantage:
no need to do complicated silhouette traces that look different in every camera, just have users send images and append them to the cases.

Just have a perspective alignment function that prints out a small model to determine perspective. Problem solved.

Con:
We will need to collect around tens of thousands of images (given the community size this is doable). Preferably from most users using the plugin.

To accommodate for all perspectives, lighting, model variations, camera quality, printer configuration, etc. You'll need a lot more than tens of thousands of images, which btw you'll have to retrieve by users intentionally failing their prints. I trained a neural network for my job, a model in use and in production, to do OCR using a cellphone for a single predictable target. We have 150+ images per phone, 5 phones officially supported, all split into 5+ images using heuristic over-segmentation and proprietary image processing techniques to binarize the images. Leaving us with over 4000 hand labeled clips to train our network on that are only 16x16 black and white images. Not a lot of possible variation at that resolution and color profile.

You, however, will need to train a neural network on 640 x 480, or split the image into smaller chunks, and find a way to binarize the image that takes into account that the filament may match the color of anything in the background (red matching the print bed, black matching the support, white matching a wall, etc).

Again, you're free to try, but as I said further up in the comments, a neural network would be very difficult for this application. My proposed technique would be much simpler and predictable.

[guysoft]

You're free to try it

I DONT have time for this, I have a company to run, and I am also already maintaining OctoPi. I can volunteer time to explain someone how to do the backhand, I need a GUI person for the data collection and test function (I can do the training).

You'll need a lot more than tens of thousands of images

OctoPi has over 150K downloads by now. I think if we can get a few people to tag images it should at least work for their prints. It would be an indicator to people "what your printer supported? Spend an hour and tag photos!"

To accommodate for all perspectives, lighting, model variations, camera quality, printer configuration, etc. You'll need a lot more than tens of thousands of images, which btw you'll have to retrieve by users intentionally failing their prints.
You, however, will need to train a neural network on 640 x 480

I think scaling to 200x200 would be good enough
Even if I am off by two orders of magnitude its doable. And we will see if it improves with more examples. And all the arguments about edge cases I think it would only be easier to train than code all of the light, translation and shape problems.

I trained a neural network for my job, [...] OCR using a cellphone for a single predictable target

Cool. We've been working on OCR for construction plans (of buildings). And OCR for plans is an open end problem, ABBYY which we tried failed to do this in many cases (we are still working on it with them), if you wanna talk about that, it might be relevant :)

One more thing - I ALREADY have code for this network. You are welcome to find someone to spend the pain of doing computer vision for this.
With quite a few data scientists around, at least in Tel Aviv. They are just waiting to try and build something like this.

[thinkslynk]

And unfortunately I'm too busy leading my company's machine learning initiative, or I'd just write my originally proposed solution. I'd be happy to help talk someone through the process if they'd like to take it on. Wouldn't require collecting thousands of images from hundreds of users to get started.

First simple step for anyone who comes across this, is to just write a plugin than returns a composite of what you think the printed model should look like on top of a camera still. That's honestly like 80% of the work.

[thinkslynk]

@guysoft In response to your edited post. Having 500 lines of code is unhelpful. I'm going to trust that you do machine learning for a living as well, and as such you should know that setting up a keras model is trivial; the one I'm working on right now to do time series predictions is seriously like 200 lines of code. Scrubbing data, ensuring your model is the proper size/structure, doing hyper parameter optimization, avoiding overfitting, etc, this is the heavy lifting involved for any machine learning project. Garbage in garbage out right? You can have users label their own images and upload them to a server you set up, but you'll get cell phone pics, web cam stills, dumbasses uploading memes, etc. You, or whom ever takes on the project, will have to validate all of those images by hand. Nothing will destroy a predictive model faster than garbage data.

I'm also a bit cautious when someone says "use my code that trains a model sized for 50x50 images for your 200x200 images." With a change in input size like that, you'll inevitably need to change the structure of your model. It's true, you've given someone a starting point, but if they don't know how to reshape your model for 200x200 images, the code is useless.

I'm probably coming off like an asshole right now, I just think you're oversimplifying the work needed to train a neural network to detect a non-deterministic feature. You can easily detect the features that make up a door. Round door knob, solid parallel lines, etc. There aren't too many different types of doors so a neural network works great for detecting such features. A garbled print is a bit more random. But again, haven't tried it. I could be wrong, I just have some strong doubts.

[guysoft]

Ok, thats a more detailed explanation. Now I understand why you say it might be difficult.
Sorry if I was oversimplifying myself.
I guess since we are both busy this issue will have to sit here until someone brave hero would attempt this.

[LeonFedotov]

lets do this!
I also thought of this for some time and I have some free time :)
@thinkslynk I like your idea, having played a little with opencv I understand what you are saying about the challenges, I think @dm3942 idea with the square that you print next to the print is perfect for calibrating both the angle and the filament color.
solving figuring out background problem might be solved by stereoscopic photography to figure out depth, and perhaps we could even go further and switch to IR all together?
I have a kinect I can play with i think it might produce some interesting results.
did you start working on something i can join?
I'm comfortable with most scripting languages and some low level as well.

@gddeen spectacular idea! really I love it - very complex to get right but for the right buck and and an industrial complex behind you this would be definitely doable, I liked the potato chip analogy and it reminded me of another one about the washing machine. It's design differs from industrial to home use, the industrial washer has 3 separate containers which are constantly monitored for weight and water is pumped in/out to equalize the weight of the drum to reduce vibrations, where in the home washers they just stick a slab of concrete and call it a day, both solutions are good for its own reason and use type, having played with computer vision I tend to agree with @thinkslynk about simplicity. and simplifying the problem and expectations could produce significant progress in a specific field, it might not get it right 100% of the time but, it might be good enough, I hate the feeling knowing that your printer had just been producing spaghetti for 3 hours, if i can reduce this even by 20% within a couple of weeks its already progress.

@guysoft the network idea might also work well - it might require a specialized printer to reduce the amount of training required.

[thinkslynk]

@LeonFedotov I have not started anything. But if you need any guidance feel free to reach out.

[udif]

Here is an alternative idea:
Some of the problems I run into are non-sticking models. How about tracking the pixels with the filament color (assuming it's different than the color used for your extruders' parts), and alerting the user when pixels which previously contained that color now have a different color, possibly indicating that your print has been detached from the print surface.
This is much simpler than the other ideas, and at least solves SOME of the problems people run into.

UPDATE: Upon further thinking this is not obviously going to work on any printer with a moving bed (unless you attach the camera to the bed, which you really don't want to, and even if you did, then you might get false readings due to vibrations).
A more complex variation might work if you correlate the position of the print to the image of the bed surface, for example, measuring the relative position of the part to the bed. Find a pixel with the filament color that is closest to the bed edge, then measure its relative position to the bed right and left edges. This is not too hard (easier if you locate the camera at specific places such as pointing exactly to the center of the bed. If your camera is located on the left or right side of the printer, you need to capture the bed edge, and use this info to locate the relative position of the object.

[ChrisWeiss]

UPDATE: Upon further thinking this is not obviously going to work on any printer with a moving bed (unless you attach the camera to the bed, which you really don't want to, and even if you did, then you might get false readings due to vibrations).

One solution would be to only do the image capture when the bed stops/passes through a specific location (think of those timelapses where the print head just seems to be moving vertically). Alternatively, you could dynamically update the comparison POV every few frames. That'd be computationally expensive, but the Pi _does_ have a GPU. You should easily be able to pick out the edges of the print bed to set the new POV.

I'm more curious about how you deal with the non-facing surfaces. IE - the top of the print exposing infill the pattern. Could you render out each frame of progress dynamically? I've seen deskop slicers that'll do it - not sure about the Pi.

[darkin1]

For all the cases we need photos/videos of the prints. Optimal is about 1000 videos/images per class (good prints/bad prints). First of all, in octoprint or plugin should be a mechanism to store this data in the cloud. After that, we could use some of the ML/NN technics to teach the model to recognize some of the failures types (ex. spaghetti print, or blobs). Unfortunately, we are not able to do anything without data.

[guysoft]

1000 is probably underestimating.
Because prints that fail look differently

[redxeth]

I think something to just catch a blob forming on the print head would be a better use case. Print head should say relatively the same change/image throughout a print. You have a mode where you teach it-- then let the print go and have it stop autmatically if too much filament starts appearing on top of the nozzle.

[jneilliii]

This plugin was added recently that appears to be promising...

https://plugins.octoprint.org/plugins/thespaghettidetective_beta/

[darkin1]

There is another promising tool https://www.savemyprint.com/

[guysoft]

There is another promising tool https://www.savemyprint.com/

Not Open Source by far :-/

[LMS0815]

The Spaghetti Detective: https://plugins.octoprint.org/plugins/thespaghettidetective/

[maxupp]

I'd love to take a crack at this, but based on computer vision and image classification.

My background is in AI and I have built a multitude of object detection and image classification models. The only thing hindering me from doing this is the lack of a dataset. This dataset would need to contain upwards of 1000 images of both failed and ongoing succesful prints to build a reliable model.

Is there a suitable dataset anywhere? If not, could we collect one through OctoPrint itself? ("Contribute to our next feature by uploading a few frames of your webcam feed.")

[foosel]

Just to make this clear, this is not a feature I'd be willing to merge into OctoPrint core and an absolute prime candidate for a plugin (hence also the plugin idea label on this request).

An initial dataset to start out with would indeed be the biggest challenge here. If anything you'd prolly need to create a plugin first that does this collection, make it very clear what it's meant for, and then try to get people to adopt it. Someone already did that, but AFAIK it hasn't led anywhere so far as far as I can see, probably also due to a lack of transparency and active presence in the community: https://plugins.octoprint.org/plugins/kronos_data_collector/. I suggest you get in touch with @MrBreadWater on this.

[maxupp]

Unfortunately I don't see this happening in an opt-in kind of way. The barrier of installing a plugin and enabling data collection is pretty high for the average 3D printing enthusiast.

[foosel]

Anything but opt-in goes completely against the GDPR though and thus is not something I'd touch with a ten foot pole myself considering that that could become bloody expensive for me personally.

[thinkslynk]

My background is in AI and I have built a multitude of object detection and image classification models. The only thing hindering me from doing this is the lack of a dataset. This dataset would need to contain upwards of 1000 images of both failed and ongoing succesful prints to build a reliable model.

I'd recommend reading through the discussion further up in this thread about why ML would likely be a poor candidate for this feature. But if that doesn't deter you, go for it.

[maxupp]

I read what you wrote, thinkslynk, I do however disagree strongly. You seem to come from classical computer vision, which I do appreciate. With modern object recognition methods there is no need to define what constitutes a certain object, the AI will find that out by itself. I also pose that a huge spaghetti nest is still a valuable data point to detect the beginnings of one forming, its just a matter of what size of bounding boxes you draw.

[thinkslynk]

I do not come from classical computer vision. I use LSTMs for time series predictions and a derivative of LeNet5 for our OCR product (which is a CNN.) I do not believe one can accurately determine whether an image contains spaghetti, or intentional model features with a neural network (see the thingverse model posted as an example.) I'd also pose that detecting "a huge spaghetti nest", as you put it, kinda misses the point; at the point that it's a huge mess, the value of automatic detection is basically pointless since you've already wasted a huge amount of filament.

[maxupp]

Hey man, you're of course entitled to your opinion! I've built models that detect bird shit on cars, defects in PCB etchings and a multitude of other stuff, I like to think I have a pretty good grasp on object recognition in general.

What I meant by the spaghetti nest comment is that you don't need imagees of these failures right when they happen. A small section of the aftermath doesn't differ that much from a smaller nest. I'd love to prove it, but if I had the data we wouldn't have this conversation ;)

[thinkslynk]

Detecting a white splotch on a car is dramatically different from determining whether a feature is spaghetti or an intentional piece of a model. The originally posted solution (the first post of the thread) is the best approach, in my opinion, and would result in the quickest shut off, with the least number of false positives. I only comment to prevent others from wasting their time on a solution that won't work, or would "work" but with a high false positive rate. A high false positive rate will ruin just as many prints as a low detection rate.

At best, to make ML work for this problem, you'll need to create the originally proposed silhouette to compare the currently printed model against anyway, at which point some extremely simple image processing will give you a better solution and requires no test data set.

To be clear, I'm a hired Machine Learning Engineer, and I support the use of ML for a lot of solutions, but I have little confidence in it being an applicable solution for this problem. You're like the 3rd person in this thread to say "what about ML" and this thread is 4 years old, so it might be safe to assume that others have already tried it at this point.

E.g.

https://plugins.octoprint.org/plugins/thespaghettidetective/
"The Spaghetti Detective is still at early stage so it doesn’t always get it right." - After a year of the project. They have a gallery of detection videos, look at how much wasted filament exists before they even detect an issue.

https://www.savemyprint.com/ - No longer exists, presumably gave up.

[maxupp]

Like I said, your opinion is your opinion. I've been working with machine vision systems for 6 years. In order to not escalate this into some kind of flame war I'll leave it that.

[thinkslynk]

It's not a flame war, I'm telling you it's already been attempted, despite my recommendations against it, and everything I said would happen did. I realize that'll come off conceted, but all you have to do is look at the spaghetti detective for proof:

https://www.thespaghettidetective.com/docs/failure-detection-false-alarms/

"The Detective has openly admitted that the most common mistakes she makes are false alarms (but secretly she thinks it's just because she works too hard). "

"Behind the scene The Spaghetti Detective is leveraging AI technology to detect print failures. Specifically, it is a Deep Learning algorithm that is trained to look for patterns of spaghetti shape."

What are the common false alarms?
Our users have reported quite a few things The Detective is likely to get wrong, including:

Bed clips.
Flower vase.
Extruder's reflection on the glass bed.
The beautiful logo on Ender 3's bed or other text.

You can say it's "my opinion" but I can point to an existing project that uses the same technique that you propose and has all of the issues I mentioned.

[MrBreadWater]

Just to make this clear, this is not a feature I'd be willing to merge into OctoPrint core and an absolute _prime candidate_ for a plugin (hence also the plugin idea label on this request).

An initial dataset to start out with would indeed be the biggest challenge here. If anything you'd prolly need to create a plugin first that does this collection, make it very clear what it's meant for, and then try to get people to adopt it. Someone already did that, but AFAIK it hasn't led anywhere so far as far as I can see, probably also due to a lack of transparency and active presence in the community: https://plugins.octoprint.org/plugins/kronos_data_collector/. I suggest you get in touch with @MrBreadWater on this.

This is my fault. I apologize for the radio silence, but I am currently working on the 'silhouette' version of this idea. I actually did train an AI quite similar to TSDs using the data I collected, but I just found it to be... well, rather awful, and certainly not very efficient. I'm quite surprised at the success TSD has seen in spite of having the same flaws.

After my initial failure with the AI version, I gave up on it for a year or two. But I'm back on it. I have an efficient GCODE renderer that corrects for camera perspective and lens distortion, and compares to the webcam view. It's still under development in a private repo, but initial tests are extremely promising, the one downside being that it requires you to place a QR Code somewhere on your printer along with some initial setup to calibrate the system. It does suffer from some issues with lighting on the print, but not nearly as much as TSD does.

I've not made much progress recently due to workload from my classes, but I'm very close to having an alpha version of the plugin. The main issue with the code right now is that it's slow, so I'm implementing every single line from scratch using Cython without the help of libraries. _Should_ be fast enough to let it run directly on a Pi. It's essentially what people in this thread have been proposing from the beginning, and now that I'm using pure CV rather than AI, it really isn't bad.

[MrBreadWater]

@maxupp @thinkslynk By the way, if you two want access to my dataset, it's extremely extensive. I have around 4.5k raw timelapses (not all of which I've extracted data from yet). I have 3.5k images with bounding box labels (print, failure, and print head), all of which are sorted by whether the frame is a failure or not. I did the bounding box labels myself, so they should be very accurate.

Including images without bounding box labels, I have ~5k images sorted by pass/fail, however only 850 of those are failures. I also generated some Synthetic Data to increase the number of failure counts, but it was fairly specific to my approach.

Anyways, I'd be happy to try and work something out. As @foosel pointed out though, we could potentially violate data privacy laws, so we would likely need to look into that.

[thinkslynk]

After my initial failure with the AI version, I gave up on it for a year or two. But I'm back on it. I have an efficient GCODE renderer that corrects for camera perspective and lens distortion, and compares to the webcam view. It's still under development in a private repo, but initial tests are extremely promising, the one downside being that it requires you to place a QR Code somewhere on your printer along with some initial setup to calibrate the system. It does suffer from some issues with lighting on the print, but not nearly as much as TSD does.

I've not made much progress recently due to workload from my classes, but I'm very close to having an alpha version of the plugin. The main issue with the code right now is that it's slow, so I'm implementing every single line from scratch using Cython without the help of libraries. _Should_ be fast enough to let it run directly on a Pi. It's essentially what people in this thread have been proposing from the beginning, and now that I'm using pure CV rather than AI, it really isn't bad.

Nice! Glad you're exploring it!

As far as lighting issues, I'm not sure if you're binarizing the image to create the silhouette or not, but if so there's a few things you can try to help with the lighting. I'd recommend using relative luminance rather than a standard grey scale squash. I'd also recommend using a post processing algorithm such as local adaptive thresholding to accentuate solids (there are probably a lot of algorithms you can swap in and play around with.)

As far as speed, if you get something working, you can always re-write it later in something faster. Even 1 frame per second is fast enough if your accuracy is good; one second of wasted filament is totally a reasonable goal.

[MrBreadWater]

Nice! Glad you're exploring it!

As far as lighting issues, I'm not sure if you're binarizing the image to create the silhouette or not, but if so there's a few things you can try to help with the lighting. I'd recommend using relative luminance rather than a standard grey scale squash. I'd also recommend using a post processing algorithm such as local adaptive thresholding to accentuate solids (there are probably a lot of algorithms you can swap in and play around with.)

As far as speed, if you get something working, you can always re-write it later in something faster. Even 1 frame per second is fast enough if your accuracy is good; one second of wasted filament is totally a reasonable goal.

Thanks for your suggestions! I'll have to look into relative luminosity, looks like it could be useful. For the silhouette, I'm using an image segmentation algorithm based on Markov Random Fields. This wouldn't work too well on its own but since we have the GCODE render as a prior, it works well. Binarizing would be much faster though, and I'll have to explore it more, but the upside to MRFs in my opinion is that it's very unlikely to give a false negative whereas binarizing runs that risk a bit more depending on filament color and background.