Octoprint: [Request] Also scan and process watched folder on startup

[VibroAxe]

What were you doing?

Starting octoprint with files already in the watched folder

Ideally provide exact steps to follow in order to reproduce your problem:

1. Shutdown octoprint
2. Upload gcode file to octoprint watched directory (samba, scp etc)
3. Start octoprint
4. New file (Bug_Patch_1x1_Supports.gcode in the logs below) is not found / imported

If you encountered a problem with specific files of any sorts, make sure
to also include a link to a file with which to reproduce the problem.]

What did you expect to happen?

Octoprint would import all files in the watched folder on startup

What happened instead?

Octoprint ignores existing files in the watched folder on startup

Did the same happen when running OctoPrint in safe mode?

Yes

Version of OctoPrint

1.3.6

Operating System running OctoPrint

Ubuntu 17.10

Printer model & used firmware incl. version

Robo3D, not relevant to bug

Browser and Version of Browser, Operating System running Browser

Firefox and chrome

Link to octoprint.log

https://gist.github.com/VibroAxe/442bbbfdc1bc546e5abdbb826aa6e428

Link to contents of terminal tab or serial.log

Not serial related issue

Link to contents of Javascript console in the browser

Not js related/browser related issue

Screenshot(s)/video(s) showing the problem:

[If applicable. Always include if unsure or reporting UI issues.]

I have read the FAQ.

[ido-nl]

YES, yes, I wanted to open a ticket for this request, and here it is. Happy to see I am not the only one.

I use the watch folder functionality a lot, it saves me a whole manual step. I have my printer 2 floors up. When I slice somthing its on my laptop downstairs, I then put the gcode files on my NAS. I then go upstairs to turn the printer on.
When my PI (with octoprint) starts, the watched folder is mapped to my NAS folder.
This works perfectly when Octoprint is already running, but not when its being turned on. Then I have to add the files manually.

Possible solution?
I can imagine that the challange is, how does octoprint know what files in the watched folder are new after startup. When it's running you're probably using the startup time, or do you match by file comparing?

Perhaps you can see whats the newest file in the file folder and use that timestamp to see what files are new in the watched folder. Or store a timestamp after each upload.

[foosel]

It's actually removing any files put in there after successful "upload". The challenge thus would not be to identify new files added before startup (it would basically be anything in there). More tricky would be what to do with files that can't be processed. Right now I think it just leaves them in there, so that would pose a problem on next startup - it would keep trying to process files that have already failed in the past. Then again, maybe that is desirable?

[VibroAxe]

@foosel it depends if you think octoprint has authority over that directory. Couple of ideas:

1) Remember bad files
* when a file fails to import store the filename, and modified date in octoprint's db
* On startup if a file exists in the db, check it's modified time, if it's not changed ignore, else reprocess
2) Move bad files
* when a file fails to import move it into a "failed" directory inside watched

From the concept of reimporting failures though: The most likely failure to import is bad settings / permissions. At which point it trying again on next startup (someone fixed their issues and rebooted) could be very desirable. If it fails just ignore it again.

How "expensive" is importing, if we ended up with 30 large gcode files in watched, how much would this affect the startup time?

[ido-nl]

How "expensive" is importing, if we ended up with 30 large gcode files in watched, how much would this affect the startup time?

Would be the same as moving 30 files to the watched folder when Octoprint is already running.

I like the idea of having a failed folder, Or perhaps even easier, rename the file to .failed. I presume octoprint only processes .gcode files

[foosel]

Or perhaps even easier, rename the file to .failed. I presume octoprint only processes .gcode files

Not necessarily. OctoPrint allows extension of the supported files by plugins, so if a plugin added supported for a failed extension we'd be in trouble ;)

The idea with the folder on the other hand might collide with another feature request (or was it a forum thread?) where people asked for folder support in the watched folder that gets mapped to the uploads.

I'm leaning towards adding on an .failed extension probably being a fairly safe approach though, plugins may be able to add support for this extension, but why would someone do that. And I could even forbid it in the extension hook.

it depends if you think octoprint has authority over that directory

IMO it has.

How "expensive" is importing, if we ended up with 30 large gcode files in watched, how much would this affect the startup time?

Like @ido-nl already said, it boils down to the same overhead as we'd have with uploading the same files through the API (minus the network latency). It could be done in the background, so not block server startup, but it would still add overhead if there are a huge number of failed files in there that continuously fail to process. And trying to keep track of stuff through some form of external tracking DB or flat file in my experience is just asking for trouble considering how happily people manipulate stuff outside of the control of OctoPrint already and then complain about things not working as expected ;)

[VibroAxe]

I think from what you've just said, .failed seems like the best option. .octofailed or similar might be a more definite filename that would never reasonably used outside of octoprint.

I guess the only benefit of not blacklisting a .failed extension in the api would be that then someone in theory could create a plugin to either cleanup or retry failed files. That would obviously be a pretty advance use case though

[foosel]

Just pushed the above commit to maintenance (and soon devel) that will run an initial scan of the watched folder on startup.

Also any failed files will now get a suffix of .failed so they don't get scanned again.

Will be part of the 1.3.11 release.

[VibroAxe]

Awesome. Thanks @foosel

[ido-nl]

Gina, that is such great news, made my day! I have been wondering about this request for a while, wondering if it would ever make into octoprint. I have been really missing this feature, everytime I use my printer. So good to hear its going to be included.

If you want me to test, let me know. Or is the idea I update to the dev version.

[foosel]

@ido-nl the easiest way is probably going to be to just subscribe to the RC Maintenance release channel ;) Otherwise you can of course checkout the maintenance branch and run that.

I hope to get out a first RC for 1.3.11 early next week.

[ido-nl]

I have tested the happy flow, and its working perfectly. It saves me time and frustration. Now I can prepare my gcodes and put them my NAS. Than when I turn on my printer and Octoprint, the watch folder is mapped to my NAS. Now my pre-prepared gcodes get loaded into Octoprint automatically. I am very happy about this. Thank you for including this feature!

[foosel]

1.3.11 was released yesterday.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.