Octoprint: [Request] display long filenames from SD card - use M33 to get long filenames from printer

[doughadfield]

For printers that support returning the full filename of SD card files (Marlin) via the M33 command, please can the full filename be displayed in the Octoprint file list. Octoprint uses the M20 command to return a list of files in the SD card directory. These are returned in Microsoft short form, and are usually almost completely un-human-readable. The M33 command can then be used to obtain the full filename, but this needs to be iterated for each file name separately. Please can Octoprint be enhanced to do this after issuing the M20 command, and display the full filenames on the web page.

Thank you
Doug

[foosel]

Sadly the M33 command is pretty much useless for that. The problem is that I'd have to call it once for every single file (including those in sub folders and sub sub folders...) on the SD. Imagine a SD card with only 20 files and you are already looking at 21 commands to be able to populate the file list. That simply doesn't scale at all.

There was a discussion a while back on the marlin firmware bug tracker where it was talked about enhancing the M20 output to directly include the long file names in the response but I think that didn't end in any firmware implementation.

Additionally the long file names are not guaranteed to be ASCII, whereas the communication over serial is actually supposed to be only ASCII (even though a lot of firmware variants out there simply don't care). So the long file names would also need to be transported in some encoded format.

I'd be happy to add long file names from the sd to OctoPrint once there is an efficient way to retrieve that kind of information. But sorry I refuse to implement (or merge) a solution utilising M33, which would effectively nuke the whole printer communication for an indeterminate amount of time (depending on the number of files on the card) every time the file list needs to be refreshed. This will just lead to people opening tickets because "their printer locked up" because they had hundreds of files on the attached sd card.

[Richie76]

would it be possible to get the long filename just for the one file printed ? so it can be used for the filename of the timelapse ?

And sorting of the sd-card files in newst first order would also be quite helpful (at least for me: I´m printing from and flashair sdcard and using octoprint only for controlling and timelapse due to the slow serial interface of my printer and the artifacts this cuases).

tia, rgds, richard

[drdelaney]

What about getting the long filename for the file currently loaded / printing file?

[Andrei-Pozolotin]

perhaps M33 could be made salable by lazy running it in background along with periodic M105 polls
this way, you show short names first, and long names will "eventually" show up

[MStylesMS]

The non-readable file names on the SD card are problematic, but I also understand the implementation problem. I think background processing when the serial connection is idle is a sufficient approach (with caching of the names in Octoprint of course). However, the ASCII vs non-compliant filenames is more tricky. If a was around this (or firmware updates to a better standard) ever happen this would be very useful.

[feldon30]

A local cache would do quite nicely here. Cache would be a text file or sqlite with each 8 char filename + timestamp and then the full filename. Expire every cache record after 30 or 60 days.

[lamberaw]

Bump, I understand there may be some technical issue implementing a full file name support. But, when a local display (LCD) on a printer running Marlin can handle it, the RaspberryPi should also be able to handle it. Additionally, if a file name is less than 8 characters OctoPrint still shows the tilde number. I need to test it, but I assume because in the case of case of abcde.gcode) 5.5, is greater than x.3.

Anyways, kudos in general to the team for OctoPrint, as I just started using it and do like it.

[drdelaney]

I would like to also point out, that as of Marlin BugFix 1.1.x (or what will be 1.1.9), a new parameter C was added to M27, which should show the current file being printed via SD.
https://github.com/MarlinFirmware/Marlin/commit/a560c0f7cd8f5c02e35e309926530b07442542d8 shows the addition. Perhaps this can be added to the polling mechanism to display the long filename.

[koenkooi]

@feldon30 Toshiba Flashair cards seems to change the modification date of half the files on the card when uploading a single new file, so a cache wouldn't help much in that case.

@lamberaw as pointed out before, it's not a matter of processing power, the printer firmwares all lack a 'give me a list of all filenames, don't do 8.3 conversion' command, you can only get the long filenames one by one by asking for each 8.3 entry what the long one is.

[foosel]

@drdelaney that's nice for Marlin users, but OctoPrint needs to support a multitude of printer firmwares (including vendor forks of Marlin which will probably never see that change) and sending such a parameter with M27 C calls blindly might confuse those printers and cause horrible issues there. Or it might not. But I have to err on the side of caution here.

@lamberaw what @koenkooi said. As repeatedly explained in this and other issues and topics on this, the problem here is not the processing power of the system OctoPrint runs on, it's that the only way to retrieve the long names right now would saturate the serial connection completely for a duration of time expanding with the number of files on the card. And on top of that we have an encoding issue on our hand since according to the only thing we have in terms of a "spec" (a wiki page on the reprap wiki) the communication over the serial line is supposed to fit into ASCII and long file names don't necessarily.

But everyone is welcome to send a PR to open up file metadata collection from the SD to plugins and then implement a plugin which sends an M33 for each and every SD file.

[firelizzard18]

@foosel Can you give me some pointers for getting started on a PR like that?

I understand and agree that requesting long names for every file isn't scalable. What I would like to see is a request for a long file name when I select a file.

[ringerc]

If Marlin (or ADVI3++) could report an abbreviated long file name with 7-bit flattening, that'd be a usability help I think. But then the printer firmware would have to recognise those file names and map them to the real local file names when it went to print the requested file, and the abbreviation + flattening means there's no guarantee they'll be unique. So it'd have to send a mapping ... in which case we're back to issue of sending the 8.3 and the LFN since there's no point abbreviating anymore.

So I doubt Marlin and forks will accept that.

Disabling long file names on FAT32 isn't too practical since the 8.3 format isn't what you'd call informative. Plus I don't know if there's even any FS option to disable it or if any platforms honour it, so it's a nonstarter really.

To be honest I'd be all for the M33 iteration when the serial connection is idle except for temp polling, with a delay-loop so other work gets time on the serial link too. And perhaps a local cache.

But even with that, and the resulting slow refresh where the filenames would start out as 8.3 and be progressively re-rendered as LFNs, the text encoding issue is a serious problem.

It sounds like it's not an 8-bit-clean serial link and I'm guessing we can't hope the M33 command does anything sensible like specify a 7-bit encoding (base85 etc) of the filename. So we have no way to know what text encoding the files are in. Octoprint could have a config option to tell it what the usual text encoding files on the printer's SD card should have, but FAT32 is an awful prehistoric filesystem that lacks any way to record a native encoding, so it's always going to be a best-guess. Plus, how many people do you know who could tell you at a glance if their computer will write filenames as UTF-8, ISO-8859-1, ISO-8859-15, windows codepage 28605, ibm codepage 923 (hint: the last 3 are all the same thing), windows codepage 1252, etc. And that's just for English speaking systems! The reality is that octoprint is pretty much going to be stuck with some icky encoding heuristics and/or flattening things... which is workable albeit ugly if you're French or Italian, but completely nonfunctional if you're Japanese, Chinese, Korean, etc.

Then we have our friendly Shift-JIS encoding to throw further insane chaos in the works.

So @foosel I totally get that this is far from the simple issue it might look like at face value.

I've done a fair bit of work with text encodings and I'd be happy to help with that side of things if you're interested in exploring this. I won't currently be able to start on a feature from scratch myself (young kids) but would be very happy to be involved with testing, review, advice, some development, creating torture test sd card images, etc. I'm decent with Python, C, C++ and the Python/C API and I work on Linux as my main daily platform.

[foosel]

I'm still very much opposed to the n+1 M33 approach. If there was a way to get the long file names via M20 (as I also proposed at least once already) I'd be happy to take another look at the encoding situation, but n+1 M33 will not happen in core.

[radex]

@foosel I've been playing around with Marlin, and it seems pretty trivial to me to add long file name to M20 output. I could send a PR there to add that as an option.

What output would you prefer for M20? Right now we have dos-name size. Would dos-name size full-name be a good format? Marlin still requires dos-name to be sent as an argument to other commands (like starting a print), and they seem to be resistant towards allowing full names as an argument: https://github.com/MarlinFirmware/Marlin/issues/11403

[foosel]

I found the discussion again that I mentioned here and elsewhere that took place on a Marlin ticket a couple years back on inclusion of the long names in the M20 output:

https://github.com/MarlinFirmware/Marlin/issues/4733#issuecomment-243691051

tldr:

Begin file list
UGLYNA~1.GCO 2342 Uglyname with some_longer-name.gcode
End file list

would be best. Still leaves the encoding issue. We could just assume UTF-8 for reading the long names, but that feels kinda meh.

[radex]

Still leaves the encoding issue. We could just assume UTF-8 for reading the long names, but that feels kinda meh.

Oh. I missed that part when reading that issue yesterday :/ I don't know enough about marlin code and serial port comms to figure this out myself, but I think I'll try to make a POC PR and maybe Marlin maintainers will have some guidance as for what to do with the encoding problem

[jep1955]

Although I understand the comments about saturation and M20 limitations as well as "which Marlin do I support" type thing, I also understand, and am exceptionally frustrated as well, by the 8.3 limitation. That went out with DOS 25 years ago. It makes it exceptionally difficult to name files in any kind of meaningful way for selection. Good example. I have multiple copies of Venus de Milo.

Venus de Milo 8" fine resolution
Venus de Milo 8" standard resolution
Venus de Milo 12" fine resolution
Venus de Milo 12" standard resolution

How the heck do I name those files (let alone the others on the SD card) in such a way as I can clear see not only the model I am opening, but the size and resolution variant I am opening?

I agree with an earlier suggestion - what if I had an option for displaying full filename for the "current" file?

Or, my choice? I've been warned that I can saturate the serial line with data for a time. But give me a button that allows me to click it and create a local copy of full filenames at my discretion?

I'm also a bit puzzled about the saturation thing anyway. Raw directory (filenames) are flat text. Even figuring an absurdly low speed USB (serial) connection of 115K, that implies I should be able to transfer hundreds of 50 character filenames in < 1 second. Who is actually the limiting factor here? (I don't see it in the history). The controller firmware/hardware? The USB 2.0 (at LEAST) connection? The Raspberry PI?

[draeath]

I'd say it's a firmware-side issue. There should be a flag for the M20 command that spits out the long filenames. The firmware should be handling the iteration.

I've found some other silly fimrware level issues with the SD card access as well, like prusa3d/Prusa-Firmware/pull/1739 - using creation time instead of lastwritetime for sorting for example.

I think a bunch of this is related to the Arduino FAT library this is all built on. It seems... hokey.

[draeath]

I also think the encoding issues would be a non-issue if the data could be BASE64 or uuencoded.

These are issues we solved way back in the day, when we used modems via serial ports. How this is still an issue these days...

[DavidUnboxed]

Thinking through this problem I wonder if it might be possible to mitigate this by:

1. leveraging a database in OctoPrint containing 8.3 friendly primary keys and long names
2. having OctopPrint use the primary key value when writing or reading from the card
3. having OctoPrint always display the long name alongside the primary key

I cannot help but wonder if the character flow rate is so impactful with long names if there may be a performance gain by switching to only leveraging 8.3 or even much smaller primary keys in this manner.

Of course, this would mean no more friendly names in the display if the entire 8.3 space was used but perhaps that would be an acceptable option as a 'we are protecting your privacy in multi-user environments'. And - not having to wait to watch for printer LCDs to scroll - may be time saved even while having to look at my job's 'ID' when published via Telegram integration, etc. :)

Or, perhaps coexistence is possible if not all of the 8.3 space is consumed by the key but rather all managed by OctoPrint use a leading character?

Or, worked around by allowing a browser plugin to see the SD card when loaded on a client and update the database?

[Haprog]

This makes me sad.

I would be glad even if this was implemented as an opt-in feature with idle polling and caching and with a note that if you enable it you should make sure you only use ASCII characters in your file names (if it makes things easier). I only use ASCII characters anyway on my printer SD cards (and most any files anyway).

[Haprog]

I use an Original Prusa i3 MK3S and I would even consider compiling my own firmware with a small patch to make this work better (if it can be helped from firmware side).

[feldon30]

I am seeing a recurring development trap here. It can be boiled down to "we can't make it perfect, so we shouldn't do it at all." Development effort ought to be weighed based on quality of life for the user, effort required, future compatibility, whether a forthcoming change is expected elsewhere that would render development time moot.

I know of no changes announced or planned that would make this any easier or more reliable. Therefore, waiting for improvements is not realistic.

Adding this functionality would provide a tremendous quality-of-life. 8 character filenames are a thing of the past and not something most users want to mess with trying to make unique enough to identify at a glance in a file listing.

Development effort to support this are reasonable for the QoL it would return for the user.

Octoprint is able to read long file names, file size, and modification date. That should be enough information to maintain the unique relationship to the short filename between reboots. Is it conceivable that in some rare instance two files might match up perfectly? Yes. Is that exceedingly rare edge case enough reason to kill the whole idea? I think not.

And when selecting a file to print, one final M20 can be done to confirm that the filename is in fact correct. That would eliminate any chance of printing the wrong file unless the user is very inattentive. A javascript message could even be thrown to warn the user that the filename changed.

[foosel]

To quote myself:

But everyone is welcome to send a PR to open up file metadata collection from the SD to plugins and then implement a plugin which sends an M33 for each and every SD file.

Pull requests doing this are welcome. If this is such an important quality of life improvement, then please, someone finally go ahead and do the above, this is an Open Source project after all.

[gruvin]

IMO, this is a Marlin (et al) issue. M20 should output _filenames_, not prehistoric 8.3 abominations from a bygone error (pun intended.) Marlin is an open source project as well, after all. ;-)

[Haprog]

@gruvin I think it would be a breaking change for Marlin to change the behaviour of that M20 (since it is explicitly documented as: "List the entire contents of the SD card to serial output in the more compact DOS 8.3 format"). It would probably make more sense for Marlin to introduce another new gcode (or an additional parameter for M20) to enable easier listing of all file names from SD in long format (and probably in a specific encoding like UTF-8) to make it possible to do in a backwards compatible way.

But even if that was implemented in Marlin right now (e.g. in current development version), it would take a long time to get a new version of Marlin available for many printers and some printers might never get an official firmware update with the new feature/fix, so it would be much more useful in the short term if this can be worked around (e.g. using that M33) to have an improved user experience in OctoPrint soon which would immediately work with a lot of printers (not requiring a firmware update to the printers).

[firelizzard18]

Pull requests doing this are welcome. If this is such an important quality of life improvement, then please, someone finally go ahead and do the above, this is an Open Source project after all.

@foosel For those of us who are interested in working on this but are not familiar with the code, would you point out where this could be implemented?

[Haprog]

I've also been thinking about wanting to contribute a bit of code maybe to OctoPrint or some plugins, but I stopped using SD cards pretty soon after I started using OctoPrint so this particular issue doesn't annoy me anymore (usually I'm just sending my gcode directly from PrusaSlicer to OctoPrint nowadays).

[mikeshub]

@foosel perhaps there is some low hanging fruit in between listing the correct file names and being stuck with an obsolete format because of the wonky way that Marlin offloads the data. I certainly can't blame you for digging your heels in and refusing the kludge. Anyway, what if clicking on a file name in the file list sent the request for the file name that was clicked on.

This should scratch a few itches. For users it is a would be a nice compromise that would save any hassles with similarly named files. From a programming point it should be a lot easier to implement and trivial to maintain. No large list to scan through and no sending M33 repeatedly.

Considering as how about once a year the idea of fixing file names comes up at Marlin and then dies I would assume it is never going to happen unless someone just decided to code it and make the PR. As I don't care all that much it sure isn't going to be me.

Nice work on this project as well. TYVM keep up the good work.

[Tr0llmops]

Anyway, what if clicking on a file name in the file list sent the request for the file name that was clicked on.

This! It wouldn't even have to happen automatically when clicking on the file name, it could be linked to the 'chevron down' button which currently seems to be disabled for SD files. This would still be far from perfect, but at least the user could make sure that the right file is selected before starting a print. It also wouldn't break important functionality for unsupported firmwares.

Even if that's not going to happen in core, I think it might make a nice little plugin.

Thank you for the hard work you put into OctoPrint!

[foosel]

The above commit will allow OctoPrint 1.4.2 and later to parse extended M20 reports that also contain the long name out of the box:

Begin file list
UGLYNA~1.GCO 2342 Uglyname with some_longer-name.gcode
End file list

As to my knowledge no firmware currently implements M20 with long file names like this, however, as proposed by myself back in 2016 that would make the most sense from a host perspective (contrary to M33, which requires n+1 requests to get all metadata from the printer), so OctoPrint now supports it and firmwares are welcome to implement it accordingly to finally implement a usable way to retrieve long file names that doesn't saturate the serial connection for an undeterminable amount of time just to retrieve a list of file names.

Additionally the above commit allows OctoPrint 1.4.2 and later to parse the response to M33 (which btw turns out is decidedly parser unfriendly due to the nature of the involved protocol, firmware bugs out there and out-of-band communication due to auto reporting, so there might be corner cases where parsing fails). It will however not send M33 on its own. There's no capability flag for this as far as documented, and there appears to exist a GCODE clash according to https://reprap.org/wiki/G-code, so just blindly sending M33 to the printer from core might cause severe issues.

Users may instead add active querying for the long name (if not included in the M20 response, which again, is the preferable and only clean way to do this) through a plugin, as demonstrated in this single file plugin (place in ~/.octoprint/plugins or upload through the plugin manager on OctoPrint 1.4.2 and later, earlier versions won't work).

And I know I'm repeating myself here, but again, the clean way is to implement this in M20, on the firmware side. The support for that will be there with OctoPrint 1.4.2 (and yes, I know, 1.4.1 is still in RC phase and not even out yet, so yep, it'll be a few months until a stable release of OctoPrint with this functionality lands).

That's it from my side on this topic, back to more pressing issues.

[foosel]

Make that 1.4.3, 1.4.2 just got turned into a hotfix release.

[foosel]

Actually make that 1.5.0 because I've decided to finally fully adhere to semver and maintenance releases will see a minor increase in the future instead of patch, patch will be used for hotfixes if needed.

[Haprog]

We call bugfix releases maintenance releases (when only patch version changes). But of course if you add any new functionality, then it should be at least minor version bump according to semver.

[bitboy85]

Suggestions for the use of M33.
I agree that requesting the long filename of every single file might lead to slowness and hang up but just showing the 8.3 filenames isn't user friendly too.
So one method could be that the long filename is requested if the user clicks on it. Of course this is not a 100% clean implementation but the user has at least the chance to get the information.

Use some sort of cache or database as suggested before. This will reduce the M33 requests to the number of added files since the last scan.

[foosel]

As mentioned above, OctoPrint 1.5.x will support reading the long file name from an extended M20. Alternatively it will also support enhancing the file name from a plugin. From here on out this is on firmware to implement in M20 as already sketched out years ago. Nothing but the above will be added to core OctoPrint and this is my final word on that.

Sorry, but I've really had it with this topic. There's way more pressing matters to take care of in OctoPrint, and having to constantly have this kind of discussion and explain again and again what would be the clean solution in firmware, that it has been discussed in the past and why M33 is simply the wrong way eats a TON of valuable time that is then missing elsewhere.

Tldr: 1.5.x will support reading the long filename from M20 if it's included, or add it from a third party plugin, and that's it. This ticket will be closed after release of 1.5.0. It's now the firmware's turn (and not just mainline Marlin's btw). Stop asking for solutions in OctoPrint that won't scale or are really just stopgap workarounds for a missing/incomplete implementation in existing firmwares.