Manim: Folder Structure

[XorUnison]

Following the rework of constants.py (and associated files) the question of Folder Structure and dirs has come up.

For now the PR I finished has at least moved images out of the videos folder (because really, why would they be there), but there's some open questions.

There's two questions we have to briefly address.

First, how is the folder structure supposed to look like?
Here's I think the most current suggestion by @eulertour

  <module name>
    <scene name>
      svg
      images
      video
        <resolution>p<framerate>

I'm fine with that in principle, but have some reservations. In practical application this could create quite a bit of code overhead if the DIR variables aren't broken down to one. The reason for that is if we go with this folder structure, but also have a separately defined, say for example, VIDEO_DIR, then we need separate code to move stuff around for it to look like this, lest we drop module/scene:

  <module name>
    <scene name>
      <resolution>p<framerate>

With respect to that something like this:

  svg
  images
    <module name>
      <scene name>
        <resolution>p
  video
    <module name>
      <scene name>
        <resolution>p<framerate>

Is generally a bit more consistent in behavior, whatever the DIR variables, and the code would be simpler too.

And while we could also split images by resolution like that, and I can think of some cases where that would be helpful, do we actually want to? I reckon that with the "save all images" options it definitely would be better, because without being split by resolution that could turn into a real serious mess for a user, but eh. I'm not partial to either.

Second, what are our DIR variables going to be?
Currently we have:

VIDEO_DIR
TEX_DIR
TEXT_DIR

As @eulertour suggested and I thought of too TEX and TEXT should probably be merged into SVG. Them having to be split would be a real fringe case, and even then the cli output should be enough to do so.
What about the others? Do we remove VIDEO and put it into whatever MEDIA is by default? For that matter do we just remove TEX/TEXT and put that there too, leaving only MEDIA?
Or do we go in the other direction and instead also add IMAGE to place images separately?

My suggestion would be the second folder structure like that, with a resolution folder for images, TEX/TEXT put together and add IMAGE.

Any ideas or opinions?

[PgBiel]

I'm partial to

media
  svg
  images
    <module name>
      <scene name>
        <resolution>p
  video
    <module name>
      <scene name>
        <resolution>p<framerate>

[leotrs]

I don't have a strong opinion about the full folder structure when it's needed.

But can we also provide an option for manim to just output the one file? I think specially for beginners, getting a whole folder structure as output is unnecessary and even confusing. Sometimes I'm just trying out something new and I need a 3s mp4, not a host of files and folders.

[kilacoda]

I have a suggestion. Can we output the videos and other outputs to the media folder in the source file's current location?Like:

<source_file>.py
media
    <source_file>
        <1440p>
            <...files...>

We could add a --use-global-media-dir option to use the media folder as earlier.

[PgBiel]

~imo global media dir should be the default and you should opt-in to a media file in the cd~
edit: see below

[eulertour]

@PgBiel That could make it tricky to view a video you rendered previously, at least without rendering it again.

[PgBiel]

how so?

[PgBiel]

update: thinking better about this, if manim is installed as a pip package then it makes more sense for the default to be curr dir yea

[eulertour]

I'm not sure if you still have the question or not, but if you run manim from a far off directory, we write the video to a fixed location, and you later want to access the video you made, you'll have to reference the fixed location with a long path.

[PgBiel]

gotcha

[XorUnison]

curr dir is also currently default, which can be overwritten via dirs.py (or some future config thingy) or cli commands, with ascending priority.

We probably could add a command for singular file dumping, but is it worth the hassle? Ultimately beginners will have to look up and understand that command too, which strikes as not really much better than a folder structure. I think the assumption has to be that people do more with manim than render a single 3s video once in their lifetime if they already bother with it, so this cloaked short-term convenience could even lead to long term messy practices and anti-convenience.

I have a suggestion. Can we output the videos and other outputs to the media folder in the source file's current location?Like:

If we add any command in that direction like this, then a command to do this strikes me as much better. I could see how someone who usually outputs at curr dir or a fixed location might occasionally want something dumped at project location.

[leotrs]

I think the assumption has to be that people do more with manim than render a single 3s video once in their lifetime if they already bother with it, so this cloaked short-term convenience could even lead to long term messy practices and anti-convenience.

I agree we have to provide powerful tools that would cover all cases, not just the hypothetical quick-n-dirty case. However, I think that providing a powerful to beginners is usually confusing. You don't teach someone python by showing them the ins and outs of stdlib. You show them "hello world".

So I don't think that manim's "hello world" should involve an entire directory structure that is three-levels deep. I think manim's "hello world" should output a single file, and place it on the current directory. Once a user requires more than that (and they will), we can change to the full-directory output by changing a single line in the (soon-to-be-real) .cfg file.

What do you think?

[XorUnison]

I agree we have to provide powerful tools that would cover all cases, not just the hypothetical quick-n-dirty case. However, I think that providing a powerful to beginners is usually confusing. You don't teach someone python by showing them the ins and outs of stdlib. You show them "hello world".

Maybe. Then again I'm more the type that starts by doing the former and completely forgoing the latter. Not that we should aim for that type though. I also wouldn't compare a well defined folder structure to the ins and outs of a library. A folder structure takes me like 30s to look through. A library... not so much. Or rather much more, you know what I mean.

So I don't think that manim's "hello world" should involve an entire directory structure that is three-levels deep. I think manim's "hello world" should output a single file, and place it on the current directory. Once a user requires more than that (and they will), we can change to the full-directory output by changing a single line in the (soon-to-be-real) .cfg file.

What do you think?

We... could? The thing is, the amounts of complaints and questions I've heard about the directory structure so far is precisely 0, and I've heard a lot of complaints and questions.
I've sifted through the Discord, searching for "folder" and really, not one complaint of this kind, and I mean, once manim successfully renders something it explicitly tells you where the results are, so it's not like anyone ever had to search.
On the flip side there were complaints about folder structure not working properly.
It's mostly cairo, on spot two a little more cairo again, and then people are usually starting to instantly switch over to "how do I animate this thing?" kind of discourse.

So it sounds like we'd appease an unspoken minority with something kind of trivial, while annoying all the more proficient users who are taking a folder structure for granted. I know I take such a structure for granted with a program like this and just want to get accustomed to it asap.

[leotrs]

You make fair points. I guess it's fine to have this be the default as long as it's clear where the end product is going to be.

My main concern is: when/if the output is a single file (say a single frame, or a single video), then this directory structure seems excessive and unnecessary to me. But I can open another issue for that in the future.

[eulertour]

It isn't clear how we'd get to a place where manim will always output only a single file because it needs temporary tex and svg files to handle latex. But it should be possible to specify the name of the output file with the -o flag.

[XorUnison]

It isn't clear how we'd get to a place where manim will always output only a single file because it needs temporary tex and svg files to handle latex. But it should be possible to specify the name of the output file with the -o flag.

Right. Tex and svg files. Thinking about those makes me consider more that any way of circumventing the folder structure just doesn't seem like a really good idea. Either you put them in a folder, and you already create a new folder, just that for beginners you risk them not really picking up on that detail contributing to long term confusion.
Or you don't, and aside from the one file they wanted you also litter the same folder with dozens of files most people never need to interact with themselves.

[leotrs]

But... those are temporary, right? It's actually very easy to create a tempfile that will even be deleted automatically. Not sure why this is a problem. I think I'm missing something...

[kilacoda]

No. They save time once you're working a lot with them, previewing and rerendering them. Instead of tempfiling them, store them in a better place.

[XorUnison]

But... those are temporary, right? It's actually very easy to create a tempfile that will even be deleted automatically. Not sure why this is a problem. I think I'm missing something...

I mean, you could tempfile them, sure.
But then you'd also bloat the code of this tool, making the program run slower because its generating files repeatedly, to save some few particularly uninitiated users from a folder structure they should generally get accustomed to rather sooner than later.
As kilacoda said, having them stick around can save lots of time. The creation of all files for my second project alone takes something like 2 minutes. Not something I'd want to add to a 10 second test rendering process every time. On the other hand if it's not the default then the time users spend to learn how to enable making the files temporary could as well be spent getting accustomed to 4 or 5 folders.
While I see the general benefit of making stuff simpler for some people, even if very few and very specific use cases, the downsides strike me as overwhelming, in this case.

Now putting them in a better place... Maybe? But where would that be? It'd have to be some sort of permanent folder, that wouldn't be the manim folder, or the project folder, nor the target folder or a folder defined by the user. Something like the standard documents folder? That strikes me more like something to consider, but if it wouldn't work out of the box across all OSes it could just end up being more user overhead too.

[kilacoda]

Maybe create a folder in the project file's directory with name of the project file or whatever's kept as OUTPUT_DIR in it, so when one views their output files, they can easily find them when sorted by name in a file explorer?

e.g.

|_ file1.py
|_ file1 (folder)
|_ file2.py
|_ file2 (folder)

[XorUnison]

Maybe create a folder in the project file's directory with name of the project file or whatever's kept as OUTPUT_DIR in it, so when one views their output files, they can easily find them when sorted by name in a file explorer?

Hm, but the current output already almost does that. Except the top folder is usually called media and then we get the project names, instead of the reverse. And if you do reverse it you code-wise end up with the issue of strapping the wagon before the horse again. And with multiple project files that would end up littering the folder again, just this time with folders.

While that wouldn't be a problem for all people, from what I see there are quite a lot of people who use manim for many small projects, not a few big projects. To them this would turn into an eyesore quickly.

Well... we could also go for the insane, but insanely adaptable solution of allowing people to completely customize their folder output via some special variable, then saved to the .cfg when that happens. Kinda like my Graph() class takes one big chunk of JSON/Dictionary formatted data.
But that's a lot of planing and coding for tiny to small levels of convenience for people. It doesn't strike me as a worthy investment.
It can be done. But do we want to?

[eulertour]

After reading through some of the arguments here I can think of at least two ways we can make this a little better:

1. The svg directory should probably have an option to be global if it isn't done by default; it's only really checked for debugging and there's no reason to regenerate the exact same svgs in for different scenes/projects/etc.
2. There shouldn't be a directory with the scene name because it would be redundant (e.g. media/video/basic/SquareToCircle/1440p60/SquareToCircle.png).
3. It seems to me that manim is used to create video much more often than it's used to create images, so we could save a level by eliminating the video dir and only specifying an images one for when it's needed.

Accordingly, I think a good solution would be

  svg <--- This should optionally be global and be named so that it doesn't conflict with a module
  <module name>
    <resolution>p<framerate>
      <scene name>.mp4
    images
      <resolution>p
        <scene name>.png

Then a first-time user will end up checking ./media/basic/1440p60/SquareToCircle.mp4. Short(ish), sweet, and to the point.

Regarding @XorUnison's comment on changing the directory structure if the user specifies a VIDEO_DIR, I would consider modifying a user-specified VIDEO_DIR at all to be unexpected. For example if the user specifies /path/to/video/output and we write to /path/to/video/output/1440p60 then we're presumptuously assuming that we know what the user wanted better than they do.

[leotrs]

@eulertour Agreed on all four accounts (3 list items + not changing the VIDEO_DIR).

Once we decide upon and implement these changes, I will open a different issue on the following topic (I mention it here for future reference). I'm all for having the default output dir be ./media/<module>/<resolution>p<framerate>/<scene>.mp4, but I also think it would be desirable to allow the user to modify it as freely as possible. For example, a mad user (like myself) might prefer ./media/<module>_<resolution>p<framerate>_<scene>.mp4. It'd be possible to implement this easily with the config.cfg that I'm working on rn. More on that later.

But, let's keep this discussion about the default values for now.

[leotrs]

How is this going to affect the CLI flags --media_dir, --video_dir, --tex_dir, and --text_dir? Need to know in order to finish #98.

[eulertour]

In the example I have above,
--media_dir and --video_dir are both ./media
--tex_dir is ./media/Tex
--text_dir is ./media/text
that said the directory refactoring doesn't have to happen in the same PR as the addition of the config file.

[leotrs]

Yeah, you are right. There is some logic inside config:_init_dirs that should be addressed when anyone makes a PR for this issue though. I've left it alone for now in #95.

[leotrs]

We can come back (or close) this discussion since #98 has been merged.

BTW, #98 has eliminated the flags --video_dir, --tex_dir and --text_dir. The only one that remains for now is --media_dir, and all other dirs are set according to the discussion here. if we are going to change that, perhaps optionally, as #132 suggested, we should discuss adding a new section to the default.cfg file that overrides the defaults. Perhaps the way that the --dry_run or -t flags are used now would work for this.

[leotrs]

On second thought, I think this issue has ran its course of deciding the default folder structure. If someone wants to work on setting an optional version using the new config system, let's open a new issue/PR for that (cc @kolibril13). Closing this now.