Godmode9: [Feature Request] Increase MAX_DIR_ENTRIES

[gingerbeardman]

I have a directory containing 1966 files

GodMode9 only lists the first 1025

#define MAX_DIR_ENTRIES 1024

Thanks for your consideration.

https://github.com/d0k3/GodMode9/blob/master/arm9/source/filesys/fsdir.h#L5

[wabberz]

Having too many files in a directory will make listing them _very_ slow. Of course it is possible to raise the limit (if that doesn't create memory issues), but I would advise you to sort the files into multiple folders. It will speed things up dramatically.

[gingerbeardman]

I think you're right, though GodMode9 has the fastest directory listing I've seen on 3DS.

RetroArch and FBI listing speed are "OK".
CTRXplorer listing speed is very slow.

[SirNapkin1334]

The real problem is not the entries limit, but why you would have so many entries in the first place. 1024 is a good number. I suppose you’re welcome to edit that value and compile yourself.

[gingerbeardman]

Original 3DS has 128 MB RAM.
New 3DS has 256 MB RAM.

How much extra memory is used increasing MAX_DIR_ENTRIES from 1024 to 2048?

A quick calculation shows that the DirEntry struct takes up about 270 bytes,
so DirStruct with 1024 entries takes up 270KB total.
and DirStruct with 2048 entries takes up double, so 540KB total.

Surely that's a quite reasonable request?

[urherenow]

How exactly did you come up with those calculations? Did you use a word processor to count how many characters are in the code that make up the struct? I don’t think that’s how it works... 🧐 I can’t be bothered to stare at the code myself to figure out if all entries take the same amount of ram or not (actual length vs maxlen), but 270 bytes seems a rather arbitrary number 🤔

[linouxis9]

Not sure what's funny.
I'm getting similar figures using sizeof, but he could have also pretty much calculated it himself knowing the size of the different types smartass.

[gingerbeardman]

It's funny because @urherenow assumes I have no idea what I am doing because they have no idea what they are doing 😄

I calculated it based on the contents of fsdir.h

char* = 1 byte
char[256] = 256 bytes
u64 = 8 bytes
EntryType = 1 or more bytes (I picked 5 bytes to be generous)
u8 = 1 byte
TOTAL = 256+1+8+5+1 = 270

I'm interested what the results of sizeof() are @linouxis9

[urherenow]

1) I was asking a question, hence the question mark at the end of the first sentence.

2) Github isn’t a kiddie forum. There’s no need for name calling, and it will not be tolerated. Grow up.

3) How exactly does using sizeof tell you how much ram is being used? I’ll ask again: Does every entry use a whole buffer block (maxlen of an entry) regardless of actual length? Or are you simply calculating the maximum possible amount of RAM needed for 2046 entries of maxlen? If the former, that would be very inefficient.

[Wolfvak]

Gonna chime in to settle the size once and for all...

Current DirEntry:

typedef struct {
    char* name;        -> 4 bytes (it's a pointer)
    char path[256];    -> 256 bytes
-> since this isn't aligned to an 8 byte boundary, add 4 bytes to pad
    u64 size;          -> 8 bytes
    EntryType type;    -> 2 or 4 bytes, depending on how GCC is feeling today
    u8 marked;         -> 1 byte
} DirEntry;            -> not packed, so it'll be aligned to the higher 4 byte boundary

This gives it a size between 276 and 280 bytes, pretty close to the one estimated in the first post.

Now, MAX_DIR_ENTRIES could be increased but the heap is already pretty close to it's limit as-is, mainly due to stuff we can't control like lodepng. Taking space from the ramdisk would be the only feasible option, and I don't see why it can't be done (4-8MB less wouldn't hurt too much, I think).

In the end, though, what about when people starting complaining about directories with >2048 files? What about >4096, >8192, etc? Increasing this value isn't a fix, it's merely a temporary solution.

The real solution is being sane for once and have directories with less than 1024 files.

[d0k3]

Eh, be nice to each other. ~300kB is around right. Problem is, if I just increase it, the next guy comes around and wants more. Not good. You shouldn't even have that many files in one dir. But I get the problem. Let me think about that one for a bit, alright?

[Wolfvak]

@urherenow "Does every entry use a whole buffer block (maxlen of an entry) regardless of actual length? [...] If the former, that would be very inefficient."

Depends on what you mean by "inefficient". It does take a lot of memory (relatively speaking) but it's also _really_ fast.

Having to allocate and free N 276 byte blocks every time you open a directory would be the inefficient choice.

[gingerbeardman]

Thanks all, and thanks @d0k3 for your consideration

@urherenow How exactly does using sizeof tell you how much ram is being used?

See: https://www.geeksforgeeks.org/sizeof-operator-c/ and https://stackoverflow.com/a/119128/28290

[urherenow]

Awesome. Thank you very much! For both explanations.

[SirNapkin1334]

Is this a romFS dir/subdir? Because perhaps the best solution would be to set it to above the amount of files in the romFS dir/subdir known to have the most. No sane person should have that many files in a dir besides that.
Again, you could increase it and compile yourself.

[d0k3]

Alright, I did it. I don't like this solution for the reasons mentioned earlier, but I guess it works for now. If we ever have to increase that again, I'll have to think of something better.

[gingerbeardman]

Thank you

[ihaveamac]

Because perhaps the best solution would be to set it to above the amount of files in the romFS dir/subdir known to have the most.

for future ref: Stella Glow has a directory with over 16,000 items in it.

[d0k3]

Ugh

[SirNapkin1334]

Oh, dear.

[d0k3]

Handling that properly would require either reserving an unreasonable amount of memory or putting in an unreasonable amount of time and rewrite a lot of stuff. Both are off the table for now, I may think about it later.

Also: goddammit, who in their right mind puts that many files into one dir? :/

[gingerbeardman]

A computer program? 😂