Jigsaw: Jigsaw Scalability Max # of Markdown Files

[dgursh]

Hello all, I am wondering what the scalability of jigsaw is? What is the most number of markdown files / pages has the build command been tested with? I'm currently working on a project with over 1K+ markdown files and I'm running into issues when trying to build.

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes)

Not sure if it has to do with buggy foreach loops that I may have written by mistake @damiani @adamwathan can you chime in here?

Thanks.

[dgursh]

I seemed to have fixed the bug, by setting the memory limit in the php.ini file to the following:

memory_limit = 256M

I know this isn't the best solution. But, I checked my code and it wasn't buggy. So, I guess due to the number of files I was working with I needed more memory.

Hope this helps someone in the future.

[damiani]

We're working on testing the limits to see where we might be able to optimize things for larger sites. Do you have a repo you could share with us that we can test against?

[dgursh]

@damiani I do not have a test repo currently available, I will see if I can make one.

[dgursh]

@damiani Returning to this thread. I am running into issues with builds of 6,000+ pages. It takes around 5 minutes each time I build the site. I had to increase memory to

memory_limit = 2048M

My follow up question to you:

Is there a way to only build the files that change. For example, if I make a small change in one markdown file, is there a way I can only build up that one new file?

[damiani]

6,000+ is a lot of pages :)

There's no way to do this at the moment, and it's a tricky feature to add because a change to a single markdown file could potentially have repercussions on every other page (an added collection item that needs to appear in a menu, for instance, or a changed variable in a YAML header).

(Is your repo private? I'm looking into #226 and would love to test against a real-world example of a very large Jigsaw site.)

[dgursh]

@damiani Unfortunately this is a private repo, but what I can do is spin up a test website that has this many pages to test on.

226 is interesting and it looks like it speeds up the site build.

I want to stick with Jigsaw because I am using blade in this project and the ease of portability between Jigsaw and Laravel is a must. However, I've been looking into Hugo (Go) because it takes < 1ms to build a file instead of 1s with Jigsaw. This performance boost is necessary.

[damiani]

I tested #226 with a 1,000-item collection, and it does indeed speed things up by about 70%. My files were very simple—1,000 pages took just under 3 seconds—and your site obviously has much more going on than my simple test does. This speed boost alone would cut your build time down to about 1.5 minutes, which is still really long for regular rebuilding.

[damiani]

OK, I believe I have a solution that will speed up build times even more dramatically, cutting my 1,000-page test build down to about 1 second. While I work on this, it would be super helpful if you were able to put together a demo repo that mimics yours (in terms of size, complexity, and build time) so I can use it for testing and speed/memory profiling.

[dgursh]

@damiani Here is the repo with 6,000 lesson files I generated. I can make more files if needed and increase the complexity by building more collections. Just let me know. Hope this helps you test!

https://github.com/dgursh/jigsaw-bugs

[dgursh]

@damiani I added dummy text to the files, so that we can also test the speed of the markdown parser.

[damiani]

@dgursh Yes, this is super helpful, thanks for putting it together. The speed improvements I mentioned above will help substantially with filesystem-related operations; but for a site this large, there's still a bottelneck when building the collections array in CollectionDataLoader, which on this test site takes 90 seconds. I'm looking into ways to optimize that at the moment.

---

Update: The change from Finder to SplFileInfo cuts the CollectionDataLoader execution time from 90 seconds to about 4.3 seconds, which is incredible. I still want to see if I can cut that down a bit.

[dgursh]

@damiani just checking progress on this. How is it going? Would love to test it.

[damiani]

I'll be opening a PR for folks to test with tomorrow 👌🏻

[dgursh]

@damiani I couldn't wait to try it I was too excited... With the current branch "kd-speed-optimizations", my private repo is running the following specs:

jigsaw build -v -c
1/3374 [------------------------------]
Build time: 47.59 seconds (using cache)

It is definitely an improvement, but still too slow if any small change requires a re-build. Not sure why my repo isn't gaining the significant speed improvements. Perhaps it's because I have multiple collections?

Sorry to be the bearer of bad news... Have we thought about a way to detect and only re-build the files that have been changed via git?

For example, what if we could detect the type of file that was changed by reading the git status of the source directory. If a .md file is changed, we only rebuild that one file. If a blade file changes we re-build all files. Thoughts?

[dgursh]

@damiani after further analysis, I believe the bottleneck is my include statements. When I remove them from my blade template files, the files are built in 13 seconds. The include statements rely heavily on for loops that perform eloquent queries on these massive collections like this:

 @foreach ($lessons->where('course', strtolower($course->title)) as $lesson)

There's a lot of nested for loops like these that are required for the logic of the site. I can provide some examples and update the test repo I gave you to mimic these includes. Stay tuned!

[damiani]

Make sure your first benchmark (the 47 second one) was taken after running with -c twice; the first time, it's not actually using the cache at all, as it needs to build it first.

That said, the optimizations in #260 are going to have different results based on the exact nature of how your site is constructed. Your @foreach is definitely going to be slow, and is the reason we can't rebuild individual changed files only—because Jigsaw has no way of knowing if any given file is going to be referenced somewhere else. Changes to an individual Markdown file might need to find their way into hundreds or thousands of other files, for a side nav, for instance.

To be able to selectively re-build individual files would require Jigsaw to build a dependency tree, which isn't in the cards at the moment.

[dgursh]

@damiani yes, much faster the second time running with -c at 36.7 seconds.

What I'm noticing is that my website runs foreach and eloquent queries using the collections (of > 1000 items) and embeds them on every single page as in the case of a sidebar navigation.

Can you think of any way where we can speed up this use-case? It shouldn't have to perform these massive foreach queries on big collections when building up every page. Maybe a possible solution is a pre-build step for any foreach clause that uses a collection.

[damiani]

Let's move this discussion over to the PR https://github.com/tightenco/jigsaw/pull/260...

[raniesantos]

@dgursh I'm just curious, if you do a search for the keyword @include in your source folder, how many results turn up?

Mine is 7 matches across 3 files.