Zola: Out of memory, process killed

[ianare]

Machine info:

32 GB of RAM
8-core CPU: Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz

Running zola v0.5.0

The site I'm generating has a content directory with 105336 files, 105732 sub-folders. Almost all of the files are section markdown.

I understand this is probably not a very common use case, but hopefully will be useful for you in optimizing memory management of Zola.

dmesg -T | grep zola

Out of memory: Kill process 29015 (zola) score 908 or sacrifice child
Killed process 29015 (zola) total-vm:30630456kB, anon-rss:30549904kB, file-rss:0kB, shmem-rss:0kB
oom_reaper: reaped process 29015 (zola), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Grafana chart over 3 unsuccessful runs:

[Keats]

100k files :o
Where does it error? Does it manage to load everything (ie do you see something like -> Creating 43 pages (0 orphan), 1 sections, and processing 0 images)?

[caemor]

It seems like the hashmaps for get_taxonomy, get_page and get_section, which are made from register_tera_global_fns, are just getting too big for blogs over 10k files

For huge blog:

For giant blog (bench example with 100k files generated) until it ran out of memory:

about the tool: heaptrack was the first thing I found by searching and therefore I used it for this (+ it is in the arch repos) and one needs to add the following to the main file: (for more infos see: https://speice.io/2018/10/case-study-optimization.html - this also might change the result a little bit)

use std::alloc::System;

#[global_allocator]
static GLOBAL: System = System;

[ianare]

100k files :o
Where does it error? Does it manage to load everything (ie do you see something like -> Creating 43 pages (0 orphan), 1 sections, and processing 0 images)?

No it never got to that point, the only console output was "building site", followed by "Killed".

For what it's worth, I was able to generate the site on a machine with 128Gb of RAM.

Usage was around 39 GB on each run.

[Keats]

It seems like the hashmaps for get_taxonomy, get_page and get_section, which are made from register_tera_global_fns, are just getting too big for blogs over 10k files

Yeah that was my guess as well but it is tricky to fix: https://github.com/Keats/tera/issues/340
I don't have any good idea on that currently other than rewriting a custom serde format that doesn't clone like serde_json

@ianare
What's the average size of a section?

[Keats]

get_page and get_section are redundant now, we can have only get_content so it should be better.
Most of the improvements are likely to be in Tera rather than Zola though.

[ianare]

Sorry for the delay.

I've run some more generations and it seems that when the sections are small (~ 5 kb) and not complex the memory problem crops up.

When the sections are more complex and larger (10-15 kb), and processing times are slower, the memory usage is better (even if still having large peaks).

It might be a suitable workaround to be able to specify the number of CPUs used during generation.

This would also be useful on machines that are both generating and serving with a dedicated server like nginx at the same time, so as not to introduce lag on the site during generation. Although this would be better in a dedicated ticket... should I open this?

Also really wanted to thank everyone for taking the time to look into this... and apologies for not having (yet) the ability to help out in the code.

[ianare]

I've run some more generations and it seems that when the sections are small (~ 5 kb) and not complex the memory problem crops up.

When the sections are more complex and larger (10-15 kb), and processing times are slower, the memory usage is better (even if still having large peaks).

It might be a suitable workaround to be able to specify the number of CPUs used during generation.

Apologies, there was an error in the configuration and the reason it was OK on the 32GB server was because there was "only" 60K files.

[Keats]

@caemor @ianare

I've updated the next branch and it should use way less memory now, can you try again?
Running heaptrack again surfaced another source of allocations which will be fairly easy to fix afaik

[Keats]

@caemor @ianare ping

[caemor]

I'm gonna try it once more later today. Sorry for the late answer :-D

[ianare]

Sorry for the delay, I'm getting an error message related to macros:

--> 4:1
|
4 | {% import "macros/image_list.html" as image_list %}
| ^---
|
= unexpected tag; expected end of input or some content

[Keats]

I believe I fixed that in Tera but didnt push a new version yet. Is there
an empty line above? Try to remove it if there is one

On Thu, 7 Feb 2019, 20:19 ianaré sévi, [e-mail] wrote:

Sorry for the delay, I'm getting an error message related to macros:

--> 4:1
|
4 | {% import "macros/image_list.html" as image_list %}
| ^---
|
= unexpected tag; expected end of input or some content

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/getzola/zola/issues/536#issuecomment-461560413, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AApho6RJJ2QVcpITYe8Ee0CPrjl8aQNoks5vLHxAgaJpZM4Yv1c_
.

[caemor]

the next-branch doesn't compile for me with the following errors:

cargo bench bench_loading_huge_blog
.......
Compiling site v0.1.0 (/home/caemor/git/tests/zola/components/site)                                                        
error[E0599]: no method named `pages_values` found for type `std::sync::Arc<std::sync::RwLock<library::Library>>` in the current scope
  --> components/site/benches/site.rs:46:49                                                                                   
   |                                                                                                                          
46 |     b.iter(|| site.render_rss_feed(site.library.pages_values(), None).unwrap());                                         
   |                                                 ^^^^^^^^^^^^                                                             

error[E0599]: no method named `sections_values` found for type `std::sync::Arc<std::sync::RwLock<library::Library>>` in the current scope
  --> components/site/benches/site.rs:64:32                                                                                   
   |                                                                                                                          
64 |     let section = site.library.sections_values()[0];                                                                     
   |                                ^^^^^^^^^^^^^^^                                                                           

error[E0308]: mismatched types                                                                                                
  --> components/site/benches/site.rs:65:55                                                                                   
   |                                                                                                                          
65 |     let paginator = Paginator::from_section(&section, &site.library);                                                    
   |                                                       ^^^^^^^^^^^^^ expected struct `library::Library`, found struct `std::sync::Arc`
   |                                                                                                                          
   = note: expected type `&library::Library`                                                                                  
              found type `&std::sync::Arc<std::sync::RwLock<library::Library>>`                                               

error: aborting due to 3 previous errors                                                                                      

Some errors occurred: E0308, E0599.   

[Keats]

Looks like I forgot to update those benches, will do later

On Thu, 7 Feb 2019, 21:23 Chris, [e-mail] wrote:

the next-branch doesn't compile for me with the following errors:

cargo bench bench_loading_huge_blog
.......
Compiling site v0.1.0 (/home/caemor/git/tests/zola/components/site)
error[E0599]: no method named pages_values found for type std::sync::Arc<std::sync::RwLock<library::Library>> in the current scope
--> components/site/benches/site.rs:46:49
|
46 | b.iter(|| site.render_rss_feed(site.library.pages_values(), None).unwrap());
| ^^^^^^^^^^^^

error[E0599]: no method named sections_values found for type std::sync::Arc<std::sync::RwLock<library::Library>> in the current scope
--> components/site/benches/site.rs:64:32
|
64 | let section = site.library.sections_values()[0];
| ^^^^^^^^^^^^^^^

error[E0308]: mismatched types
--> components/site/benches/site.rs:65:55
|
65 | let paginator = Paginator::from_section(&section, &site.library);
| ^^^^^^^^^^^^^ expected struct library::Library, found struct std::sync::Arc
|
= note: expected type &library::Library
found type &std::sync::Arc<std::sync::RwLock<library::Library>>

error: aborting due to 3 previous errors

Some errors occurred: E0308, E0599.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/getzola/zola/issues/536#issuecomment-461582302, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAphow_yx6Lscn1OIr4ZWq9ZnFWrmAyDks5vLItZgaJpZM4Yv1c_
.

[Keats]

Both issues should be fixed on the next branch now

[caemor]

Looks much better than last time. Approx. only 25% of last times maximum heap usage for the huge blog. And the giant blog which didn't build last time also worked with a max usage of 5.8GB
Great work Keats! :+1:

Huge Blog (10000 pages, 0 orphans, 0 sections, 0 images): 579,5MB at max

Giant Blog (100000 pages, 0 orphans, 0 sections, 0 images): 5,8GB at max

In Comparison current master with the same huge blog from above needs 2,3GB at max

Building time also got reduced to about half it's previous value: (zola build)

Type | Without heaptrack | With heaptrack
----| ------| -----
huge on next | ~6s | ~70s
huge on master | ~15s | ~148s
giant on next | ~71s | ~711s

[Keats]

Looks like the RSS feed doesn't have a limit so it serialises and renders
all the pages. Can you try to put a realistic limit like 100 for rss_limit
in the config?

On Fri, 8 Feb 2019, 20:41 Chris, [e-mail] wrote:

Looks much better than last time. Approx. only 25% of last times usage for
the huge blog. Great work Keats! 👍

[image: zola_huge_blog_18858]
https://user-images.githubusercontent.com/11088935/52501678-32b6fd80-2be1-11e9-9ba7-205946ccfed7.png

[image: zola_giant_20174]
https://user-images.githubusercontent.com/11088935/52501686-36e31b00-2be1-11e9-924d-a98df4e3c6f0.png

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/getzola/zola/issues/536#issuecomment-461922357, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AApho34ridCR1K_J5-Nm6d1-m7svESGyks5vLdLigaJpZM4Yv1c_
.

[caemor]

After setting the rss_limit to 100 the time got down to ~65s/~5.8s (with/without heaptrack) and ~455MB max heap usage for the huge blog on next. Also be aware that the screenshot is only showing the state at its max usage.
Now the serializing of the taxonomies takes the most memory.

@Keats maybe you want to set that limit in the gen.py autogeneration in the future for the benches?

Note for myself: Steps to reproduce these heaptracks later:

pacman -S heaptrack
cd zola/components/site/benches
python gen.py
cd zola/
cargo build --release
cd zola/components/site/benches/huge_blog/
heaptrack ../../../../target/release/zola build

Now use the commandline that heaptrack returns to open and analyze the recorded heaptrack (sth similar to _heaptrack --analye "path_to_folder/.../heaptrack.zola.25242.zst"_)

[Keats]

Now the serializing of the taxonomies takes the most memory.

I see that the toc serialization is taking lots of time because it is on page rather than being added to the context of the specific page rendering itself. Probably an easy win as I don't think many people want to show the table of contents while displaying a list of pages but I could be wrong...

Still using a bit too much memory for my taste...

[Keats]

I moved the toc out of page and that's a bit better.
The reason the taxonomy rendering takes so much time if that they are not paginated so it is basically serializing all the pages, which is going to take time/memory in those benches.
The blog benches are not super realistic, the huge-kb has 10k pages as well but renders in 3.6s for example.

[Keats]

It should be fine with 0.6.0 (currently being built).
Re-open the issue if you still encounter problems!

[ianare]

Can confirm that memory usage is much much lower now. Great job!