Stdlib: Very slow building of stdlib, and high memory usage

[vmagnin]

I have git cloned the repository in a Fedora 32 virtual machine.
I have encountered two problems, one minor, one critical:

• The README.md file told me to install fypp using $ pip install fypp but I obtained that error:

...
    copying build/lib/fypp.py -> /usr/local/lib/python3.8/site-packages
    error: could not create '/usr/local/lib/python3.8/site-packages/fypp.py': Permission denied
...

It was easily fixed by using sudo: $ sudo pip install fypp

• Then I typed: $ cmake -B build, then $ cmake --build build but the stdlib_experimental_stats_moment.f90 gives a lot of warnings (hundreds ?) and is freezing the build process:

/home/osboxes/stdlib/build/src/stdlib_experimental_stats_moment.f90:47:12:

   47 |         n = size(x, kind = int64)
      |            1
Warning: Possible change of value in conversion from INTEGER(8) to REAL(4) at (1) [-Wconversion]
/home/osboxes/stdlib/build/src/stdlib_experimental_stats_moment.f90:24:12:

   24 |         n = size(x, kind = int64)
      |            1
Warning: Possible change of value in conversion from INTEGER(8) to REAL(4) at (1) [-Wconversion]

I have waited several minutes, but it is still frozen. The compiler is burning the CPU.
The gfortran version is:

[osboxes@localhost stdlib]$ gfortran --version
GNU Fortran (GCC) 10.1.1 20200507 (Red Hat 10.1.1-1)
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[jvdp1]

It was easily fixed by using sudo: $ sudo pip install fypp

You could also do

pip install --user fypp

which does not require root access.
I guess this should probably mentioned in the README.md

The submodule stdlib_experimental_stats_moment contains hundreds (800?) functions (and also about the same numbers for other stats submodules). If make uses parallelization, this might take some times and RAM indeed.
You may want to limit the number of dimensions to a smaller number than 15 (e.g., 7), by adding the CMake -DCMAKE_MAXIMUM_RANK=7

[vmagnin]

The submodule stdlib_experimental_stats_moment contains hundreds (800?) functions (and also about the same numbers for other stats submodules). If make uses parallelization, this might take some times and RAM indeed.
You may want to limit the number of dimensions to a smaller number than 15 (e.g., 7), by adding the CMake -DCMAKE_MAXIMUM_RANK=7

Thanks for information.
Yes, the build finally succeeded, but it took a long time (probably more than 10 minutes).
I have had the same experience with Kubuntu 20.04. Very long, and it used 75% of my 8 GB RAM.
I think the README.md should give some information on this, because I really thought the build was frozen on the stdlib_experimental_stats_moment.f90 file...

Note that my machine is not a big workstation, but not ridiculous too: a 4 cores Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz

[vmagnin]

Concerning fypp, under Ubuntu I didn't need sudo:

$ pip3 install fypp

but I needed adding its path to PATH:

$ PATH="${PATH}:/home/mylogin/.local/bin"

before building.

[14NGiestas]

In my machine I couldn't compile the library since the SO killed the process due the high amount of memory needed at same point. There's something odd in the compilation process of stdlib_stats_moment.f90.
EDIT: Yeah... 55,673 lines of generated code...
EDIT2: Didn't noticed the DCMAKE_MAXIMUM_RANK option.

f95: fatal error: Kill signal terminated program f951
compilation terminated.
make[2]: *** [src/CMakeFiles/fortran_stdlib.dir/build.make:290: src/CMakeFiles/fortran_stdlib.dir/stdlib_stats_moment.f90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:307: src/CMakeFiles/fortran_stdlib.dir/all] Error 2
make: *** [Makefile:160: all] Error 2

[milancurcic]

@14NGiestas Try limiting the maximum rank of generated procedures. You can go as low as 4. For example, from stdlib directory:

mkdir build
cd build
cmake .. -DCMAKE_MAXIMUM_RANK=4

My 2014 laptop can't build the default rank (15) either, but with -DCMAKE_MAXIMUM_RANK=4 it builds in just over a minute.

[14NGiestas]

Thank you @milancurcic. I think this option should be mentioned to warn future contributers with low end machines, in the readme.

[milancurcic]

I agree--we'll fix that.

[zmiimz]

I have access to workstations with many hundreds GB of RAM, so I tested RAM usage with the default configuration. gfortran 10.1 (and gfortran 8.3) requires approx 26 GB during stdlib_stats_moment.f90 compilation and produces ~1.8 GB assembler code file in /tmp

[jvdp1]

I have access to workstations with many hundreds GB of RAM, so I tested RAM usage with the default configuration. gfortran 10.1 (and gfortran 8.3) requires approx 26 GB during stdlib_stats_moment.f90 compilation and produces ~1.8 GB assembler code file in /tmp

Thank you @zmiimz for these detals. Did you check these numbers when the number of ranks is limited to 7 (instead of 15)? Would it be more manageable?

A (temporary) solution could be to set the default number of ranks at 7, and to add an option to increase the number of ranks until 15 when the user wants it.

[zmiimz]

Yes, with cmake .. -DCMAKE_MAXIMUM_RANK=7 it looks definitely better and requires <2 GB of RAM. Cannot say about exact RAM usage ( I am detecting this just visually in htop)