Dhall-haskell: Too much memory use in compiling Dhall.Parser

[wavewave]

On my laptop with 8GB memory, it took almost 100% memory to compile Dhall library, especially on Dhall.Parser. If we can reduce GHC memory usage by simple module splitting or code reorganization, it's much desirable.

[Gabriel439]

@wavewave: Yeah, that seems like a good idea

Just out of curiosity, what version of GHC are you using to build the dhall package?

[wavewave]

I use GHC 8.2.2.

[FintanH]

Fwiw: This occurred once we clamped to 2f3934ddd3bbffa5da778ef106680d9f122a16a2 updating from 4a085aa3d622886cf7dd96a1ad475ba914d5ab1f.

[f-f]

Now that #448 went in, compilation performance should be more acceptable.
@wavewave can you try taking a look at this and see if it's good?

[FintanH]

@f-f We would like to test this but to update past the sha we're using we would need #453 to be fixed.

[Gabriel439]

@FintanH: Yeah, #453 close to being fixed. ETA is about 1-2 days

[FintanH]

Thanks @Gabriel439! Very much appreciated :)

[Gabriel439]

I have a pull request up with the fix now: https://github.com/dhall-lang/dhall-haskell/pull/471

You should be able to test memory usage off of that branch

[FintanH]

Unfortunately we're still seeing the same issue.

[2018-06-18T16:57:55Z]     Configuring dhall-1.14.0...
[2018-06-18T16:57:55Z]     Preprocessing library for dhall-1.14.0..
[2018-06-18T16:57:55Z]     Building library for dhall-1.14.0..
[2018-06-18T16:57:55Z]     [ 1 of 13] Compiling Dhall.Context    ( src/Dhall/Context.hs, .stack-work/dist/x86_64-linux-nix-9ab91c4/Cabal-2.0.1.0/build/Dhall/Context.o )
[2018-06-18T16:57:55Z]     [ 2 of 13] Compiling Dhall.Core[boot] ( src/Dhall/Core.hs-boot, .stack-work/dist/x86_64-linux-nix-9ab91c4/Cabal-2.0.1.0/build/Dhall/Core.o-boot )
[2018-06-18T16:57:55Z]     [ 3 of 13] Compiling Dhall.Pretty.Internal[boot] ( src/Dhall/Pretty/Internal.hs-boot, .stack-work/dist/x86_64-linux-nix-9ab91c4/Cabal-2.0.1.0/build/Dhall/Pretty/Internal.o-boot )
[2018-06-18T16:57:55Z]     [ 4 of 13] Compiling Dhall.Core       ( src/Dhall/Core.hs, .stack-work/dist/x86_64-linux-nix-9ab91c4/Cabal-2.0.1.0/build/Dhall/Core.o )
[2018-06-18T16:57:55Z]     [ 5 of 13] Compiling Dhall.Parser     ( src/Dhall/Parser.hs, .stack-work/dist/x86_64-linux-nix-9ab91c4/Cabal-2.0.1.0/build/Dhall/Parser.o )
...
Exited with code 1

[Gabriel439]

I'll need more details on how you build dhall because I cannot reproduce this problem on my personal laptop (8 GB RAM, OS X), or on Dhall's CI (4 GB RAM, Linux).

The main details I need are:

• Operating system
• Build tool (i.e. stack or cabal or Nix)
• Any unusual configuration flags (i.e. profiling, debugging, dead code elimination)

[FintanH]

Yup, I can follow up with full details :)

[f-f]

@FintanH are you compiling the latest master? The submodules should compile before compiling Parser (in my case Parser.Token takes a while to compile, while Parser is pretty much instantaneous):

...
[ 5 of 21] Compiling Dhall.Core       ( src/Dhall/Core.hs, .stack-work/dist/x86_64-osx/Cabal-2.0.1.0/build/Dhall/Core.o )
[ 6 of 21] Compiling Dhall.Parser.Token ( src/Dhall/Parser/Token.hs, .stack-work/dist/x86_64-osx/Cabal-2.0.1.0/build/Dhall/Parser/Token.o )
[ 7 of 21] Compiling Dhall.Parser.Expression ( src/Dhall/Parser/Expression.hs, .stack-work/dist/x86_64-osx/Cabal-2.0.1.0/build/Dhall/Parser/Expression.o )
[ 8 of 21] Compiling Dhall.Parser     ( src/Dhall/Parser.hs, .stack-work/dist/x86_64-osx/Cabal-2.0.1.0/build/Dhall/Parser.o )
...

[FintanH]

@f-f That's a really good point, nice catch! I did update to the latest master (as far as I remember..)

I'll take another stab at it soon and investigate further. Stay tuned 👂

[f-f]

@FintanH I believe this has been fixed with #448, so I'll close it, but please reopen if it still holds!

[FintanH]

Ya I think we might be ok. I'll keep an eye out on our side of things 👍