Serenity: CI: Improve build

[lnzero1dev]

Unforunately, when changing something that relates to the build toolchain (e.g. LibC), the CI build gets a timeout after 50 minutes. The build finished successfuly, but the upload seems to take too long (?)...

Excerpt of the build log (https://travis-ci.com/github/SerenityOS/serenity/builds/160846076#L13680-L13681):

The command "./makeall.sh" exited with 0.
12.74s$ ../Meta/lint-shell-scripts.sh
[OK]: sucessfully linted ./Kernel/build-image-grub.sh
[OK]: sucessfully linted ./Kernel/build-image-qemu.sh
[OK]: sucessfully linted ./Kernel/build-root-filesystem.sh
[OK]: sucessfully linted ./Kernel/debug-kernel
[OK]: sucessfully linted ./Kernel/makeall.sh
[OK]: sucessfully linted ./Kernel/mkmap.sh
[OK]: sucessfully linted ./Kernel/run
[OK]: sucessfully linted ./Kernel/sync.sh
[OK]: sucessfully linted ./Libraries/LibWeb/Scripts/GenerateStyleSheetSource.sh
[OK]: sucessfully linted ./Meta/lint-shell-scripts.sh
[OK]: sucessfully linted ./Meta/refresh-serenity-qtcreator.sh
The command "../Meta/lint-shell-scripts.sh" exited with 0.
0.01s$ du -ch ../Toolchain/Cache/* || true
259M    ../Toolchain/Cache/ToolchainLocal_00b1807bc36977518d002da9f7ed0d18.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_0246c435730198b6cd20d53c35f5e5c5.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_036b8813ca92dedbb7af591c321d6761.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_07bb6cecae1625768ded922b90c2387f.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_0ea9b76e227501ed250da10ad6c4e6ce.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_10cee58a8405871b9c7a8b5eb48ffb4c.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_139a890e47b25801b430b083952bbf42.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_1a86baf99b7db9bdd173b47e9682eec4.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_276b372f62204c3b6861343d11a24caa.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_2f50fb133d7b4ee01eae54fda59a8c83.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_3bb6244be62ff4c937f5ca992cc5ad90.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_5148fae13cf1941fda732d83d0aea54e.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_516a2eb6ebb6416003edd4a59d4cf585.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_57cb7b21685cca07fed521cd8f9f0cc4.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_58eb44bfdd7f8664b22c3c1babf7634c.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_66ab3fb351ccaa88ac971e7236fc173d.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_673ef865537d200f832d3340c6d3294e.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_6d2753606f55e8d2d014af79e577c144.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_84c6b816d586dab145f66c0a27e13488.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_8e3f5bfa7001c19f757bfff232658f5f.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_a48d4b1e45be1945392f23cb8386250f.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_a68c7e66921ebcdf9e3b24168545de2e.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_a93cc0e48a61dfbde0d05e8e68d507b9.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_a9be3d923d1708588d320a97f8ee73a7.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_b04b7ed5c9958f8fa84a255574bbed9e.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_b11b7296ee794c101fe54292d5de0608.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_b55780030e6f62e13624b1e5b3c661dc.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_b6eb10484329706c82b435c455b6ed6c.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_b741763a01e44606e0c31c0ed04adce8.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_be012eb91f362530b752144ebedaf72b.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_c3924dfdf42cbc946cc90850c71bff12.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_cc4d4b8faa785c75e0a34633c04f464c.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_d5b5e0bb9599bda10ebf4727aa807f14.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_eb74d349988fb08a2d64befa27b8ac27.tar.gz
259M    ../Toolchain/Cache/ToolchainLocal_f63ee887af02f303cc45b1ac607b3e66.tar.gz
8.9G    total
The command "du -ch ../Toolchain/Cache/* || true" exited with 0.
store build cache
0.00schanges detected (content changed, file is created, or file is deleted):\n/home/travis/build/SerenityOS/serenity/Toolchain/Cache/ToolchainLocal_a68c7e66921ebcdf9e3b24168545de2e.tar.gz\n
changes detected, packing new archive
uploading PR.1654/cache--linux-xenial-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855--compiler-gcc.tgz
The job exceeded the maximum time limit for jobs, and has been terminated.

[lnzero1dev]

/cc @BenWiederhake

[BenWiederhake]

Hmm. My best guess is that Travis deals badly with the cache piling up over time. I was hoping that Travis automatically deletes the "oldest" files; even deleting them randomly would have worked. Damn.

Temporary workaround: @awesomekling should be able to access the Travis cache and clear it.

Long-term solution: Uhh, not sure. Maybe the build script could look at the tar-timestamps and delete everything except the youngest 10 toolchains? What do you think about that? (Note that the filesystem timestamps are unreliable on Travis, at least I don't see why they should be trustworthy.)

[lnzero1dev]

Im not sure, one thing that comes to my mind: The tars have the hash of the commit, so even if the tar timestamps are invalid, we would be able to look up the timestamp of the git commit. And maybe (hopefully) there is a even smarter solution to that :-D

[BenWiederhake]

The tars have the hash of the commit

No, it is the hash of the relevant .c and .h files (in a way). Very important difference, since the entire caching-thing hinges on that. It also means you can't easily derive the time from the hexits in the name.

Instead, one could use the timestamp embedded in the tar itself. Usually bytes 4-7, if I remember correctly.

[predmond]

Would ccache help if the cache can be saved between runs?

[BenWiederhake]

Does ccache delete/overwrite outdated files? If not, then ccache has the same issue.

(And, yes, ccache is a much better idea than my NIH-cache. Why didn't I think of that first?)

[predmond]

Ccache has a default size of 5gb so it will evict old objects.

[bugaevc]

I think this works now. I've definitely changed things about LibC / the build / toolchain, and haven't seen any timeouts on CI.

[lnzero1dev]

Nice, then we can close this one :^)