Serenity: Filesystem is half a gigabyte: Is it time for shared objects?

[BenWiederhake]

So our filesystem has grown a lot. Currently it looks like this:

~/workspace/serenity/Build$ dusch
12K ./mod
304K    ./res
4.5M    ./boot
52M ./usr
446M    ./bin
503M    total

(dusch is just an alias I like to use to order everything by human-readable size. The long form is find -maxdepth 1 ! -name . -print0 | xargs -0 du -sch0 | sort -hz | tr "\0" "\n", in case you're interested.)

That's half a gigabyte, just for various small applications!

(Also: Whoa, our kernel is only 4.5 MB? Awesome!)

~/workspace/serenity/Build/Libraries$ du -h ./LibWeb/libweb.a ./LibGUI/libgui.a
11M ./LibWeb/libweb.a
13M ./LibGUI/libgui.a

Because we link LibWeb and LibGUI into most applications, many binaries carry a copy of these around. LibWeb and especially LibGUI are used quite often, so any gains by LTO code elimination are offset by the sheer number of programs that link statically against (=carry a copy of) them. So most of those 446M are basically copies and copies of the same static libraries, in each binary.

A solution would be dynamic linking. For now, making only LibGUI dynamically The "manual kind" as demonstrated in LinkDemo already works nicely! However, it's "manual", and wrapping LibWeb or LibGUI into C-style functions sounds like a bad idea.

@awesomekling and everyone else: What do you think about this?
CC @ADKaster, because you implemented LinkDemo, and maybe I can interest you in writing more on dynamic linking? :)

[ADKaster]

There's been some WIP work in this area already by myself and @itamar8910

Issue:
https://github.com/SerenityOS/serenity/issues/1747

WIP MRs
https://github.com/SerenityOS/serenity/pull/1087
https://github.com/SerenityOS/serenity/pull/2461
https://github.com/SerenityOS/serenity/pull/2838

Always seems to get kind of close and then other things come up that cause the contributor to lose interest or get focused on other events. I know I've been focused on work and my master's recently so haven't been contributing much.

If you'd like to pick it up, go for it! I don't have anything WIP and Itamar just closed his latest MR saying he was going to focuson on something else.

General consensus seems to be that a good design would include a program, /bin/Loader that is the dynamic loader specified in the gcc spec files. That program would use AK and LibELF (but not LibC) to implement dynamic loading. Probably adapting the Dynamic loader that's in LibELF to do more things, have some kind of linked list of loaded objects (including the main executable), etc.

I added ELF Auxiliary Vector support a while back, which would make passing either a file descriptor, or pointers to the loaded main executable's program headers straightforward.

[awesomekling]

The time for shared objects is whenever someone adds support for shared objects :)

[nico]

Not to discourage anyone from adding support for shared objects, but static-link-everything feels fairly modern to me. Disk space is cheap, and static and self-contained binary are easy to deploy (...assuming a stable syscall interface ;) ...), don't have LD_PRELOAD vulnerabilities (but admittedly also less ASLR, assuming ASLR) and generally feel cloud-native :)

[awesomekling]

My main personal interest in shared objects is faster incremental builds :smile:

[nico]

Just switching to a faster linker (ideally lld, but even gold would help) will likely help a lot with that already, see slides link here: https://archive.fosdem.org/2019/schedule/event/llvm_lld/ Haven't tried this for serenity since things build reasonably quickly over here, but I'd expect a ~3-5x speedup for linking from moving from BFD ld to lld.

[BenWiederhake]

A faster linker doesn't (yet) help with the fact that each incremental build builds a new half-gigabyte image from scratch, and syncs it to the harddrive. All of this becomes a lot faster if the size goes from 500 MB to 80-or-something MB.

Also, LD_PRELOAD isn't just for security holes, you can also implement /dev/null as a webservice and other very important things :P

[bugaevc]

Yes please! The sooner we get everything linked dynamically, the better.

[nico]

If you care about file system size, you can also enable dead code stripping, which makes everything a bit smaller:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 279188baa..b551e01da 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -151,10 +151,11 @@ set(CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS "-shared")

 #FIXME: -fstack-protector

-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Os -g1 -fno-exceptions -fno-rtti -Wno-address-of-packed-member -Wundef -Wcast-qual -Wwrite-strings -Wimplicit-fallthrough -Wno-nonnull-compare -Wno-deprecated-copy -Wno-expansion-to-defined")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Os -g1 -fno-exceptions -fno-rtti -Wno-address-of-packed-member -Wundef -Wcast-qual -Wwrite-strings -Wimplicit-fallthrough -Wno-nonnull-compare -Wno-deprecated-copy -Wno-expansion-to-defined -ffunction-sections -fdata-sections")

 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DDEBUG -DSANITIZE_PTRS")
 add_link_options(--sysroot ${CMAKE_BINARY_DIR}/Root)
+add_link_options(-Wl,--gc-sections)

 include_directories(Libraries/LibC)
 include_directories(Libraries/LibM)
diff --git a/Kernel/CMakeLists.txt b/Kernel/CMakeLists.txt
index 43a9d4bc1..b341d2ff6 100644
--- a/Kernel/CMakeLists.txt
+++ b/Kernel/CMakeLists.txt
@@ -243,8 +243,9 @@ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pie -fPIE -ffreestanding -fbuiltin")
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mno-80387 -mno-mmx -mno-sse -mno-sse2")
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-asynchronous-unwind-tables")
 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -nostdlib -nostdinc -nostdinc++")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-function-sections -fno-data-sections")

-add_link_options(LINKER:-T ${CMAKE_CURRENT_BINARY_DIR}/linker.ld -nostdlib)
+add_link_options(LINKER:-T ${CMAKE_CURRENT_BINARY_DIR}/linker.ld -nostdlib -Wl,--no-gc-sections)

 add_library(boot OBJECT Arch/i386/Boot/boot.S)
 add_library(kernel_heap STATIC ${KERNEL_HEAP_SOURCES})