Hello, I find there is a huge boost in artifact size of cdylib library files in recent versions of rust. In our case, the lib doubles its size. I also find similar situations in other projects.
Here are the steps to reproduce (OS: 64bit Mac OS X 10.12.4 16E195):
git clone https://github.com/king6cong/sysinfo
cd sysinfo
on nightly-2017-06-30
rustup default nightly-2017-06-30
cargo clean
cargo build --features="c-interface"
ls -lht target/debug/libsysinfo.dylib
nm -g target/debug/libsysinfo.dylib|less
latest nightly (rustc 1.23.0-nightly (269cf5026 2017-10-28))
rustup default nightly
cargo clean
cargo build --features="c-interface"
ls -lht target/debug/libsysinfo.dylib
nm -g target/debug/libsysinfo.dylib|less
Comparing the output:
# nightly-2017-06-30
751K Oct 29 17:59 target/debug/libsysinfo.dylib
# latest nightly
2.2M Oct 29 17:52 target/debug/libsysinfo.dylib
and there are many rust specific symbols in the latest nightly generated lib.
Library size matters, especially on mobile devices such as iOS and Android, definitely eager to figure out the reason :)
king6cong
All 18 comments
target/debug
This is a debug build. I鈥檓 not surprised at all if it began putting more symbols in the files.
nagisa
on 29 Oct 2017
cc @michaelwoerister is this fixed by one of your #[inline] instantiation fixes?
nagisa
on 29 Oct 2017
@nagisa similar result with release version:
on nightly-2017-06-30
rustup default nightly-2017-06-30
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.dylib
nm -g target/release/libsysinfo.dylib|less
latest nightly (rustc 1.23.0-nightly (269cf5026 2017-10-28))
rustup default nightly
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.dylib
nm -g target/release/libsysinfo.dylib|less
Comparing the release output:
260K Oct 29 17:59 target/release/libsysinfo.dylib
1.7M Oct 29 17:52 target/release/libsysinfo.dylib
still, many rust specific symbols in the latest nightly generated lib, while nightly-2017-06-30 works as expected.
king6cong
on 29 Oct 2017
Nominating. Would be good to track this down, to at least diagnose the cause and add to the release notes. Probably not too high priority...
Mark-Simulacrum
on 30 Oct 2017
Thanks for the report, @king6cong! This looks indeed like a bug. We haven't changed anything in our compilation strategy the would require such a change.
is this fixed by one of your #[inline] instantiation fixes?
That might have a positive effect here, but since inline instantiation seems to have regressed already at the beginning of this year, something else must be the problem.
cc @alexcrichton
michaelwoerister
on 30 Oct 2017
I believe this is all debug information. On Linux tonight's nightly is 3.0MB and nightly-2017-06-30 is 2.4MB. After strip -g both are 1.4MB.
alexcrichton
on 30 Oct 2017
@alexcrichton
I tried on Linux (x86_64 Linux 4.12.12-1-ARCH) and reproduced the problem:
on nightly-2017-06-30
rustup default nightly-2017-06-30
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.so
strip -g target/release/libsysinfo.so
ls -lht target/release/libsysinfo.so
nm -g target/release/libsysinfo.so|less
on latest nightly
rustup default nightly
cargo clean
cargo build --release --features="c-interface"
ls -lht target/release/libsysinfo.so
strip -g target/release/libsysinfo.so
ls -lht target/release/libsysinfo.so
nm -g target/release/libsysinfo.so|less
Comparing the output on Linux:
# nightly-2017-06-30
# before strip
2.1M Oct 31 13:52 target/release/libsysinfo.so
# after strip
403K Oct 31 13:52 target/release/libsysinfo.so
# latest nightly
# before strip
3.0M Oct 31 13:54 target/release/libsysinfo.so
# after strip
1.4M Oct 31 13:54 target/release/libsysinfo.so
similar result for strip -x on macOS:
# nightly-2017-06-30
260K # before strip
152K # after strip
# latest nightly
1.7M # before strip
1.1M # after strip
And we can see many rust specific symbols in the striped lastest nightly version.
king6cong
on 31 Oct 2017
Huh that's odd, I'm not sure what happened before... Now I can reproduce the results you're getting!
Somehow this is related to --crate-type rlib --crate-type cdylib (both at the same time). When I drop the rlib crate type then current nightly is as small than the older nightly.
alexcrichton
on 31 Oct 2017
Bisection via nightlies points to https://github.com/rust-lang/rust/compare/ae98ebfcb...15aa15b03 (rust-nightly-2017-07-21 vs rust-nightly-2017-07-22) of which the most likely candidate is https://github.com/rust-lang/rust/pull/43183.
@michaelwoerister I think the change here was that the set of exported symbols doesn't take the crate type as input, so we pessimistically export all symbols as if we were making an rlib, but apparently before we did a better job of internalizing things via the linker I guess?
@king6cong is it problematic to split the project into a rlib crate and a cdylib crate? (vs having both in one). Failing that we can try to fix this in rustc.
alexcrichton
on 31 Oct 2017
@alexcrichton Great, that works!
# macOS
256K/148K # older nightly w/o strip
305K/194K # latest nightly w/o strip
# Linux
2.1M/403K # older nightly w/o strip
2.0M/393K # latest nightly w/o strip
For our project, the dylib size dropped from 23M to 9M, a huge difference!
king6cong
on 31 Oct 2017
This looks kind of wrong:
https://github.com/rust-lang/rust/blob/6713736275181abb3304730603afd785b0470ae3/src/librustc_trans/back/linker.rs#L747-L755
It does not take the crate type into account when computing the export threshold.
michaelwoerister
on 31 Oct 2017
https://github.com/rust-lang/rust/pull/45650 should improve the situation here. @king6cong, could you give it another try with the latest nightly?
michaelwoerister
on 2 Nov 2017
triage: P-medium
Discussed in compiler team. @michaelwoerister believes may be fixed, and in any case will pursue. Calling it P-medium because it only affects binary size and only if you use two crate-types at once.
nikomatsakis
on 2 Nov 2017
@michaelwoerister Tested in sysinfo and our project, both worked as expected, great!
Thanks 馃槃
king6cong
on 3 Nov 2017
There is still a big size difference on windows-gnu when building DLLs with crate-type = ["cdylib"], and it wasn't introduced in the suspected 2017-07-22 nightly.
Latest nightly (nightly-2017-11-02-x86_64-pc-windows-gnu):
cargo clean
cargo build --release
DLL size: 2566KB
Older nightly (nightly-2017-07-22):
cargo clean
rustup run nightly-2017-07-22 cargo build --release
DLL size: 1111KB
After stripping them with strip.exe from mingw64, both are around 650KB.
rkarp
on 3 Nov 2017
@rkarp which project is that for?
alexcrichton
on 4 Nov 2017
I used this repo: https://github.com/rkarp/rust-dll-demo
I turned off LTO for the above test, but that didn't seem to change anything besides all generated DLLs (stripped & non-stripped) being smaller overall. With LTO, the old nightly still produces significantly smaller DLLs. When stripped however, the difference is once again almost gone.
rkarp
on 4 Nov 2017
@rkarp I think your regression is unrelated, I've opened https://github.com/rust-lang/rust/issues/45809 to track that.
Looks like this issue is fixed though, so closing.
alexcrichton
on 6 Nov 2017
Related issues
dwrensha
路
3Comments
jmegaffin
路
3Comments
wthrowe
路
3Comments
zhendongsu
路
3Comments
defuz
路
3Comments
Most helpful comment
This looks kind of wrong:
https://github.com/rust-lang/rust/blob/6713736275181abb3304730603afd785b0470ae3/src/librustc_trans/back/linker.rs#L747-L755
It does not take the crate type into account when computing the export threshold.