Rfcs: Lazy-loading of large byte literals

[SoniEx2]

The compiler should emit binaries with lazy loading for large byte literals. That way the OS/kernel doesn't load a full 60GB binary into RAM, but instead loads parts of the embedded 60GB tar file as they are accessed.

[sfackler]

Isn't this how the kernel already behaves? If not, what specific change would this involve in e.g. the ELF binaries that we generate?

[SimonSapin]

Sounds like what you want is not a literal but shipping a file next to your executable and reading that with std::fs?

[SoniEx2]

Find a portable way to read an attached/concatenated zip file. Needs to work everywhere.

[fstirlitz]

Just read it as-is. ZIP files are read from the end.

[SoniEx2]

Find a _PORTABLE_ way to read an attached/concatenated zip file. Needs to work everywhere.

[emberian]

Yes, @sfackler, this is already how the loaders on at least Linux and macOS work. There's nothing rustc can do differently here, and it already has the desired behavior. Eg:

static LARGE: &'static [u8] = include_bytes!("big.data");

fn main() {
    let mut s = String::new();
    loop {
        std::io::stdin().read_line(&mut s).unwrap();
        match s.trim().parse::<usize>() {
            Ok(idx) => println!("{}", LARGE[idx]),
            _ => println!("try again with a number")
        }
        s.clear();
    }
}

With head -c 53687091 /dev/urandom > big.data. The binary is 52M, but the resident memory after launch is only ~1M. The rest is mapped, waiting, and this is as "lazy" as you'll get.

[sfackler]

Find a PORTABLE way to read an attached/concatenated zip file. Needs to work everywhere.

What are the portability concerns with reading a zip file? Why is this a thing that the standard library should provide?

[SoniEx2]

There are portability concerns regarding getting access to the original executable file. (as well as a related race condition.)

Windows pretty much guarantees you can get access to the .exe, including a bundled zip file on that .exe (created with, say, copy /b exe.exe + zip.zip bundle.exe), and even the full path. POSIX on the other hand does not.

[emberian]

Fortunately, you can just use include_bytes!. Can you close this issue? The OS behaves in exactly the way you describe already.