Rust: Using std::io::{Read, Write, Cursor} in a nostd environment

[roblabla]

I'm surprised there's no ticket for this, so here we go.

It isn't possible in nostd to use std::io::Cursor. That seems to be mostly because it requires std::io::Read, Write, and Seek. However, I'd argue that those traits should also be part of libcore, and only the specific implementations be in libstd. After all, the Read/Write/Seek traits and the Cursor struct don't seem to rely on anything that's part of the stdlib: they don't need allocation, etc...

My use-case is, I have to write data into an array, and treating that array in a file would lead to much more idiomatic code. Furthermore it'd allow using crates like byteorder in nostd environment.

[retep998]

The problem with this, which has been covered in previous issues, is that Read and Write have std::io::Error hardcoded in their method signatures which requires quite a bit of std machinery including the ability to allocate.

[roblabla]

Do you have a link to previous issues ? I tried looking for it, but nothing came up :(.

[retep998]

I... can't find the previous issues. Maybe it was just scattered comments over the place and IRC discussions...

[jD91mZM2]

These traits being absent is the reason rust-core_io exists. Relibc would thank you if core::io became a thing, and I could then continue on my newly started quest to make more crates support no_std.

[burdges]

Anything here requires the std::error::Error trait, which requires std::backtrace::Backtrace, but maybe all that only requires core+alloc. Assuming so..

We should explore replacing std::io::Error with some more flexible error type, vaguely inspired by the SmallError type suggested in https://github.com/rust-lang/rfcs/pull/2820#issuecomment-567439210 that encodes the io::error::Repr enum with some vtable pointer.

In essence, it should act like Box<dyn Error+Send+Sync+'static> except small error types get passed by value.

We could maybe preserve mem::size_of::<SmallError>() being only twice mem::size_of::<usize>() too.

Also, I'm unsure if std::error::Error could feature gate the std::backtrace::Backtrace methods on alloc, but maybe. We must encode Drop inside the errorr::Error trait though, which I suspect either requires magic similar to Box, or else makes Drop a supertrait for errorr::Error. We'd wind up with an Error trait like

pub trait Error: Drop + Debug + Display {
    fn description(&self) -> &str { "description() is deprecated; use Display" }
    fn cause(&self) -> Option<&dyn Error> { self.source() }
    fn source(&self) -> Option<&(dyn Error + 'static)> { None }
    fn type_id(&self, _: private::Internal) -> TypeId where Self: 'static { TypeId::of::<Self>() }
    #[cfg(feature = "alloc")]
    fn backtrace(&self) -> Option<&Backtrace> { None }
}

We must then use wrapper types that handle drop correctly.

#[derive(Clone)]
pub struct BoxError {
    inner: Box<dyn Error+Send+Sync+'static>,
}
// delegate Deref+DerefMut+Drop to inner

impl Drop for BoxError {
    fn drop(&mut self) {
        self.deref_mut().drop()
    }
}

use core::mem;
use core::raw::TraitObject;

const SMALL_ERROR_SIZE : usize = mem::size_of::<usize>();

#[derive(Clone)]
pub struct SmallError {
    vtable: *mut (),
    data: [u8; SMALL_ERROR_SIZE],
}

impl Drop for SmallError {
    fn drop(&mut self) {
        self.deref_mut().drop()
    }
}
impl Send for SmallError {}
impl Sync for SmallError {}

impl SmallError {
    pub fn new<E: Error+Clone+Send+Sync>(err: E) -> SmallError 
    // where mem::size_of::<E>() <= SMALL_ERROR_SIZE  // sigh
    {
        assert!(mem::size_of::<E>() <= SMALL_ERROR_SIZE);
        let data = [0u8; SMALL_ERROR_SIZE],
        core::intrinsics::copy(&err as *const E, &mut data as *mut [u8; SMALL_ERROR_SIZE] as *mut E , 1);
        let vtable = unsafe { mem::transmute::<&mut dyn Error,TraitObject>(&mut dyn err).vtable };
        // mem::forget(err);
        SmallError { vtable, data }
    }
}

impl Deref for SmallError {
    type Target = dyn Error+Send+Sync;  // Should we add +Clone when mutli-trait objects exist
    fn deref(&self) -> &Self::Target {
        let SmallError { vtable, ref data } = self;
        mem::transmute::<TraitObject,&mut dyn Error>(TraitObject { vtable, data })
    }
}

impl DerefMut for SmallError {
    fn deref(&mut self) -> &mut Self::Target {
        let SmallError { vtable, ref data } = self;
        mem::transmute::<TraitObject,&mut dyn Error>(TraitObject { vtable, data })
    }
}

We then need some outer type that calls Drop for BoxError and SmallError, but forwards Error methods, like

pub struct DerefError(&dyn Deref<Target = dyn Error+Send+Sync+'static>+DerefMut+Drop)
impl Drop for DerefError {
    fn drop(&mut self) { self.drop() }
}
impl Error for DerefError {
    // delegate all methods to self.deref()
}

[roblabla]

Backtrace cannot really exist outside std. It is highly dependent on OS features. For instance, to get function names (symbolication step of printing a backtrace), it requires opening the executable as a file and parsing it to access the debug information table, e.g. it requires doing open(argv[0]). Or it might require dlresolve or other kind of libc stuff.

In an ideal world, binaries compiled in no_std could just have an "empty" implementation of Backtraces. I don't think that's possible however, since libstd simply reexports everything in libcore verbatim. It's not possible for libstd to use a different implementation of things than libcore AFAICT.

Unfortunately, moving backtraces to be an exposed interface of std::error::Error seems to permanently kill any possibility of std::error and std::io being available in libcore.

[burdges]

Could we replace &Backtrace with some opaque pub struct BacktraceRef(usize); type that exists libcore? We'd then provide inherent methods for that type only in std, assuming that flies with whatever magic.

[roblabla]

Std cannot (currently) add inherent impls for structs it does not itself define. It plays by the same rule as the rest of the ecosystem when it comes to coherence, there's no magic going on here (modulo some escape hatches for fundamental types that don't apply here). Libstd can implement libstd traits on libcore structs, or implement libcore traits for libstd structs. But it can't add its own inherent impls for (libcore) structs, only the crate defining a type can do that.

The best I can figure out is changing the backtrace function to return a &'a dyn BacktraceTrait and do away entirely with the Backtrace structs (they'd be an implementation detail). This would prevent adding new functionality to backtraces (since they're now a trait that anyone can impl) but it means libcore's Error could return an empty backtrace impl.

Except that's also undesirable because now all the errors living in libcore won't get to use backtraces - only errors in libstd would use the "right" backtrace implementation.

Maybe that'd be fixable using the same mechanism global allocators use? Have some kind of backtrace factory, have libstd give out a default implementation, and ask libcore users to provide their own (probably stubbed) implementation.

[burdges]

I suppose std cannot use features for dependencies like many crates do if we want std to ever actually be the standard library on some future operating system. In other words, if features are additive then std should always have all its features enabled.

I think &dyn BacktraceTrait sounds fine because std::backtrace::Backtrace indicates it'd resemble

trait BacktraceTrait : Debug + Display {
    fn status(&self) -> BacktraceStatus;
}

I still think BacktraceRef could be defined in core, but you define BacktraceTrait in std. Also we've several proposals for extensions like (a) doing inherent methods across distinguished crate boundaries, or (b) allowing similar opaque extern types like you gets from C header files, but your &dyn BacktraceTrait works well enough here I think.

As for implementation, there are several options here:

• We could treat Error special with the trait object tricks I wrote above.
• We could implement those same tricks via some proc macro, and some SmallBoxDyn type, so that other trait objects could benefit form a small Copy types being passed, and do not require even alloc, while larger types get boxed, requiring alloc.
• We could do some SmallBox type that handled the forwarding via compiler magic for the forwarding vtables, but also worked without always passing through trait objects.

[burdges]

I think anyhow::Error already provides no_std wrapper for std::io::Error. At present, it requires alloc but it looks like a reasonable starting point if we want to make this dependency optional.

We'd need to remove the Box in https://docs.rs/anyhow/1.0.26/src/anyhow/lib.rs.html#322-324 which increases the size from one to two usizes.

It appears anyhow::Error already manually implements its trait object internally, which I considered overkill but if they needed it then maybe my suggestions here also do.

We should reimplement the std::error::Error trait so that its new method does not depend upon std.

I think anyhow::Error addresses the Backtrace machinery by depending upon the backtrace crate, so maybe that already works no_std but requires alloc. We need a BacktraceTrait that avoids even needing alloc, as discussed above.

We should also remove the backtrace that anyhow::Error attaches https://docs.rs/anyhow/1.0.25/src/anyhow/error.rs.html#672 because it requires alloc. We could however permit layering one internally when using alloc via another internal wrapper type.

After all this we could then reimplement all traits from std::io and their async cousins. Appears doable although not sure until you try.