Rust: RFC: Merging the avr-rust fork upstream

[dylanmckay]

Hello all,

I would like to know the general opinions on merging the avr-rust fork into Rust proper.

The fork itself has became a lot more stable with less bugs in the recent months. It has also started attracting a number of people interested in using it.

You can find one of the more interesting projects using avr-rust on GitHub.

Blockers

LLVM 5.0

~Rust is currently on LLVM 4.0, which contains a working AVR backend, but there have been many bugfixes since then. We would need to wait for LLVM 5.0 to be supported (nearly finished: #43370) before we get a version of the AVR backend that has a few important bugs fixed.~

This is no longer a blocker. Upstream Rust is at LLVM 6.0 as of 2018-02-20.

Questions

Cherry-picking fixes

If AVR support was built into mainline, we'd need to be able to cherry-pick patches into Rust's LLVM fork. I don't imagine this would be much of a problem, as we already cherry-pick a number of important fixes into there.

All of the bugfixes cherry-picked into the avr-rust repository have already been upstreamed into LLVM trunk, which would continue to be the case if we merged the fork, as I am not a fan of the LLVM fork diverging too far from trunk.

Cherry-picking is necessary because of LLVM's 6-month release cycle.

Current issues in the AVR backend

There aren't any known bugs in the avr-rust/rust repository - all of the known bugs are issues in the AVR LLVM backend; here are some of the more interesting/impactful ones.

libcore cannot be compiled without modification

There is a milestone set up to track what bugs need to be fixed in order to get libcore compiling successfully without modification.

This hasn't been much of a problem for users so far, as xargo will transparently compile libcore on the fly when needed, and we can override libcore in Xargo.toml.

I am unsure what the Rust team thinks of merging a target which can't use the stock libcore.

Any operations on function pointers other than 'call' access RAM, not program memory (avr-rust/rust#68)

This is a symptom of AVR being the very first in-tree LLVM backend for a Harvard architecture. LLVM currently assumes that all functions reside in "the generic address space", which corresponds to RAM. Because of this, if you attempt to load/store through a function pointer, it will access RAM instead of the program memory.

Good news is that I have pending upstream LLVM patches to fix it (D37052, D37053, D37054, D37057).

32-bit shifts generate calls to a compiler-rt routine that doesn't exist (avr-llvm/llvm#163)

Because there aren't many (if any) targets that don't support 32-bit shifts natively, libgcc and compiler-rt do not have 32-bit versions of shift routine, even though LLVM still happily generates a call to it.

This causes an undefined symbol error whilst linking. This will only happen if you actually write code or use code that performs 32-bit shifts, as the linker is quite good at removing all dead code.

Note that I've had one user hit the missing shift bug due to compiling in release mode, which promoted a multiplication to a shift as an "optimisation".

Actual changes to merge

You can find all AVR-specific differences by looking at this diff.

Note that over half of that diff is just the README and Travis CI configuration - the actual code being upstreamed is very small; just some glue code to enable the AVR backend and a target specification.

This diff also conditionally disables parts of libcore for AVR - these fixes would not upstreamed, and are not strictly required as downstream users can use Xargo to compile a minified libcore for AVR).

Links

AVR-Rust on Gitter
AVR-Rust on GitHub

[Restioson]

+1! Avr rust built into the compiler proper would be super useful. It's almost free of bugs now.

[dylanmckay]

Not completely free of bugs :)

I will update the description to include some information about the state of the backend w.r.t bugs

[Restioson]

Almost though 😄. Just a couple to go

[alexcrichton]

In general we're quite welcoming of platforms upstream in rust-lang/rust so long as they don't place a maintenance burden on us! Some specific thoughts from what you're thinking:

• Cherry-picking LLVM commits every so often is totally OK, I think y'all've gone through the process a few times already as well :)
• Missing intrinsics in compiler-rt is fine by us, you have the option of implementing them in Rust as well through the compiler-builtins project.
• The patch you have looks pretty good, although I think we'd want to work more through the various #[cfg] in libcore. Are these items left out due to "bugs in LLVM" or because they're fundamentally not supported at all on AVR? The former would be best done by "fixing LLVM" somehow whereas the latter makes things more tricky.

In general we currently have all platforms with the same uniform interface of libcore/libstd, but it's not clear that'll remain true into as we keep picking up more platforms.

[dylanmckay]

The patch you have looks pretty good, although I think we'd want to work more through the various #[cfg] in libcore. Are these items left out due to "bugs in LLVM" or because they're fundamentally not supported at all on AVR?

It is because of bugs in LLVM.

The only question in my mind is on support of i128/u128 types - I think that for AVR, these aren't really useful. The i128 support in libcore is currently commented out because of a bug, but there may be other undiscovered bugs as it is not a well tested codepath, and really exercises the register allocator as the AVR only has 32 bytes worth of general purpose registers.

It is still quite likely that we could get i128 working on AVR though, I don't know too much of the specific problems it triggers in the backend.

I think that the best way forward would be to propose the patch for real once libcore _does_ compile without modification, or at least without many.

[alexcrichton]

I think that the best way forward would be to propose the patch for real once libcore does compile without modification, or at least without many.

Sounds reasonable to me!

[est31]

The only question in my mind is on support of i128/u128 types - I think that for AVR, these aren't really useful.

My main fear of not supporting i128 on AVR is that it will have a chilling effect on adoption of 128 bit integers for the sake of compatibility with AVR or other embedded platforms. E.g. if there is a library that uses 128 bit integers, AVR users that will want to use it will file issues to drop the usage. This might make 128 bit integers not "safe" to use in Rust code that strives to be portable, which I wouldn't like to happen.

[shepmaster]

a chilling effect on adoption of 128 bit integers for the sake of compatibility with AVR or other embedded platforms

I don't think that it's a terribly huge deal here. Small embedded devices already have giant limitations (no stdin / stdout, generally no allocator, etc.) that make it real hard to just drop in an arbitrary library. I recently learned that AVR GCC's double is actually an alias for float! I don't know if we are going to keep that strangeness or not, but it would affect crates way more than i128.

I think we are always going to have special features that are used to make a crate suitable for embedded, just like we do for no-std.

[est31]

no stdin / stdout, generally no allocator, etc.

You are describing the #![no_std] ecosystem. There are libraries that target this ecosystem. And the general rule for that ecosystem is to take libcore as a given, which also includes i128. Each target that doesn't support i128 has a larger chilling effect inside this ecosystem, because an embedded target's "market share" is bigger inside the subset of the entire Rust ecosystem where the x86 family is not a very relevant player.

I don't know if we are going to keep that strangeness or not, but it would affect crates way more than i128.

Interesting! I do agree that if we'd alias f64 to f32 (or not provide it), it would affect the ecosystem more. However, if we can strive for consistency, why shouldn't we? It is definitely possible for us to implement i128.

[shepmaster]

It is definitely possible for us to implement i128.

Absolutely, and I realize I didn't clearly state that I think we should implement i128 for AVR. However, any code that actually uses an i128 on an AVR is going to be in for a world of pain.

However, if we can strive for consistency, why shouldn't we?

Consistency with what, is the question. Consistency with GCC (f64 == f32) or with every other Rust target (f64 != f32)?

You are describing the #![no_std] ecosystem.

Yep, which is why I said "special features that are used to make a crate suitable for embedded, just like we do for no-std." 😇

[shepmaster]

A larger problem that's been in the back of my mind since we originally landed the 16-bit usize patch is that, fundamentally, Rust and Rust programmers tend to assume that usize is the "native" size of a register. AFAIK, this is true for all the other platforms Rust targets, but not for AVR.

[est31]

Consistency with what, is the question. Consistency with GCC (f64 == f32) or with every other Rust target (f64 != f32)?

The name f64 literally indicates that it has 64 bits. If you don't abide by the name then it becomes meaningless just like it is in C.

[dylanmckay]

Good points here, I can see the concern around 128-bit integers. I definitely think that they should be supported, even though we shouldn't encourage their usage. I would hate to see crates having to add feature flags for things like trait impls on i128 types. This shouldn't really be an issue because all of the unused i128 stuff should be trimmed out by the linker.

The f32/64 problem is an interesting one. My main concern with making f64 actually 64-bits is that it means that C FFI could be very brittle. If developers don't know that AVR-GCC uses 32-bit doubles, then calls through FFI could read uninitialised memory or segfault.

I imagine we could solve this more-or-less by expecting users to use types from the libc crate instead. We could add AVR-specific functionality to set c_double to f32. I think we can reasonably expect people to use the libc crate in their FFI bindings.

[mattico]

Something to remember for merging, need to update the c_int type used in the main() signature: https://github.com/rust-lang/rust/pull/44906#discussion_r141843808

Edit: opened an issue for this since it affects MSP430 as well: https://github.com/rust-lang/rust/issues/44934

[shepmaster]

used in the main() signature

@mattico Maybe I've just been doing things in a weird way, but none of my code has made use of Rust's main:

#[no_mangle]
pub extern fn main() {

More importantly, we can't really return because there's nothing to return to. Every program needs to run forever.

[dylanmckay]

@mattico We will still definitely have to modify libc so the types match GCC for AVR

[shepmaster]

Oh, absolutely, I just don't know that main will impact us at all.

[mattico]

@shepmaster Even on non-embedded platforms, the size of argc in main doesn't matter given that we've had it wrong all this time and nobody has noticed except when inspecting the IR. There may be some RTOS that uses a standard main() entrypoint for its processes? Regardless, it's easy enough to fix.

It probably would've taken the same amount of time to submit a fix as it did to type this out, now that I think about it 😄

[chocol4te]

Is it just the libcore issues preventing this merge? Just so we know where to concentrate efforts.

[Restioson]

The only issues I've had out of libcore are weird linker errors caused by I don't know what and also 32 bit bitshifting errors (missing intrinsic I think). I don't know if those are blocking the merge though.

[dylanmckay]

chocol4te: Is it just the libcore issues preventing this merge? Just so we know where to concentrate efforts.

Yes - all of the required work here needs to be done within LLVM.

Restioson: The only issues I've had out of libcore are weird linker errors caused by I don't know what and also 32 bit bitshifting errors (missing intrinsic I think).

That's because all of the code that makes the AVR backend choke is commented out :)

Restioson: I don't know if those are blocking the merge though.

Not directly, but it'd be good to have fixed before the backend is merged. There are a couple of really annoying bugs like this that we should consider fixing before we merge, but they may not necessarily hold it up.

[kellerkindt]

@dylanmckay LLVM6 has been merged https://github.com/rust-lang/rust/pull/47828 - what does that mean to this RFC?

[shepmaster]

@kellerkindt all of the issues listed in "Current issues in the AVR backend" are still true. It's likely that the current HEAD of avr-rust could be rebased and the interesting Rust-specific code merged, but that still doesn't get working code.

I'm personally still in favor of

I think that the best way forward would be to propose the patch for real once libcore does compile without modification, or at least without many.

Although having to avoid extra rebasing is nice.

[Jimmio92]

I wonder the current state of Rust on AVR, now that we're half of a year later in development. I run a little Arduino projects group in my town, and I would _love_ to be able to use Rust instead.

[dylanmckay]

So, good news!

I think that the best way forward would be to propose the patch for real once libcore does compile without modification, or at least without many.

This is now the case!

The current avr-rust fork does not contain any modifications to libcore.

The modifications required to support AVR from stock Rus are:

• The AVR backend initialization functions LLVMInitializeAVR{Target,TargetInfo,AsmPrinter,AsmParser, ...} are declared and called.
• An base-minimum avr target specification avr-unknown-unknown has been added. This models avr-gcc's default behavior of building for the lowest common denominator unless explicitly specified. Unlike avr-gcc which explicitly supports the -mmcu=<avr mcu name> argument, no AVR-specific command line argument has been added for AVR. This means that a custom target specification JSON file must be written for every project. This is the case for many Rust embedded projects though.
• In the fork, the AVR LLVM backend is always compiled and linked in the default Rust checkout and added to the config.toml.example file. Should AVR be included in upstream Rust by default, or should it also be opt-in?
• AVR specific logic added to compiler where required for all new targets
• The "avr-interrupt" calling convention has been added. This allows extern "avr-interrupt" fn my_isr_handler() { .. }. This would probably need to go through the RFC process to become stabilized, but I could be wrong.
• Support for a conditional compilation on CPU, a la #[cfg(target_cpu = "...")] has been added. The implementation can be found here. The implementation of this is target-independent, and thus it works for conditional compilation based on CPU for other architectures too, such as ARM. This allows the ruduino crate to conditionally include a device-specific module that exposes all of the IOs, registers, and modules that are supported in the silicon. This most definitely needs to go through the RFC process before upstream.

It's probably about time I send an RFC to LLVM-dev regarding promotion of the backend to non-experimental status.

You can see the full set of changes from upstream Rust to avr-rust here.

[dylanmckay]

There's still a couple LLVM patches from the last two months we've cherry-picked at the moment, but the upstream Rust efforts to separate the emscripten version of LLVM from the version of LLVM used for all other targets has made it really easy to bring these changes in from rust-lang/llvm repo, as it is now updated upstream regularly.

The <4 cherry-picked LLVM patches we have are currently in review in upstream LLVM, and so once a reviewer finds enough time, those patches will automatically float into upstream Rust. Upstream Rust also has target-specific Rust-specific patches, and so cherry-picked LLVM patches probably isn't really a blocker for merging avr-rust upstream

[NewbiZ]

Any update on the status of Rust on AVR?

[edvorg]

Also interested to know! For now I swtiched to hacking on STM32 blue pill, but I'd definitely want to get back to arduino once support for avr in rust is ready.

[flosse]

We @slowtec would also love to use Rust for our AVR projects and of course we would open source a lot of our production code :)

[jonahbron]

Bump, would love to see support made official. Gonna try using the fork for a project.

[dylanmckay]

Update: At the moment, we're upgrading the fork to a newer version of Rust (avr-rust/rust#137). There are two bugs we've found that we've hit.

LLVM ERROR: ran out of registers during register allocation

This has been fixed in LLVM trunk in llvm/llvm-project@45eb4c7e55341c0b83a21dedecc092e273795eda. I'm cherry-picking this into our LLVM fork now. This bug was historically the biggest pain point in the backend, arising in most pointer-heavy code.

LLVM ERROR: Cannot select: t2: i16 = addrspacecast[1 -> 0] undef:i16
t1: i16 = undef
In function: _ZN4core3ptr87_$LT$impl$u20$core..fmt..Debug$u20$for$u20$unsafe$u20$fn$LP$A$RP$$u20$.$GT$$u20$Ret$GT$3fmt17h0fdf8ca7140def9b

This one is harder to fix. It is caused by Rust mishandling function pointers on Harvard architectures where there are two separate pointer address spaces for accessing main memory and program memory.

The explanation is best made by @ecstatic-morse

The underlying issue is that *const T always has addrspace(0). I think explicit casts (ptr as *const T) should preserve the address space of their input here.

The bug is exposed by the std::fmt::Debug implementation for functions and function pointers. Rust emits a pointer dereference with a target pointer type of addrspace(0)* i16, using LLVM routines that will implicitly insert a bitcast (iN -> iM bits) or an address space cast if necessary. In the case of function pointers, Rust should codegen a addrspace(1)* i16 pointer, so that LLVM doesn't have to map (addrspacecast) PROGMEM pointers to RAM pointers an impossible task because there is no memory mapping and the address spaces do not overlap.

This bug is the main blocker.

I am hoping upstream Rust pulls from LLVM master (AFAICT, it's pretty much at the 8.0 release) so we can remove a bunch of cherry-picks.

I recently got the https://github.com/dylanmckay/avr-compiler-integration-tests/ tests succeeding using an AVR emulator, which is great because there is no other test suite that actually executes the AVR assembly spit out by LLVM. I set up a GitLab runner to run tests for every AVR-Rust commit (via a GitLab repository mirror), but it isn't super useful because GitLab doesn't support CI builds on pull requests, as the actual code is hosted in a forked repository.

[rubberduck203]

Thanks for the update @dylanmckay. We all appreciate the effort you’ve put into this.

[dylanmckay]

We've now upgraded the fork ontop of Rust base rust-lang/rust@fb7cca33f, the blink program successfully compiles.

[likeabbas]

Did this issue stall?

[H2CO3]

Do you have any updates on this? Is the PR still stalled on the address space issue?

[edvorg]

Hey @dylanmckay , sorry for bugging with this.. is there any update on the issue?

[dylanmckay]

Hello all, here's a few comments from me

Cherry-picking fixes

Almost all of the required fixed to get libcore working have been upstreamed to LLVM master and currently exist in the rust-lang/llvm fork. Yesterday I started a PR to update the AVR fork to 1.39.0 (avr-rust/rust#155), I only had to cherry-pick one fix that was not already there - avr-rust/llvm@4350776601bc671e6e055bfbe32add34b70d2635.

libcore cannot be compiled without modification

We no longer require a custom libcore fork to use AVR. There is currently only one modification to libcore in the avr-rust fork - avr-rust/rust@44240ac59c5949b8a9fd191f5cd666d0206fbe85 - rewrite a pointer cast to get the right IR generating.

We are dependent on xargo for compiling AVR-Rust binaries - this will become easier with the std-aware Cargo initiatives going on (including the working group). Currently, AVR requires a compiled sysroot for each MCU type - we use Xargo at the moment to compile this lazily, but it will be much more seamless when Cargo becomes able to natively compile core/std for desired target specifications.

Any operations on function pointers other than 'call' access RAM, not program memory (avr-rust#68)

This has been fixed.

32-bit shifts generate calls to a compiler-rt routine that doesn't exist (avr-llvm/llvm#163)

This is still a painful issue. I suspect the best fix will be to write custom lowering code in the LLVM AVR backend that lowers these shifts to pure assembly, removing any ABI dependency on compiler-rt or libgcc. I am unsure how much larger the generated code size could be, it may not be a good idea.

[dylanmckay]

Questions before upstream

RFCs and unstable calling conventions

I posted this comment up the thread

The "avr-interrupt" calling convention has been added. This allows extern "avr-interrupt" fn my_isr_handler() { .. }. This would probably need to go through the RFC process to become stabilized, but I could be wrong.

Can unstable calling conventions be added without going through the RFC process?

Currently, we have the "avr-interrupt" and "avr-non-blocking-interrupt" under the #![feature(abi_avr_interrupt)] feature gate. Could these be upstreamed as unstable calling conventions, pending a future stabilization RFC?

Buildbots

Upstreaming an LLVM backend requires setting up a dedicated buildbot that runs tests for that backend, and maintaining it. Is this the case with Rust? How do we ensure that the test suite runs in AVR mode on every push? What did other backends (like WASM) do?

Distribution

Does "Merging the avr-rust fork upstream" mean simply merging the two forks into one, but still requiring a build-from-source? Perhaps it's possible for a backend to be distributed, but nightly-only? What did the other backends do?

Other than that, the avr-rust specific patches are very thin, without any gnarly hacks nowadays.

The full patchset can be seen here https://github.com/rust-lang/rust/compare/master...avr-rust:avr-support

My WIP 1.39.0 rebase patchset (which is mostly identical) can be found here https://github.com/rust-lang/rust/compare/master...avr-rust:avr-support-1.39.0-4560ea788c. This should be merged to avr-rust/master in the next few days.

I can't think of anything specific that's blocking this - perhaps, it's time to submit patches and see how it goes?

[dylanmckay]

https://github.com/rust-lang/rust/issues/44036

I mentioned this issue above, but I don't think it blocks upstreaming of the AVR backend. We should be able to land a working, useful version of rustc with AVR without it. I am sure there are workarounds and hacks we can do re. device detection in AVR crates in the meantime after the backend has been hypothetically merged, and before an official target-cpu querying API lands in official Rust.

[jonahbron]

He lives! Thanks for the update @dylanmckay , exciting progress.

[FrankvdStam]

Great work @dylanmckay! Thanks for keeping us informed.

[edvorg]

A bit off-topic, but: would rustc for avr be able to do FFI with C libraries?
There are plenty of mature libraries already available for avr but written in C/C++.
It would be great to be able to create some rust-style wrappers for them and reuse them in our rust avr projects.

[H2CO3]

Rust already can do FFI with C in general.

[edvorg]

Yes and that's really great! The question is though: does it translate to rust-avr?

[CryZe]

As long as LLVM's C ABI for AVR matches that of gcc, it should work

[shepmaster]

Can unstable calling conventions be added without going through the RFC process?

My vote is yes, as the change to support the AVR calling conventions is basically just giving them a name and mapping them to the LLVM number. There's no fundamental code changes needed as we already have support for unstable conventions.

Upstreaming an LLVM backend requires setting up a dedicated buildbot that runs tests for that backend, and maintaining it. Is this the case with Rust? How do we ensure that the test suite runs in AVR mode on every push? What did other backends (like WASM) do?

Rust's tier system is a bit loosely defined, but I think that the right thing is to merge the AVR-specific code. Then AVR will be tier 3, which means it is not automatically built or tested, just hanging out in the code. At the very least, this means that those files will be kept up-to-date with compiler changes.

Does "Merging the avr-rust fork upstream" mean simply merging the two forks into one, but still requiring a build-from-source?

To my knowledge, it would be to submit the one commit as a PR. Our custom fork would effectively die / become a place to track AVR-specific details until it becomes a more widely-used target.

perhaps, it's time to submit patches

I think so.

[dylanmckay]

@edvorg very on topic IMO

A bit off-topic, but: would rustc for avr be able to do FFI with C libraries?
There are plenty of mature libraries already available for avr but written in C/C++.

Yes.. mostly. The AVR calling convention implemented by the AVR backend is designed to match AVR-GCC's (documented here). There are some quirks, but the LLVM patch D68524 I need to review should fix them.

Rust callsites will always be able to correctly invoke Rust or avr-clang compiled functions (as avr-clang C uses the same extern "C" calling convention implementation as the Rust frontend), and inter operation with AVR-GCC _should_ work, especially for simple function signatures, but it can choke at times (see the description of D68524 for more details).

[luqasz]

Any update on this ?
Just curious.

[dylanmckay]

Submitted request to make LLVM AVR backend an official backend - https://reviews.llvm.org/D75099

[Foo42]

@dylanmckay If accepted, what are the remaining pieces to closing this issue?

[dylanmckay]

@dylanmckay If accepted, what are the remaining pieces to closing this issue?

Technically, Rust will work with both official and experimental backends. I've raised #69478 for upstreaming of the bulk of the backend.

I am not sure if merging AVR as a Tier 3 target is predicated on AVR becoming an official LLVM backend - thoughts welcome.

Thinking about it, LLVM's official versus experimental distinction can be mapped onto Rust's tier system, of which Rust has three tiers and LLVM has two. An LLVM official backend corresponds to a mix of Tiers 1 and 2 - basically, included in test suites and official releases. An LLVM experimental backend, as far as I can tell, is semantically the same as a Rust Tier 3 backend; included in the source tree, not included in releases by default, no inclusion into the default CI tests, etc.

Then I suppose the only remaining pieces to closing this issue will be those that come up in the code review of #69478.

tl;dr the AVR backend would become a Tier 3 upstream target in rust-lang/rust#69478 if merged, thus killing the fork.

[dylanmckay]

The only difference between the avr-rust/rust fork and #69478 is the target_cpu cfg flag that exists in the AVR fork but not upstream. I have left that out of the initial PR.

[shepmaster]

what are the remaining pieces

I’m pretty sure there are still a number of miscompilations of various types of code, so it still needs more people to try things and minimize bugs and fix them.

Getting upsteamed will lower the bar to have people try things though, as they can just get a regular Rust nightly instead of compiling the usually out-of-date fork.

[est31]

Once it's upstreamed as a tier 3 target, will AVR development work with cargo's -Z build-std feature, or will it require xargo? Does the LLD that ships with Rust have AVR support, or will gnu linkers be required?

[shepmaster]

will AVR development work with cargo's -Z build-std feature, or will it require xargo? Does the LLD that ships with Rust have AVR support, or will gnu linkers be required?

I can't say those won't work, but everything I've done has used Xargo and GNU linkers.

[est31]

GNU linkers are hard to set up on Windows and xargo requires one more tool to install, that's why I'm asking.

[aykevl]

Does the LLD that ships with Rust have AVR support, or will gnu linkers be required?

LLD only has very basic support for linking AVR, it cannot link any real programs yet. Therefore, avr-ld will be required.

[shepmaster]

that's why I'm asking.

Make no mistake, I also think that is the correct end goal, it just may not be the current state of the world. Honestly, I'd be happy to make the overhead lower and lower. Getting it merged is just the first step towards that.

[dylanmckay]

The AVR backend has now been enabled as an official LLVM target :tada: LLVM master contains AVR support by default now. The first LLVM release to include AVR by default will be LLVM 11 in about 6 months time.

[FrankvdStam]

Nice! Thanks for all the effort you're putting into this.

[H2CO3]

Very cool!

[jonahbron]

Congratulations @dylanmckay on this incredible achievement. The community thanks you for your work on this.

[Jimmio92]

I've been following this issue for a while now and while I know this is mostly off topic, I'm interested what this means for the average-not-from-source-user... Is it a complicated process to get an Arduino Uno up and running with native Rust using this? I'm most interested in this as Rust has enough protections I likely can't cause nearly as many oops moments xP

[H2CO3]

@Jimmio92 From what I can tell, yes. As in "yes, it's complicated". Or at least long and tedious, even if not particularly complicated, since you still have to build this custom fork of the Rust compiler from source, which also entails building LLVM from source. That usually takes a couple of hours, if everything goes well.

_{My personal experience with LLVM is, however, that more often than not, it just doesn't like something on your system (at least certainly on macOS – it's easier on Linux and certainly don't even dream of doing this on Windows), so you inevitably end up massaging install paths and custom libraries until you either succeed or give up.}

What the (excellent) news above mean is only this: the developers of LLVM are now targeting/supporting AVR officially (?), and the next release of LLVM will include AVR support. There's nothing to say that Rust now also supports or will support AVR in the short term. If you look at the issues and the above comments, you can see that it's going to take a lot of work till AVR support "just works" with a stock Rust toolchain.

[dylanmckay]

Is there a process/example that I could look at cherry-picking AVR fixes from LLVM master into the Rust LLVM fork?

Also, does Rust still track LLVM master? From the looks of it, it tracks the current release plus some cherry-picked fixes. Are PRs to update the LLVM version to master still accepted?

[mati865]

@dylanmckay

Also, does Rust still track LLVM master?

As you have noticed it tracks releases now.

Are PRs to update the LLVM version to master still accepted?

I don't think so but cc @nikic

[est31]

@dylanmckay there are a bunch of fixes on the LLVM fork of Rust compared to the 10.0 release: https://github.com/rust-lang/llvm-project/commits/rustc/10.0-2020-05-05

[nikic]

Is there a process/example that I could look at cherry-picking AVR fixes from LLVM master into the Rust LLVM fork?

There are instructions available here: https://rustc-dev-guide.rust-lang.org/backend/updating-llvm.html#bugfix-updates

Also, does Rust still track LLVM master? From the looks of it, it tracks the current release plus some cherry-picked fixes. Are PRs to update the LLVM version to master still accepted?

Rust currently tracks LLVM releases, because this allows people to easily perform cross-language LTO with a corresponding clang version. So the default answer here is probably "no".

That said, there has been some discussion on updating to LLVM master to mitigate LLVM 10 compile-time regressions. I'm currently working on that for evaluation purposes. I think it's safer to assume that this won't happen though, and submit cherry-picks against our LLVM 10 fork (are there a lot of them?)

[dylanmckay]

The fork has been merged! I will write a proper update this afternoon

https://github.com/rust-lang/rust/pull/69478

[wezm]

I will write a proper update this afternoon

I am interested in said update. Where do you plan to post it @dylanmckay (so I know where to look :slightly_smiling_face:)?

[nixpulvis]

I've been eagerly awaiting this addition to the Rust ecosystem for literally years now! My naive attempts to use this on master seem flawed however.

I built master rust with the standard ./x.py build, linked as toolchain master, then attempted to build the avr-rust/blink example (after updating src/main.rs to use llvm_asm!):

RUST_TARGET_PATH=`pwd`
XARGO_RUST_SRC=/home/nixpulvis/Projects/rust

rustup run master xargo build --target avr-atmega328p --release

This fails with:

error: no matching package named `core` found
location searched: registry `https://github.com/rust-lang/crates.io-index`
required by package `sysroot v0.0.0 (/tmp/xargo.oXlxlujoXvXJ)`
error: `"cargo" "build" "--release" "--manifest-path" "/tmp/xargo.oXlxlujoXvXJ/Cargo.toml" "--target" "avr-atmega328p" "-p" "core"` failed with exit code: Some(101)
note: run with `RUST_BACKTRACE=1` for a backtrace

I'm assuming there's configuration still required which I'm missing.

[shepmaster]

@nixpulvis see https://github.com/rust-lang/rust/issues/44052#issuecomment-591396417

I wouldn’t expect much of anything to work now as libcore is part of what is miscompiled.

[dylanmckay]

The AVR rustc fork has been merged upstream.

The upstream fork is not usable in general yet. There are a few things that need to happen first.

AVR LLVM upstream fixes that Rust doesn't have yet need to be cherry-picked

To answer @nikic's question, there are a maximum of 20 LLVM commits that need to be cherry-picked.

I have been using a script to automatically cherry-pick _all_ AVR commits from LLVM upstream into a local fork.
This is a large hammer and there will be cherry-picked fixes that are unnecessary - there are upstream LLVM fixes that need to be pulled to upstream Rust before the backend becomes usable. they will need to be filtered out and cherry-picked to upstream Rust.

Compiling the AVR LED blink program from the upstream Rust master branch

As the upstream effort has been going on for a few months now, the downstream avr-rust fork
is still on a particularly old version of Rust. Since then there has been at least one change in upstream Rust that requires generalization to support AVR without hitting LLVM assertion errors. I've opened pull request #73270 to fix it.

I am not sure if there have been other changes in upstream Rust which need changes for AVR - likely not.

Once I've gotten the blink program working, I will post another update because at that point, the upstream AVR support should be ready for use/experimentation.

Where should AVR communication occur/be posted

@wezm you raise a good point - there isn't necessarily a "blessed" communication channel for updates. This ticket has served well, but it will be inevitably closed soon.

The AVR-Rust Gitter channel is the most natural existing choice. I like the idea of a mailing list (but I don't like the idea of hosting a mail server if possible). The Gitter channel does have discussion in it so perhaps something like a mailing list would be helpful.
Perhaps setting up a blog would be good to centralize announcements/updates. I would like to have an "official medium" so that people who want casual updates don't miss out. I suppose a blog requires constant checking for new posts, whereas a mailing list inherently notifies
everybody who is interested. Suggestions welcome.

Open questions

• Should GitHub issues on the avr-rust fork be moved to the upstream Rust repository?
  ** Regardless, new issues should be raised on the upstream repository - the old issue tracker will need to be wound down.
  
  
  
  • Where should the instructions for compiling Rust with AVR live?
  
  

The most important stuff for the future (read: priorities)

• Packaging and distribution of AVR Rust binaries. Compiling Rust can be a big headache to the casual user,
  many distros seem to have issues from time to time in certain workflows. Several OOM errors misreported as
  compiler bugs. The barrier to entry is unnecessarily high and needs to be lower - just download and go.
  Tracked by https://github.com/avr-rust/rust/issues/162
• Building a list of "supported", or at least tested, configurations. We need a table of (rustc version, Xargo version) so
  that changes to private Rust APIs that Xargo depends on do not break the AVR Rust toolchain upon upgrade.
  Xargo is in maintanence mode - cargo-xbuild seems quite limited (I tried it a few weeks ago, I can't remember the reason),
  we may need to fork Xargo. Perhaps better to add the tooling directly into Cargo (there is a tracking issue for this).
• Setting up some kind of homepage for the project, including links to download
• Integration testing (on an emulated/simulated AVR, upon each commit)

For the moment, I will make sure to repost updates into this GitHub issue, until a better medium can be found. I will post them in Gitter too.

[wezm]

Perhaps setting up a blog would be good to centralize announcements/updates. I would like to have an "official medium" so that people who want casual updates don't miss out. I suppose a blog requires constant checking for new posts, whereas a mailing list inherently notifies
everybody who is interested. Suggestions welcome.

My personal preference is strongly in favour of a blog (with an RSS feed). I think blogs typically show up better in search engine results and are nicer to link to than mailing list threads. The RSS feed solves the checking/notification aspect.

[rubberduck203]

I’m not 100% sure it’s the best place for it, but there is the Embedded WG blog. Might be a low effort channel for communication.

https://rust-embedded.github.io/blog/

[aykevl]

Perhaps also a Twitter account? It can be used to share new blog posts (to stay up to date).

[jonas-schievink]

I think the embedded working group would be happy to help here. We have a Twitter account @rustembedded and can certainly include AVR-related news in the newsletter too.

I've also created the O-AVR label a while back, so feel free to use the Rust issue tracker for AVR-specific issues too (but be mindful that there are 1.5k people watching the repo). In addition to that, you might want to coordinate on the rust-lang Zulip, since that's where most of the Rust teams reside. We're also in the process of ramping up "target groups" that focus on specific Rust targets (eg. Windows or Arm), AVR might be a good fit for that as well. Feel free to reach out on Zulip for this.

[dylanmckay]

Update.

There are two things remaining before libcore can be compiled for AVR on a stock rust-lang/master branch:

1. There is one pull request that fixes a series of "address space cast invalid" bugs on AVR, relating how function pointers for Harvard architectures must be tagged as addrspace(1) compared to Von-Neumann targets which are fine with Rust's current default of addrspace(0). rust-lang/rust#73270 - it is currently in code review.

~2. There is one other bug blocking the blink example from working - avr-rust/rust#92. There is a not-yet-upstreamed patch which will fix it - https://reviews.llvm.org/D68524. This will be upstreamed and then cherry-picked into Rust's LLVM fork very soon once it passes against the unit and integration tests.~ it has been merged into upstream LLVM

Once these two things are done, the upstreamed Rust AVR target will be able to compile AVR code to the same level as the current avr-rust/rust fork, and we can then begin the process of updating the blink example, arduino crate, documentation, guides, for the upstream branch. Then the upstream branch should be ready for experimental use.

[dylanmckay]

One more TODO:

1. ~Add AVR plumbing for the new inline assembly syntax in upstream Rust~
   
   
   
   • ~Once the new inline assembly syntax is supported, the Arduino crate must be updated to use it
     
     once the Arduino crate has been updated, update the blink crate to the new version.~ We can use the old macro via llvm_asm! for now as it still exists in nightly.
   
   
2. Cherry-pick AVR correctness patches (mostly Ayke's) from LLVM master to upstream Rust LLVM fork (EDIT: PR: https://github.com/rust-lang/llvm-project/pull/66)

Here's the branch with everything: https://github.com/dylanmckay/rust/commits/dylan/avr-workable-upstream-candidate. This branch is sufficient to compile libcore.

It is built from upstream rust-lang/master but also includes the not-yet upstreamed LLVM patch D68524, not-yet-merged Rust PR rust-lang/rust#73270, and 46 AVR related commits from upstream LLVM automatically cherry picked via script (LLVM branch: https://github.com/dylanmckay/llvm-project/commits/dylan/avr-workable-upstream-candidate). Note that not all 46 are required, the final list will be smaller.

[dylanmckay]

Update: Opened a pull request that includes all of the upstream LLVM fixes.

Here is an exhaustive list of the remaining work before the LED blink program can be compiled and run successfully.

Remaining work

• [x] Land Rust PR rust-lang/rust#73270. This pull request fixes a series of "address space cast invalid" bugs on AVR, relating how function pointers for Harvard architectures must be tagged as addrspace(1) compared to Von-Neumann targets which are fine with Rust's current default of addrspace(0). It is currently in code review.
• [x] Land Rust LLVM PR https://github.com/rust-lang/llvm-project/pull/66. This pulls in all of the required AVR fixes from upstream LLVM into Rust's LLVM fork. In the end, there were ~17~ 16 LLVM fixes that needed to be cherry-picked.
• [x] Land Rust PR rust-lang/rust#73658 that updates the version of the LLVM submodule. The Rust compiler docs have instructions for doing this. It is currently in code review.
• [x] Land Rust PR ~rust-lang/rust#74625~ rust-lang/rust#74631. In 79a42e37084d0 Rust started unconditionally passing the --eh-frame-hdr argument to the linker. The AVR-GCC linker does not support this flag and so a special case to avoid passing it is added for AVR, similar to the existing exclusions for Windows. Solaris, and UEFI.

[nixpulvis]

Perhaps better to add the tooling directly into Cargo (there is a tracking issue for this).

I'm assuming this is the correct issue: rust-lang/cargo#4959.

[shepmaster]

cargo build -Z build-std=core has worked well for my AVR example cases.

[nixpulvis]

@shepmaster that seems to be getting me closer anyway, thanks! I just seem to be stuck on the bitcast stuff now, so I'll wait for that to be merged in (since I appear to be missing a file needed to build the PR, and IDK what I'm doing).

In the process of using -Z build-std=core I needed to supply a target triple, so I ran rustc +master --print target-list | grep avr, and found avr-unknown-unknown. But the archived issue avr-llvm/llvm#35 seems to make it sound like the triple should in fact be avr-atmel-none (which makes more sense to me anyway). Does something need to be updated here, or am I missing something?

[shepmaster]

avr-unknown-unknown is correct.

[PointyFluff]

Open questions

* Should GitHub issues on the avr-rust fork be moved to the upstream Rust repository?
  ** Regardless, new issues should be raised on the upstream repository - the old issue tracker will need to be wound down.

* Where should the instructions for compiling Rust with AVR live?

I don't think this matters too much on the user side. This was easy enough to find through ddg and/or this-week-in-rust.
Whatever makes it easier for the devs.

[trembel]

Open questions

* Should GitHub issues on the avr-rust fork be moved to the upstream Rust repository?
  ** Regardless, new issues should be raised on the upstream repository - the old issue tracker will need to be wound down.

* Where should the instructions for compiling Rust with AVR live?

I don't think this matters too much on the user side. This was easy enough to find through ddg and/or this-week-in-rust.
Whatever makes it easier for the devs.

I think the instructions for compiling Rust with AVR should be somehow in https://docs.rust-embedded.org/

[dylanmckay]

Alright, update time.

All of the required patches for AVR exist in the current Rust nightly compiler as of today's nighty rustc 1.47.0-nightly (0820e54a8 2020-07-23). The Rust nightly compiler, without modification, can now compile the LED blink example successfully and generate an AVR ELF file.

• New, centralized project landing page created at https://avr-rust.com/
• A new book - The AVR-Rust Guidebook has been created, hosted on GitHub pages at book.avr-rust.com.
• The avr-rust/rust fork repository has been deprecated. The repository has not yet been archived as there are existing issues that should be migrated before they are permanently locked and closed.
• Xargo is no longer required - the -Z build-std flag in upstream Rust replaces the need for it on AVR. A cargo fork is no longer required - upstream cargo will do.

The Rust nightly compiler can now be considered the recommended channel for Rust with AVR support.

I am closing this issue now - we did it!

Steps for reporting bugs can be found in the AVR Guidebook.

The AVR Guidebook and the blink example at https://github.com/avr-rust/blink are the best resources to start using the target.

A deep, deep thanks to everybody who discussed and supported the project through this upstreaming effort - it is very appreciated.

FIN

[H2CO3]

Wow wow wow.

Thanks for everyone who contributed to this – I've been looking forward to this day since forever!

[edvorg]

Dylan McKay before porting rust to avr

And after

Thank you for all the hard work man! :-) Take a good rest!