Tinygo: From Go directly to ARM microcontroller

[Chillance]

Hi,

I've been wondering about something here. Go has support to build for the ARM architecture that some micro-controllers has. Such as Cortex M4 which is ARMv7 (ARMv7E-M).

Why isn't the Go code just compiled straight to ARM binary, and then flashed into a certain position in flash memory where it's suppose to be run at?

I was thinking that using for example the https://wiki.makerdiary.com/nrf52840-mdk/ board, which has extra external 64-Mb QSPI FLASH memory, wouldn't it just be super cool and easy to just do that? Compile to ARM binary, flash into position, and run it like that. Binary would be a few MB and surely fit, right?

Maybe it will use too much resources, and would need better CPU clock? And a library layer to talk to the peripherals (like is done with tinygo I suppose)

Why wouldn't such a thing work?

Thanks!

[aykevl]

Of course, I've thought about the same ;) it would have been a lot less work to just use that compiler.

Technically, it might be possible but will require a lot of work. Among others:

• The standard Go compiler does not support the thumb2 instruction set. And if it does, it probably won't support the details of the Cortex-M4, which probably doesn't support all instructions that are supported on a Cortex-A.
• There is no support for volatile loads and stores in the compiler, as you generally don't need those except for very specialized algorithms on a desktop system. TinyGo relies on them for memory-mapped I/O. You could use sync/atomic for that but it will be slow.
• The normal runtime expects to be running on a regular operating system, with lots of RAM and virtual memory.

I doubt the Go maintainers are much interested in supporting such "low-level" chips with "just" 256kB of RAM.

Also, can you actually run programs from that 64Mb flash chip? Often, external flash is not memory mapped and if it is it may not be executable.

For some more information, see: https://groups.google.com/forum/m/#!topic/golang-dev/JGrRegAfW18
Also related, reusing parts of libgo: https://github.com/aykevl/tinygo-gccgo

[Chillance]

It would have been cool, but I suppose this makes tinygo even more useful then. I would really like to see this project go very far, where as you can essentially do everything in Go. For just doing your own small program, tinygo has that supported pretty well already. It just needs more contributors and more boards to become even bigger.

It would be even cooler if one might be able to start writing Wifi/BLE stuff in Go. Lower layers/drivers and then more higher level code using these layers/drivers. I2C and SPI is still not ready right? Using those kind of things, enables a lot of functionality too.

And how about lower level micro-controller support to create, say, HID devices using tinygo? I mean, where you basically can specify the USB descriptors etc needed to do all that USB communication stuff on the lower levels. Basically, I would just love to see this project making all this possible.

I'm not sure you can run programs on that external flash memory yet. I asked the support of the board to clarify this, but no response yet. If not used to remap memory, I'm not sure what the purpose of it is.

[deadprogram]

Hi @Chillance it would be really helpful to have the ble support, but as @aykevl has pointed out that will take a fair amount of effort. One possible shortcut would be if TinyGo can use C libraries so the NRF wrappers could be used.

As far as I2C support, the AVR platform already has this implemented, and I am currently working on adding the same support for NRF. One that is working as expected, adding the SPI support should be mostly the same implementation pattern. On that note, it seems totally possible to implement HID but that will require a bit of effort that is probably pretty platform specific.

Regarding the flash RAM, I think generally it is intended to provide data storage that can survive power-cycle operations. For example http://infocenter.nordicsemi.com/index.jsp?topic=%2Fcom.nordic.infocenter.nrf52832.ps.v1.1%2Fuicr.html&cp=2_1_0_13&anchor=concept_rnp_grp_xr

[deadprogram]

As a minor update to the last post, I have a working I2C for NRF now ready to submit a PR.

[aykevl]

I would really like to see this project go very far

I'm kind of afraid it might actually have the potential to get quite popular...

Regarding available libraries: yes that's a big limitation right now and of course a limitation for any new language on microcontrollers, see this post about Rust microcontroller drivers.

One possible shortcut would be if TinyGo can use C libraries so the NRF wrappers could be used.

There is limited support for CGo although it isn't currently used so I'm not sure in what shape it is. Also, I haven't actually tested it on bare metal devices. If it works, you can call C functions by including their header files in Go and compile the Go sources to an object file to link with the object files from the C source files.

In general, I think Go provides the perfect opportunity to develop universal drivers that work on most/all chips with the given peripheral. It would be so much easier for porting. Consider for example how many Arduino libraries there are, I think for a large part because Arduino defines a common API for peripherals like GPIO, I2C, etc. that work on all supported chips. That's why I'm a bit wary of chip-vendor defined HALs.

For Bluetooth support, we might be able to use the Nordic SoftDevices which are binary blobs with a full BLE stack. They can be called with SVCalls which are like system calls. I'm hoping it is possible to convert the required .h files into .go files. On the other hand, using a SoftDevice complicates the flashing process somewhat and an open source OS is much more configurable / debuggable so I'd generally favor an open source stack, if we can get it to work.

But before BLE happens, I'd rather focus on providing a good base with common peripherals and maybe extending Go language support so that the fmt package works (which is still a lot of work).

[Chillance]

I suppose you are afraid of having too much to do if this becomes popular no? :)

Personally, I would also like to see as much as possible here to be made in Go. Because of the portability and ease of it. Maybe even a flash tool could be made in Go one day. That could be nice. I just strongly dislike having to mess around with dependencies and tools to build things. This is why I love Go so much myself. If there is no CGo, building binaries is very easy to build and to cross-compile. At least for Mac/Linux/Windows.

I suppose we won't get away from using LLVM, which is written in C++. And the gcc arm compiler is probably also needed. I haven't tested doing this in Windows, but I could imagine it's more of a hassle to get going with tinygo there. Just because of the LLVM and compiler needed there.

I think this project is a good approach though. emgo is nice, but dumping it all out to C and compile that doesn't sound very optimal. And has it's limits too as I understand it regarding Go features support and optimizations.

By the way, I recall the Go team used tools to convert .c code to .go at some point for Go toolchain. Sure, they had to optimize it afterwards, but still. Could be something that can be done with C code in the microcontroller stack too, to get that over to Go maybe.

And I agree with starting with common peripherals and extend Go language support with this project. With these things in place and growing interest, which most likely will come by that, BLE and such can come then. This project is still very early stage I'd say. But I can certainly see the huge potential with it.

[polarathene]

@aykevl

Regarding available libraries: yes that's a big limitation right now and of course a limitation for any new language on microcontrollers, see this post about Rust microcontroller drivers.

Just saw this thread from the linked issue and read through it. I remember looking for an alternative to C/Arduino in 2016 and coming across Go and Rust. I've been following Rust progress, interesting to see efforts on Go as I thought that was somethign Go was no longer suitable for anymore due to a change in direction.

Is there any advantages that Go would offer compared to Rust for embedded? Or is it mostly language preference?

In general, I think Go provides the perfect opportunity to develop universal drivers that work on most/all chips with the given peripheral. It would be so much easier for porting.

Similar would apply to Rust if I'm not mistaken? Or is the approach tinygo is using differing from the one with Rust quite a bit? It looks like a GC will be used which might reduce reliability for any programs that need to be deterministic?

Is tinygo a runtime separate from the program? Or a compiler separate from mainline Go? That's the impression I get from the start of this issue?

[aykevl]

Replying to @Chillance:

I suppose we won't get away from using LLVM, which is written in C++. And the gcc arm compiler is probably also needed. I haven't tested doing this in Windows, but I could imagine it's more of a hassle to get going with tinygo there. Just because of the LLVM and compiler needed there.

LLVM is required as TinyGo is written on top of it. However, I'd like to get rid of GCC and binutils at some point, which should be doable for ARM targets. AVR targets will need at least libgcc until compiler-rt supports AVR (which may take a while).

By the way, I recall the Go team used tools to convert .c code to .go at some point for Go toolchain. Sure, they had to optimize it afterwards, but still. Could be something that can be done with C code in the microcontroller stack too, to get that over to Go maybe.

Yes, they did. However, I would prefer to not go down that rabbit hole and instead make sure CGo works as expected (without any of the regular CGo overhead).

Replying to @polarathene:

Is there any advantages that Go would offer compared to Rust for embedded? Or is it mostly language preference?

It's mostly language preference. Rust is certainly a nice language but not exactly for beginners. I have programmed very little in Rust and I like Go so that's the main reason I started this project. Also, Rust appears to require much more boilerplate that may be avoided in Go.

A few reasons why Go might be preferred:

• Go has lightweight threading built into the language, which is basically like a RTOS built into the language. However, goroutines have only been partially implemented in TinyGo and are not yet optimized.
• Go is "batteries included" with a heavy standard library implementing lots of common functionality (however most packages cannot yet be imported by TinyGo due to missing language features).
• There are lots of existing packages for Go that just work, while Rust generally needs special "no_std" packages (AFAIK).

Of course, Rust has other advantages.

Similar would apply to Rust if I'm not mistaken? Or is the approach tinygo is using differing from the one with Rust quite a bit?

Yes, the same things hold true for Rust and Go. Both are new languages and thus are a good chance to define sane driver interfaces for portable code. All C drivers I've seen so far are either tied to a particular hardware platform or to a particular RTOS.

It looks like a GC will be used which might reduce reliability for any programs that need to be deterministic?

Yes, a GC can make a program less reliable (but also more reliable in some cases as memory leaks are mostly gone). However, carefully written code can avoid memory allocation in hot paths or even the entire main loop. At the moment, no GC has been implemented yet so this is even more important right now (you should only allocate memory in setup, not in the main loop).

This is similar to the issues in MicroPython.

Is tinygo a runtime separate from the program? Or a compiler separate from mainline Go? That's the impression I get from the start of this issue?

TinyGo is a new compiler and a new runtime implementation. However, it is able to import packages from the standard library and from GOPATH just like standard Go.

[erikwilson]

This project looks very promising, if there are any parts that you would like help with regarding code contributions please let us know what your main needs are at the moment.

Mainly I am interested in this for the ESP32, NodeMCU was an awesome project to explore using Lua on a micro-controller, would like to see something similar with Go.

[aykevl]

This project looks very promising, if there are any parts that you would like help with regarding code contributions please let us know what your main needs are at the moment.

A few ideas:

• Implementing cross-chip/arch peripheral support in the machine package (see src/machine).
• Adding new boards/chips.
• Reporting on which features are missing to be usable ;)

Mainly I am interested in this for the ESP32, NodeMCU was an awesome project to explore using Lua on a micro-controller, would like to see something similar with Go.

Sorry to disappoint you, but that is not going to happen anytime soon. There is (as far as I'm aware) no usable Xtensa backend for LLVM, which is used in the ESP chips. I would love to see support for these chips (I have a few myself), but writing a compiler backend is a huge undertaking.
See also: http://lists.llvm.org/pipermail/llvm-dev/2018-July/124789.html

[deadprogram]

Seems like this conversation is very interesting, but no longer active nor having a specific destination. Also a number of the things discussed have now been implemented. I propose we close this issue and open new specific ones as needed for further implementation details.

[aykevl]

Yes, I think the original question has been answered. I have answered many of the questions raised in this thread here:
https://tinygo.readthedocs.io/en/latest/faq.html

[achton]

There is (as far as I'm aware) no usable Xtensa backend for LLVM, which is used in the ESP chips. I would love to see support for these chips (I have a few myself), but writing a compiler backend is a huge undertaking.

@aykevl I just saw that Espressif is actively working on adding support for the Xtensa arch to LLVM (https://esp32.com/viewtopic.php?t=9226&p=38466). It's based on the 6.x branch and currently works for ESP32.
Their upcoming additions are also interesting:

• rebasing on the upstream version of LLVM, following the new monorepo layout (https://github.com/llvm/llvm-project/).
• object code generation
• support for LX106 target (ESP8266)

I'm not an expert in Go nor LLVM, but thought you might like this news :)

EDIT: Disregard this, I see that you are well aware of this news already!

[aykevl]

Thank you for the heads up anyway!
In any case, the docs should probably be updated with these developments, noting that there is a backend but that it needs some work to be useful.