Build: 32-bit Armbian for NEO 2?

[ThomasKaiser]

Today I did a simple test on Pine64: Running a NodeJS benchmark one time with armhf and one time with arm64 binaries. Performance numbers more or less the same but huge difference regarding memory consumption: https://github.com/nodesource/distributions/issues/375#issuecomment-290440706

Since I totally missed 64-bit integration in our build system I simply ask: how much efforts would it be to debootstrap armhf variants to be combined with 64 bit mainline kernel? If it's rather easy it would be really interesting to create both such variants, measure performance beyond stupid benchmarks and monitor some other workloads also using vmstat 10 especially on NEO 2 with its limited memory.

Any thoughts or objections?

[zador-blood-stained]

Changing the build scripts will be the easy part, but not the first one.

We need to start by answering a set of questions to make a roadmap for this "feature":

• Do we want to provide armhf variant for all 64-bit boards or only to some of them (i.e. based on DRAM size)?
• Do we want to provide only server 32-bit images, only desktop or both?
• Do we want to provide legacy and mainline or only mainline 32-bit images?
• What do we do with OS image file names?
• How do we implement this on the download pages? We have up to 8 image variations per board already not counting dev kernel images (legacy,mainline)x(server,desktop)x(jessie,xenial). Adding more download buttons for the God of download buttons or putting only 4 visible buttons (legacy,mainline)x(server,desktop)x(xenial) and hiding others under the spoiler (jessie based, 32-bit for 64-bit boards)
• How would inexperienced users choose between one or the other on the download pages? Several pages of benchmarks and walls of text is not a good idea, so the "guide" needs to be short and simple.

[ThomasKaiser]

Well, I would postpone all of the above questions for one simple reason (now that you answered 'Changing the build scripts will be the easy part'): doing some research first. I searched the net but did not found any ressources analyzing memory useage and real-world performance.

But creating such a test OS image can also be done manually (throwing together userland of an H3 OPi Zero image with u-boot/kernel for a H5 image). My problem is that I don't have any ARMv8 board with just 512 MB currently (waiting for an answer regarding Xunlong samples but maybe getting back to FriendlyELEC to ask for one NEO Plus 2).

[zador-blood-stained]

My problem is that I don't have any ARMv8 board with just 512 MB currently (waiting for an answer regarding Xunlong samples but maybe getting back to FriendlyELEC to ask for one NEO Plus 2).

I believe it should be easy enough to "convince" a mainline based Pine64 image that it has only 512MB memory available. We already played with limiting memory size for legacy based images for testing 2GB Mali issue on H3 (though it was the legacy kernel and a special debug kernel option), and I believe that mainline u-boot passes memory info to the mainline kernel in the DT.

[ThomasKaiser]

I believe it should be easy enough to "convince" a mainline based Pine64 image that it has only 512MB memory available

Any guidance on this?

In the meantime I also thought about your questions above. At least I don't want to care about BSP kernel and desktop stuff at all. I will only waste time here to get an idea whether server use cases on those little 64-bit boards might benefit from running armhf binaries/userland or not.

Also zram needs 3.14 or above (3.15 even better) so we end up with a) mainline kernel as a requirement and b) a clear 'do not buy' recommendation for 64-bit boards with 512MB DRAM that might 'need' legacy kernel (so we can skip H5 Orange Pi Zero Plus 2 already entirely ;) )

In fact I think this whole 32-bit thing has only some relevance for 64-bit boards that might be used as servers. Then we're talking about NEO 2 and NEO Plus 2 (which might be able to share one image with the documented requirement to do something like echo 'fdtfile=sun50i-h5-nanopiplus2.dtb' >> /boot/armbianEnv.txt on the Plus 2) and in case Xunlong releases another H5 Zero thingie with (Gbit) Ethernet then for this device too. All 4 images based on mainline, Jessie/Xenial each in 32- and 64-bit flavour.

We don't sell anything and especially no hardware so we can base our decisions on a 'use case' point of view fortunately :)

Edit: FriendlyELEC has zram enabled in their BSP kernel.

[zador-blood-stained]

Any guidance on this?

I would try patching arch/arm/lib/bootm-fdt.c first, here

ret = fdt_fixup_memory_banks(blob, start, size, CONFIG_NR_DRAM_BANKS);

[ThomasKaiser]

Hmm... in the meantime I changed my mind especially after thinking about zsmalloc/zram and this bit from Willy Tarreau:

I checked the RAM performance. I’m measuring about 55ns per 64-bit word (4 consecutive 16-bit reads) with the stock kernel, and 41ns with the Armbian 4.10 kernel.

It would be somewhat stupid to ignore memory bandwidth on those single bank DRAM configs when starting with tests at this level (zram 'performance'). So I need a NEO Plus 2 (not NEO 2 since I also need DVFS and especially the 1.3V back to be able to test what we gain simply by letting H5 run armhf binaries at 1250 MHz to compare with H3 NEO running at 1.2GHz max). Will drop Yuefei a note after I generously refused 'accepting' board samples last week ;)

[zador-blood-stained]

Are you more concerned about performance or memory footprint? Even though arm64 binaries are bigger in size, there are also kernel Kconfig settings that may be possible to turn off (they are currently turned on in sun50i-dev config)

config TRANSPARENT_HUGEPAGE
    bool "Transparent Hugepage Support"
    depends on HAVE_ARCH_TRANSPARENT_HUGEPAGE
    select COMPACTION
    select RADIX_TREE_MULTIORDER
    help
      Transparent Hugepages allows the kernel to use huge pages and
      huge tlb transparently to the applications whenever possible.
      This feature can improve computing performance to certain
      applications by speeding up page faults during memory
      allocation, by reducing the number of tlb misses and by speeding
      up the pagetable walking.

      If memory constrained on embedded, you may want to say N.

choice
    prompt "Transparent Hugepage Support sysfs defaults"
    depends on TRANSPARENT_HUGEPAGE
    default TRANSPARENT_HUGEPAGE_ALWAYS
    help
      Selects the sysfs defaults for Transparent Hugepage Support.

    config TRANSPARENT_HUGEPAGE_ALWAYS
        bool "always"
    help
      Enabling Transparent Hugepage always, can increase the
      memory footprint of applications without a guaranteed
      benefit but it will work automatically for all applications.

    config TRANSPARENT_HUGEPAGE_MADVISE
        bool "madvise"
    help
      Enabling Transparent Hugepage madvise, will only provide a
      performance improvement benefit to the applications using
      madvise(MADV_HUGEPAGE) but it won't risk to increase the
      memory footprint of applications without a guaranteed
      benefit.
endchoice

[ThomasKaiser]

Are you more concerned about performance or memory footprint?

The relationship between both on 64-bit devices suffering from 'low memory' conditions as it's the case with NEO 2. And yes, we should collect such config options to test through later :)

[daiaji]

I think this is useful
Arm64 runtime consumes more memory
As @ThomasKaiser demonstrated, there is even twice as much memory as armhf
My nanopiT3 has only 1GB of RAM
This gives me an intermittent flashback when I run tshock
In fact, most ARM boards have less than 1GB of RAM
Even a 32-bit memory address space is fully adequate

And 64-bit kernel does not seem to have brought any significant benefits

I don't think a board with more than 4GB of RAM should have a 64bit kernel