Pandoc: Can Binary Releases be Done for aarch64 and armv7h

[stormdragon2976]

Aarch64 and Armv7h machines are becoming more and more common place now days. There are Windows, Mac, and Linux binaries, but the Linux binaries do not cover these architectures. Please add binary releases for them.

[jgm]

stormdragon2976 [e-mail] writes:

Aarch64 and Armv7h machines are becoming more and more common place now days. There are Windows, Mac, and Linux binaries, but the Linux binaries do not cover these architectures. Please add binary releases for them.

The latest ghc bindist I've found for arm architecture
is ghc 7.10; debian has 7.8.x; and recent versions of
pandoc require ghc >= 8.

I'm sure there's some way to do this. Hopefully
someone who needs this can put in the time to figuring
out how, and report back here. We could then consider
making it part of the release process.

[tarleb]

AFAIK, both @vmchale and @2i3r managed to cross-compile pandoc. Maybe they have any insights on how we could support these platforms?

See also hslua/hslua#73.

[2i3r]

I have my armv7h cross-compiler setup running. unfortunately currently I don't have access to an aarch64 platform to test, but as far as I can say, armv7h binaries will run on most chips providing aarch64 such as armv8. (link and link). so it might be enough to provide only armv7h binary.
I would love to share or help on this issue.

[2i3r]

On my tries direct compilation of pandoc and ghc, was so much slow. so I cross-compiled pandoc using cabal cross-compiler, which by itself needed binutils, libffi, ghc and pandoc dependecies cross-compiled, and then Simple build type in pandoc.cabal and disabling Template Haskell flag during compilation was needed.
Someone may manage to run Template Haskell in cross-compile mode, so Simple build type may not be mandatory and data files would be embedded in the output binary.

[mb21]

@2i3r do you think it would be hard to set this up on CircleCI (or similar), which we currently use for the pandoc Linux builds?

[2i3r]

I don't have any experience with CircleCI or similar tools.
In the method I described above, what makes armv7h release is an special binary of cabal and ghc which configured to cross-compile binaries, and there are backend tools such as libraries, headers, utils and c compiler (multiple GBs) it uses. It can be configured inside a docker or a PC, then I think build tools such as CirckeCI might be set up to use those configured compilers and then if errors handled correctly then it might work as expected.

[ickc]

@2i3r, how much time such build would normally take? One concern to use CI to build is the time limit. In the past people has tried using QEMU in CI but it becomes too slow to be manageable.

Otherwise, especially when a docker can be set up, it should be easy to do it in CircleCI.

[rhenwood-arm]

I'm also interested in this activity. Drone.io have builders for AArch64 and ArmV7. If that is of interest, I could begin to work on this with a colleague.

[jgm]

@rhenwood-arm - since pandoc is open source, nothing stops you from building it and making the binaries available. I think that would be appreciated!

Whether we want to provide officially endorsed ARM binaries here (on the GitHub releases page) is another question. For now, I'd be happy to have a third-party source for them.

[ickc]

My experience on directly compiling on ARM:

For armv7l, I'm trying to provide it as an unofficial effort here: https://github.com/ickc/pandoc-arm/releases. Probably will fall in pandoc-extras later. (By the way, it took Raspberry Pi 4 about 2.8 GiB of RAM and 3 hours to compile.)

For aarch64, I have trouble getting stack/cabal compiled. If I'm not mistaken it is not supported.

If someone can give me the exact command of cross compiling pandoc targeting the 2 platforms, I can try to build them and test it on those arch.

In the long run, if it doesn't take too much time to compile with CI, and if there's a way to run tests on the target arch (current tests are run using cabal/stack in the compiling stage), then may be it can be provided as an official release.

By the way, am I right that for the current released binaries, the Windows' come from CI, but he macOS/Linux builds come from @jgm directly? I think this might be true in the past but not sure any more.

[jgm]

By the way, am I right that for the current released binaries, the Windows' come from CI, but he macOS/Linux builds come from @jgm directly? I think this might be true in the past but not sure any more.

Correct, but this may soon change. I've been working on getting releases built for all three architectures with GitHub Actions CI. See the rc/test branch.

[vmchale]

@stormdragon2976 if you want binaries, I have some here: https://github.com/vmchale/ghc-cross/releases/tag/Binaries

I also have linux -> linux cross-compilers for anyone who wants to try on their own: https://github.com/vmchale/ghc-cross/releases/tag/8.6.5 (both armv7 and aarch64)

[stormdragon2976]

Howdy Peter,

Thanks for this, it is very awesome. Could you also provide armv7h binaries? I don't currently have an X86 to try the cross compiler.

One small problem that happened when I tried to convert a file:

Could not find data file /home/vanessa/.cabal/share/aarch64-linux-ghc-8.6.5/pandoc-2.7.3/data/abbreviations

Thanks,
Storm

On Sat, Nov 30, 2019 at 08:48:57AM -0800, Peter hyman wrote:

@stormdragon2976 if you want binaries, I have some here: https://github.com/vmchale/ghc-cross/releases/tag/Binaries

I also have linux -> linux cross-compilers for anyone who wants to try on their own: https://github.com/vmchale/ghc-cross/releases/tag/8.6.5 (both armv7 and aarch64)

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/jgm/pandoc/issues/5450#issuecomment-559994325

--
⛈🐲
Accessible low cost computers for everyone! Get your slice of the Pi: https://asliceofthepi.com
My youtube channel: https://www.youtube.com/channel/UCdaTl5vl404OaRxAJPvb8Ew
get my public PGP key: gpg --keyserver wwwkeys.pgp.net --recv-key 43DDC193
Follow me on GNU Social: https://social.wolfe.casa/storm
"Have you ever been alone at night, thought you heard footsteps behind, and turned around and no one's there? And as you quicken up your pace you'll find it hard to look again because you're sure that someone's there."
Iron Maiden - Fear of the Dark

[jgm]

One small problem that happened when I tried to convert a file:

Could not find data file /home/vanessa/.cabal/share/aarch64-linux-ghc-8.6.5/pandoc-2.7.3/data/abbreviations

Set the embed_data_files flag when you compile; this will ensure that the data files are included in the binary.

[stormdragon2976]

Howdy Peter,

I just downloaded the aarch64 binary and tried it on the N2. That's when I got the error.

Thanks,
Storm

On Sat, Nov 30, 2019 at 10:27:57AM -0800, Peter hyman wrote:

One small problem that happened when I tried to convert a file:

Could not find data file /home/vanessa/.cabal/share/aarch64-linux-ghc-8.6.5/pandoc-2.7.3/data/abbreviations

Set the embed_data_files flag when you compile; this will ensure that the data files are included in the binary.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/jgm/pandoc/issues/5450#issuecomment-560009068

--
⛈🐲
Accessible low cost computers for everyone! Get your slice of the Pi: https://asliceofthepi.com
My youtube channel: https://www.youtube.com/channel/UCdaTl5vl404OaRxAJPvb8Ew
get my public PGP key: gpg --keyserver wwwkeys.pgp.net --recv-key 43DDC193
Follow me on GNU Social: https://social.wolfe.casa/storm
"I don't wanna face my fears! I'm afraid of 'em!"
SpongeBob SquarePants

[cstrouse]

I work on Chromebrew and we'd like to have ARMv7 binaries as well since Google chose to ship all ARMv8 Chromebooks with 32-bit user ABIs due to only having 4GB of memory.

[annProg]

Any progress for this issue?

I try to build pandoc-2.9.2 for raspberry pi 4b, and got error

[133 of 161] Compiling Text.Pandoc.Readers.LaTeX
ghc: internal error: getMBlock: mmap: Invalid argument
    (GHC version 8.4.4 for arm_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
cabal: Leaving directory '/tmp/cabal-tmp-8306/pandoc-2.9.2'

[ickc]

I have exactly the same issue on both aarch64 and armv7l on Raspberry Pi 4 4GB model with pandoc 2.9+. 4GB was just enough to compile pandoc briefly after Raspberry Pi 4 launched but then pandoc's need soon exceeds that.

If you can't wait there's a build in https://github.com/ickc/pandoc-arm/releases, with the last version compiled successfully on RPi4, I think 2.8.

I should emphasize that you can always add more swap. Long time ago I think I successfully compiled pandoc with 1GB RAM (forgot if it was a RPi original or another low mem. laptop.) Just add a ton of swap and it will work, albeit very slowly and your swap device's life shortened if it is flash (e.g. microSD.)

I was about to experiment with zram if I have time and see if it helps.

(As other has noted, the proper way to do it is to cross-compile and do it in CI, and distribute it officially. But until that happens...)

[annProg]

@ickc the version is 2.7.3. It has problem on epub toc. So I am looking for 2.9+

[ickc]

Then build your own, add a ton of swap and wait for a few days.

[annProg]

I successfully compiled 2.9.2 . see https://github.com/arm4rpi/pandoc-arm/releases

[ickc]

How? Just ton of swap?

Are you going to provide that continuously in the future? If so there's no need for 2 of us doing the same thing. Also, if you decided to continually provided that, you may want to think about security implication. e.g. as outlined in my repo I found that it is reproducible build so providing checksum should be suffices (may be this is the reason we still need to have 2 independent builds just for others to double check.)

As said above if people working on cross compiling has a solution then that would be preferred as doing it in the cloud in the open means less trust on us is needed. But before that it may be something you and I can do in the interim.

[annProg]

@ickc I am trying to build using github actions. But got some errors， see https://github.com/arm4rpi/pandoc-arm/runs/483167534

failed to install.
base-compat-0.11.1-HgwyiOzKEQ9KmBXaEHlBfi failed during the configure step.
The exception was:
ExitFailure 1
base-orphans-0.8.2-3ZQM2pxPHbTALAIaG3rSz1 failed during the configure step.
The exception was:
ExitFailure 1
doctemplates-0.8.1-Iuw4He9PrsFAzb6E5YhZXp depends on doctemplates-0.8.1 which
failed to install.
ipynb-0.1-5azHA6CVSg970U03zfl88V depends on ipynb-0.1 which failed to install.
pandoc-2.9.2-5EHBhYTFz4u65WvZc3TiB5 depends on pandoc-2.9.2 which failed to
install.
pandoc-types-1.20-8EQHit822IcL8VC8fC6OZi depends on pandoc-types-1.20 which
failed to install.
skylighting-0.8.3.2-C2Li4Bm4LPW5TVFmKvF7GB depends on skylighting-0.8.3.2
which failed to install.
skylighting-core-0.8.3.2-BDFsVUriuS2BVnrNvKwqBs depends on
skylighting-core-0.8.3.2 which failed to install.
texmath-0.12.0.1-2htPgXHYu9jEzl2nKKanYP depends on texmath-0.12.0.1 which
failed to install.
time-compat-1.9.2.2-B4M2FpSJt3R4h2zQkxlzCq depends on time-compat-1.9.2.2
which failed to install.

##[error]Process completed with exit code 1.

do you know how to solve it?

[ickc]

Sorry no experience on GitHub Action.

What was the earlier errors? Do you see something fail to compile?

Could you bring me up to speed about the GitHub Action? e.g. do they provide armv7l and aarch64 or are you cross-compiling?

In the past experience when dealing with CI, is the "debug loop" can be troublesome. One way people often try to solve situation like this is to use the docker image of the CI instance itself (e.g. Travis CI) and debug it interactively.

[annProg]

Seem no armv7l and aarch64 provided, I am using qemu-user-static

for debug. run

docker run -it --privileged --rm ann17/pandoc-ghc-arm:latest /bin/bash
cd rootfs
cp /etc/resolv.conf etc
mount -t devtmpfs devtmpfs dev
mount -t devpts devpts dev/pts
mount -t sysfs sysfs sys
mount -t tmpfs tmpfs tmp
mount -t proc proc proc

# chroot to arm
chroot . /bin/bash
uname -m
cabal update
cabal install -v time-compat-1.9.2.2

docker image ann17/pandoc-ghc-arm:latest was build from https://github.com/arm4rpi/ghc-arm

[annProg]

Now I am trying to build for aarch64 and armv7l using Github Actions with qemu-user-static. Encountered two problems, can anyone help?

cabal resolving dependencies problem

Limited by execution time(6 hours), it is not possible to build successfull. So I want to pre build dependencies( see https://github.com/arm4rpi/pandoc-deps), but the problem is pandoc and pandoc-citeproc resolving deferent dependencies, libraries below need rebuild when build pandoc-citeproc, can not use prebuilt library for pandoc, the long suffix of library is different, for example: pandoc-2.9.2-cd37f71aada58e7c054fb0d9850c9c6facc99a5599ef764195fc2ac2b25bb988 and pandoc-2.9.2-028d873232afbbca4423e4e65d416bc237bf7809e93e4d4cd251f51fd681e9b8. So, now only pandoc will build successful.

In order, the following will be built:
 - time-compat-1.9.3 (lib) (requires download & build)
 - aeson-1.4.6.0 (lib) (requires download & build)
 - yaml-0.11.3.0 (lib) (requires download & build)
 - skylighting-core-0.8.3.2 (lib) (requires download & build)
 - pandoc-types-1.20 (lib) (requires download & build)
 - ipynb-0.1 (lib) (requires download & build)
 - doctemplates-0.8.1 (lib) (requires download & build)
 - aeson-pretty-0.8.8 (lib) (requires download & build)
 - HsYAML-aeson-0.2.0.0 (lib) (requires download & build)
 - skylighting-0.8.3.2 (lib) (requires download & build)
 - texmath-0.12.0.1 (lib) (requires download & build)
 - pandoc-2.9.2 (lib) (requires download & build)

armv7l always out of memory

I had add huge swap.

swapon: /mnt/swapfile: insecure permissions 0644, 0600 suggested.
swapon: /home/swapfile: insecure permissions 0666, 0600 suggested.
              total        used        free      shared  buff/cache   available
Mem:           6945         712         124          25        6108        5904
Swap:         30719          31       30688

alway out of memory when Compiling Text.Pandoc.Readers.LaTeX

[133 of 162] Compiling Text.Pandoc.Readers.LaTeX.Parsing
[134 of 162] Compiling Text.Pandoc.Readers.LaTeX
ghc: out of memory (requested 1048576 bytes)
CallStack (from HasCallStack):
  die', called at ./Distribution/Client/ProjectOrchestration.hs:977:55 in main:Distribution.Client.ProjectOrchestration
cabal: Failed to build pandoc-2.9.2 because it depends on pandoc-2.9.2 which
itself failed to build.
Failed to build
pandoc-2.9.2-a37414eb634713f674eb2c181278989d68c5c7af029fece038bbeefb7e04ca90.
The build process terminated with exit code 251

[tarleb]

I think the only solution for the out-of-memory error is to further break up Text.Pandoc.Readers.LaTeX into smaller modules.

Not sure about the dependency problems. The differences in hashes are probably due to different flags being used. Maybe try using more specific constraint --constraint='pandoc:embed_data_files instead of global --flag embed_data_files. You could also experiment with cabal.project.freeze files to ensure reproducible builds.

[annProg]

@tarleb thanks. I will try.
only armv7l will out-of-memory. aarch64 does not have this problem

[znmeb]

I'm working on this via Docker on the NVIDIA Jetson Xavier AGX (arm64 plus CUDA and tensor cores). OS is Linux for Tegra (l4t), which is Ubuntu Bionic plus all the NVIDIA software. See http://github.com/znmeb/dockerfiles.