Loki: Can't scrape journal on Raspberry Pi

[hairyhenderson]

Describe the bug

On a Raspberry Pi running the official Raspbian bistro, promtail doesn't support reading from journal:

WARNING!!! Journal target was configured but support for reading the systemd journal is not compiled into this build of promtail!

It seems like this was done in #888 as a temporary measure, though it's not entirely clear why (is it because no non-AMD64 hardware is available for the CI builds?). Having to enable CGO when cross-compiling can be a pain, so maybe this is why?

It'd be _super_ useful to be able to use the released armhf promtail binary to scrape logs on a Pi, though!

To Reproduce

1. run promtail with a journal configuration
2. see the logs and notice that it doesn't scrape the journal

Expected behavior

It should scrape the journal 😉

Environment:

• Infrastructure: Raspberry Pi

Screenshots, Promtail config, or terminal output

relevant config is:

scrape_configs:
- job_name: journald
  journal: {}

[hairyhenderson]

It's maybe worth noting as well that I've been able to get the build to _start_ by removing the lines added in #888. However I'm running into memory constraints on my Raspberry Pi 3B (4 cores, 1GB RAM). After getting by some OOMs by building dependent packages directly, linking fails:

$ GOMAXPROCS=1 CGO_ENABLED=1 go build -ldflags "-s -w -X github.com/grafana/loki/pkg/build.Branch=HEAD -X github.com/grafana/loki/pkg/build.Version=v1.2.0-WIP -X github.com/grafana/loki/pkg/build.Revision=ccef3da2 -X github.com/grafana/loki/pkg/build.BuildUser=pi@mypi -X github.com/grafana/loki/pkg/build.BuildDate=2019-12-30T01:33:41Z" -tags netgo -mod=vendor -o cmd/promtail/promtail ./cmd/promtail
# github.com/grafana/loki/cmd/promtail
/usr/local/go/pkg/tool/linux_arm/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory

My gut feeling is that all the different SD dependencies are adding lots of bloat at compile time. I wonder if there's a way to disable them selectively at compile time... For my use-case I don't need the majority of the SD mechanisms.

[hairyhenderson]

Update: got it building after I added a whole pile of swap (with an external drive, to spare the poor SD card 😅)

[rfratto]

Thanks for reporting! We did have to disable it because we were using a cross-compiler for the ARM64 builds. and I forgot about it being disabled once we switched to using native ARM machines :slightly_smiling_face:

[hairyhenderson]

Thanks @rfratto!

[ikogan]

I'm getting this error on 1.5.0 and the thread makes me think that 1.5.0 should be built with journal support...?

[Alpha200]

I have the same issue on 1.5.0 with the arm build.

[slim-bean]

How are you running promtail? Are you running from the binary or from the docker image?

[Alpha200]

I'm running it over the binary (arm, not arm64).

[ikogan]

Also running arm, not arm64.

[slim-bean]

the binaries are cross compiled with cgo disabled because it's a huge pain in the butt to cross compile with cgo, we would need a C cross compiler to make it work. A quick "fix" would be to use the docker image or even pull the promtail binary out of the docker image and see if that works.

The long term solution is going to be figuring out how to build arm binaries with cgo, not sure when we will be able to get to this.

[Alpha200]

I've tried to copy the binary out of the docker image, but now the binary fails to start with the following error message: fatal: systemstack called from unexpected goroutine.

I do not want to install docker on the Pis and sadly I'm stuck at Raspbian 9 because of some legacy software so I can not try it with podman.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.