Core: OPNSense hangs after a while

[tadrien]

After some hours of operation, i can't login into the administration panel anymore, the web address of the panel is simply not responding. A console login also fails, and it shows some strange errors:

In most cases, data transfer through the firewall is still possible.
After a hard reboot, the OS is repairing the filesystem and everything is working again...for a few hours.

OPNSense is version 19.1 on PC Engines APU 4c4 with AMD Embedded G series GX-412TC processor.

[AdSchellevis]

broken sd card?

[fichtner]

We hear APU4 needs BIOS updates, SATA operation is flaky... cc @lattera

[tadrien]

No, I'm using a ssd disk.

The former version of OPNSense worked without errors, if that helps.

[AdSchellevis]

quite often these kind of issues are hardware related, if connected via a sata cable, it might be worth a try replacing the cable.

[tadrien]

There's no cable, it's a mSATA disk, mounted directly on the mainboard

[fichtner]

As I said, please investigate if BIOS updates exist...

[lattera]

I reached out to PC Engines directly. They verified it's a BIOS issue. They said downgrading to the latest mainline BIOS should resolve the issue.

[lattera]

(BTW, I'm in the same boat with multiple APU4 devices. I haven't been able to successfully downgrade the BIOS, yet.)

[klada]

Sorry for being a little off-topic here, but I'd like to inform you guys that PCEngines has release a new coreboot (v4.0.23) for the APU2 boards yesterday:

https://pcengines.github.io/

It fixes a pfSense issue and introduces ECC support for the APU's memory.

I have upgraded my BIOS to v4.0.23 and OPNSense 19.1 has been running fine so far with it. Also make sure to use the "Legacy BIOS" for FreeBSD-based OSes like OPNsense.

[fichtner]

@klada highly appreciated, thanks for the heads-up! :)

[lattera]

I've now verified that flashing v4.0.23 resolves the mSATA issues.

Here's the procedure I followed:

1. Download HardenedBSD installation media
2. Write HardenedBSD installation media to USB memstick with dd
3. Boot the installation media on the APU4c4
4. Partition the mSATA with a single 2gb partition, type freebsd-swap
5. swapon /dev/ada0p1
6. Set up a memory-backed filesystem (3gb in size): mdconfig -a -s 2g && newfs md0 && mount /dev/md0 /mnt && mkdir /mnt/update && mkdir /mnt/root
7. Populated the filesystem by running hbsd-update (as if I were creating a new jail from scratch): hbsd-update -V -T -t /mnt/update -d -n -r /mnt/root -U
8. Mount devfs in the new filesystem: mount -t devfs devfs /mnt/root/dev
9. chroot into the new filesystem: chroot /mnt/root
10. pkg install flashrom
11. fetch --no-verify-peer http://pcengines.ch/file/apu4_v4.0.23.rom.tar.gz
12. tar -xf apu4_v4.0.23.rom.tar.gz
13. flashrom -p internal:boardmismatch=force -w apu4_v4.0.23.rom
14. shutdown -p now

Once the system powers off, you can now boot the OPNsense 19.1 installation media and proceed with installation. :)

I now have to follow these steps again on two other APU4 devices I'm deploying. :)

[tadrien]

You may also do this from an running instance of opnsense (starting from step 10, which worked in my case), as described here:

https://www.bsdforen.de/threads/pc-engines-apu2-bios-update.33587/

[lattera]

Yup. In my case, I didn't have a running install of OPNsense.

So, instead of the large amount of steps above, you could install OPNsense 18.7, flash the v4.0.23 BIOS, then move forward with the install of or upgrade to 19.1. :)

[lattera]

@tadrien is this still an issue for you? Should we keep the ticket open?

[HaraldSimon]

The argument -w is missing in step 13:
flashrom -p internal:boardmismatch=force -w apu4_v4.0.23.rom

Thanks to lattera for sharing the steps.

[lattera]

Whoops! Good catch! I'll edit the comment to add that. Thanks!

[tadrien]

@lattera Currently not. I'm currently running my APU on the latest bios (4.9.0.1) and everything seems to be working fine, though i wasn't able to have my OPNSense running for a longer period of time since the bios upgrade.

The newest bios also gave me a nice boost in performance.

Thanks a lot for your help so far!