Openbmc: [Barreleye - v1.99.0-351/#422] Barreleye system stucks at BMC_STARTING after reboot

[rahulmah]

Barreleye system is getting stuck at BMC_STARTING after reboot. See below step to reproduce for more details. This issue is seen inconsistently.

1. Bring Barreleye system to HOST_POWERED_OFF.

2. Execute /sbin/reboot command on BMC cli.(Here after this command, we will see that BMC will get stuck BMC_STARTING state. Waited for almost 10 mins)

BMC FW:
root@barreleye:~# cat /etc/os-release
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="v1.99.0-351"
VERSION_ID="v1.99.0-351-g8c1f920-dirty"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) v1.99.0-351"
BUILD_ID="v1.99.0"
root@barreleye:~#

Let me know if more logs are need from the failure.

[williamspatrick]

I see two issues:

gevent[717]:     message, timeout)
Dec 12 19:01:39 barreleye phosphor-gevent[717]: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name xyz.openbmc_project.ObjectMapper was not provided by any .service files

This implies a dependency problem with the rest server.

Also, I see settingsd fail with the "bus busy" error we've been seeing elsewhere.

My suggestion is we use this issue to fix the dependency issue and we track the other part as a duplicate.

[spinler]

I wonder if this is related to the state related changes I made for Witherspoon...

[rahulmah]

Sometime I am also seeing that BMC getting stuck at BASE_APPS after reboot. Below you can find journal logs for the failure. Should we log a new issue for this.

Base_APPS_Journal_logs.txt

[geissonator]

Rahul, if you see this in your log, and you're not getting to READY, it's 99% chance to be #862 :

Nov 21 06:09:45 barreleye settings_manager.py[740]: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.ObjectPathInUse: busy processing bus traffic

[rahulmah]

Okay got it.

[rahulmah]

862 issue is now fixed and closed. But I still see bmc stuck at BMC_STARTING some time after reboot. Tested with v1.99.1-2-g047ea60/#470/24th Jan build.

[geissonator]

@rahulmah please update on this one, we've had a lot of bugs fixed in this area. Can we close out?

[rahulmah]

@geissonator : Yes we can close this. I have not see this problem from long time