Openbmc: System-psu-monitor: Failed to read sysfs file flooding journald

[gkeishin]

Journald snippet

Sep 19 20:48:50 witherspoon witherspoon-psu-monitor[1275]: Failed to read sysfs file
Sep 19 20:48:50 witherspoon witherspoon-psu-monitor[1275]: Failed to read from device.
Sep 19 20:48:52 witherspoon witherspoon-psu-monitor[1275]: Failed to read sysfs file
Sep 19 20:48:52 witherspoon witherspoon-psu-monitor[1275]: Failed to read from device.
Sep 19 20:48:53 witherspoon witherspoon-psu-monitor[1275]: Failed to read sysfs file
Sep 19 20:48:53 witherspoon witherspoon-psu-monitor[1275]: Failed to read from device.
Sep 19 20:48:55 witherspoon witherspoon-psu-monitor[1275]: Failed to read sysfs file
Sep 19 20:48:55 witherspoon witherspoon-psu-monitor[1275]: Failed to read from device

Driver Info

root@witherspoon:~# cat /etc/os-release
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="v1.99.10-85"
VERSION_ID="v1.99.10-85-g7b8e664"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) v1.99.10-85"
BUILD_ID="v1.99.10"
uroot@witherspoon:~# uname -a
Linux witherspoon 4.10.17-00cd954c889746a65f9022088c5d83f9f43bfddc #1 Tue Sep 26 18:05:33 UTC 2017 armv6l GNU/Linux
root@witherspoon:~#

• [x] Journald
  PSU_logging_errors.txt

[gkeishin]

This keeps continuously logging even when the host is booted.

[bjwyman]

It looks like both power supplies have the CML fault bit state in the STATUS_WORD. The top power supply appears to be indicating that it has an input voltage fault.

Sep 19 20:45:34 witherspoon witherspoon-psu-monitor[1275]: The power supply has indicated an input fault or warn condition.

As part of the error log for the power supply with the "input fault or warn" condition, the power fault analysis code is attempting to collect a number of command responses for the metadata, but it is unable to collect all the desired data due to #2364.

```root@witherspoon:~# ls /sys/bus/i2c/devices/3-0068/hwmon/hwmon4
curr1_input device in1_label in2_label of_node power1_input subsystem temp2_input uevent
curr1_label in1_input in2_input name power power1_label temp1_input temp3_input
root@witherspoon:~# ls /sys/bus/i2c/devices/3-0069
driver hwmon modalias name of_node power subsystem uevent
root@witherspoon:~# ls /sys/bus/i2c/devices/3-0069/hwmon
hwmon5
root@witherspoon:~# ls /sys/bus/i2c/devices/3-0069/hwmon/hwmon5
curr1_input device in1_label in2_label of_node power1_input subsystem temp2_input uevent
curr1_label in1_input in2_input name power power1_label temp1_input temp3_input
root@witherspoon:~# ls /sys/kernel/debug/pmbus/hwmon4
status0 status0_fan12 status0_input status0_iout status0_temp status0_vout
root@witherspoon:~# ls /sys/kernel/debug/pmbus/hwmon5
status0 status0_fan12 status0_input status0_iout status0_temp status0_vout
root@witherspoon:~#

The STATUS_MFR_SPECIFIC data is missing, due to status0_mfr missing.

```root@witherspoon:~# grep . /sys/kernel/debug/pmbus/hwmon5/*
/sys/kernel/debug/pmbus/hwmon5/status0:0x2842
/sys/kernel/debug/pmbus/hwmon5/status0_fan12:0x00
/sys/kernel/debug/pmbus/hwmon5/status0_input:0x08
/sys/kernel/debug/pmbus/hwmon5/status0_iout:0x00
/sys/kernel/debug/pmbus/hwmon5/status0_temp:0x00
/sys/kernel/debug/pmbus/hwmon5/status0_vout:0x00
root@witherspoon:~# grep . /sys/kernel/debug/pmbus/hwmon4/*
/sys/kernel/debug/pmbus/hwmon4/status0:0x0002
/sys/kernel/debug/pmbus/hwmon4/status0_fan12:0x00
/sys/kernel/debug/pmbus/hwmon4/status0_input:0x00
/sys/kernel/debug/pmbus/hwmon4/status0_iout:0x00
/sys/kernel/debug/pmbus/hwmon4/status0_temp:0x00
/sys/kernel/debug/pmbus/hwmon4/status0_vout:0x00
root@witherspoon:~#

[bjwyman]

PN: 01KL779
FW: 00 00 01 12 & 01 11 01 12.

I am not sure why both power supplies have the same part number, but the top power supply has different firmware (FW).

Someone should at least check the power cables for the power supplies, more specifically, the top power supply appears to be having a problem. Check/fix the power to that supply, or repair/replace it.

[mdmillerii]

Regardless of hw fail we shouldn't be logging failure to read every 1-2 seconds.

[gkeishin]

Uma checked the power cable. In this system, there is only one power cable connected and he switched that to top port.
Boot the system still see the problem.

[bjwyman]

https://github.com/openbmc/openbmc/commit/60034dc26c690d822f642a67e3cacc08db1f7421
That commit should resolve the flood of messages in the journal.

Both power supplies need to be plugged in if you do not want to see an input fault error get logged.

[ojayanth]

Removing the updates