Openbmc: [Garrison v1.99.7-5+] Cannot access any sensors apart from the RTC_temp1

Created on 28 Jun 2017  路  10Comments  路  Source: openbmc/openbmc

For some reason, all sensors (apart from RTC_temp1) not available if using latest OpenBMC.
If I revert back to older OpenBMC (ex. "git checkout bdd2b66" for "v.1.99.5" or "git checkout fcaf72e" for "v1.99.6-2-gfcaf72e" ), then all sensors available.

I'm using the same PNOR (latest release version), so the problem inside OpenBMC (I mean that if PNOR works well with older OpenBMC, then the same latest PNOR should work well with latest OpenBMC too).

For example, this sensor RTC_temp1 works Ok in both older "v.1.99.5" and new "v1.99.7-16":
curl b cjar -k https://172.17.27.66/org/openbmc/sensors/temperature/RTC_temp1
being read from "/sys/class/hwmon/hwmon0/temp1_input".

The other sensors non available in newest version even it worked before.

I've noted that "/sys/class/hwmon/hwmon3/*" disappeared in new OpenBMC comparing with older OpenBMC, so this could be the reason (and hwmon0, hwmon1 exists both in older OpenBMC and in newest).

Also don't forget poweron host before reading sensors (because /sys/class/hwmon/hwmon3/ probably non-available before host poweron in older OpenBMC too).

Also I recommend AC poweroff/on before testing sensors, because if rebooting BMC when host keeps poweron, then sensors being lost even in older OpenBMC versions (that is separate bug).

bug
Source
picture yadron

All 10 comments

root@barreleye:/sys/class/hwmon# cat /etc/os-release 
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="v1.99.7-31"
VERSION_ID="v1.99.7-31-ge8b9354"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) v1.99.7-31"
BUILD_ID="v1.99.7"
root@barreleye:/sys/class/hwmon# ls
hwmon0  hwmon1  hwmon2  hwmon3  hwmon4  hwmon5
root@barreleye:/sys/class/hwmon# cat */name
ds3231
lm75
nct7904
nct7904
adm1278
adm1278

We have 5 showing up in some pretty recent code. Can you tell us which ones are missing?

williamspatrick picture williamspatrick on 28 Jun 2017

I can see the same issue on Palmetto and created #1852
The OCC sensors are lost.

mine260309 picture mine260309 on 29 Jun 2017

Yes, #1852 is the same for BarrelEye:

root@barreleye:~# cat /etc/os-release 
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="v1.99.7-5"
VERSION_ID="v1.99.7-5-g1afb614-dirty"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) v1.99.7-5"
BUILD_ID="v1.99.7"

and

root@barreleye:~# journalctl |grep online
Jun 19 12:49:48 barreleye sensor_manager2.py[772]: sh: /sys/class/i2c-adapter/i2c-3/3-0050/online: Permission denied
Jun 19 12:49:48 barreleye sensor_manager2.py[772]: sh: /sys/class/i2c-adapter/i2c-3/3-0050/online: Permission denied
Jun 19 13:01:09 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 20 05:02:57 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 20 05:02:58 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0051/online: Permission denied
Jun 20 13:00:55 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 22 21:25:25 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 22 21:25:25 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0051/online: Permission denied
Jun 22 21:36:11 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 22 21:36:11 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0051/online: Permission denied
Jun 22 21:39:20 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 23 09:17:45 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 23 09:17:45 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0051/online: Permission denied
Jun 23 09:20:43 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 29 12:21:25 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0050/online: Permission denied
Jun 29 12:21:25 barreleye sensor_manager2.py[772]: sh: /sys/bus/i2c/devices/3-0051/online: Permission denied

For some reason at similar platform like Barreleye (ex. "garrison") no sensors could be read at all (so the description above that no sensors available at all probably more likely for "garrison platform" and I can expect that if I try latest OpenBMC with Barreleye then even with " sh: /sys/bus/i2c/devices/3-0051/online: Permission denied" more sensors readable than I've described above).

So probably there are two separate bugs: one is #1852 for all P8 platforms and other is this one (or similar #1796) just for some similar to Barreleye platforms like "garrison".

yadron picture yadron on 29 Jun 2017

BTW, I've noted that if /sys/class/hwmon/ emtpy , then I see these two services working forever:

 1087 root       0:00 /usr/sbin/phosphor-hwmon-readd -o /ahb/apb/i2c@1e78a000/i2c-bus@80/tmp75@49
 1089 root       0:00 /usr/sbin/phosphor-hwmon-readd -o /ahb/apb/i2c@1e78a000/i2c-bus@400/rtc@68

Who call these services and is it expected?

If I kill these services, then I see:

Jun 29 17:13:29 platform systemd[1]: xyz.openbmc_project.Hwmon@ahb-apb-i2c\x401e78a000-i2c\x2dbus\x4080-tmp75\x4049.service: Service hold-off time over, scheduling restart.
Jun 29 17:13:30 platform systemd[1]: xyz.openbmc_project.Hwmon@ahb-apb-i2c\x401e78a000-i2c\x2dbus\x40400-rtc\x4068.service: Service hold-off time over, scheduling restart.

I've noted that when everything works good (for example at Barreleye), then no such "phosphor-hwmon-readd" services.

So all other sensors from /etc/default/obmc/hwmon/* skipped or failed (for example, very likely
/etc/default/obmc/hwmon/ahb/apb/i2c\@1e78a000/i2c-bus\@440/X
cannot start, because /sys/class/hwmonX not available for this device X).

Where /sys/class/hwmon* created and how to debug why not created any apart from

ds3231
tmp75

?

yadron picture yadron on 29 Jun 2017

Continuing prev. msg, it could be kernel params or .dts.

Comparing correct configuration for older 4.7 kernel (dmesg output:

aspeed-smc 1e630000.spi: no DMA support.
i2c /dev entries driver
i2c_aspeed 1e78a000.i2c: i2c controller registered, irq 21
i2c_aspeed i2c-0: i2c bus 0 registered, irq 242
i2c_aspeed i2c-1: i2c bus 1 registered, irq 243
i2c_aspeed i2c-3: i2c bus 3 registered, irq 245
i2c_aspeed i2c-4: i2c bus 4 registered, irq 246
i2c_aspeed i2c-5: i2c bus 5 registered, irq 247
i2c_aspeed i2c-6: i2c bus 6 registered, irq 248
i2c_aspeed i2c-7: i2c bus 7 registered, irq 249
i2c_aspeed i2c-9: i2c bus 9 registered, irq 251
i2c_aspeed i2c-10: i2c bus 10 registered, irq 252
rtc-ds1307 11-0068: rtc core: registered ds3231 as rtc0
i2c_aspeed i2c-11: i2c bus 11 registered, irq 253
i2c_aspeed i2c-12: i2c bus 12 registered, irq 254
i2c_aspeed i2c-13: i2c bus 13 registered, irq 255

) with newer 4.10 kernel dmesg output (and why "1e78a000" not listed anymore below?) :

[    5.480000] aspeed-smc 1e620000.fmc: Using IRQ 18
[    5.480000] aspeed-smc 1e620000.fmc: DMA support disabled.
[    5.480000] aspeed-smc 1e620000.fmc: mx25l25635e (32768 Kbytes)
[    5.480000] aspeed-smc 1e620000.fmc: CE0 window [ 0x20000000 - 0x22000000 ] 32MB
[    5.480000] aspeed-smc 1e620000.fmc: CE1 window [ 0x22000000 - 0x26000000 ] 64MB
[    5.480000] aspeed-smc 1e630000.spi: Error applying setting, reverse things back
[    5.480000] aspeed-smc 1e630000.spi: mx66l51235l (65536 Kbytes)
[    6.300000] i2c /dev entries driver
[    6.300000] aspeed-i2c-bus 1e78a040.i2c-bus: i2c bus 0 registered, irq 23
[    6.300000] aspeed-i2c-bus 1e78a080.i2c-bus: i2c bus 1 registered, irq 24
[    6.300000] aspeed-i2c-bus 1e78a0c0.i2c-bus: i2c bus 2 registered, irq 25
[    6.300000] aspeed-i2c-bus 1e78a100.i2c-bus: i2c bus 3 registered, irq 26
[    6.300000] aspeed-i2c-bus 1e78a140.i2c-bus: i2c bus 4 registered, irq 27
[    6.300000] aspeed-i2c-bus 1e78a180.i2c-bus: i2c bus 5 registered, irq 28
[    6.300000] aspeed-i2c-bus 1e78a1c0.i2c-bus: i2c bus 6 registered, irq 29
[    6.590000] aspeed-i2c-bus 1e78a300.i2c-bus: i2c bus 7 registered, irq 30
[    6.590000] aspeed-i2c-bus 1e78a380.i2c-bus: i2c bus 9 registered, irq 31
[    6.590000] aspeed-i2c-bus 1e78a3c0.i2c-bus: i2c bus 10 registered, irq 32
[    6.620000] rtc-ds1307 11-0068: rtc core: registered ds3231 as rtc0
[    6.630000] aspeed-i2c-bus 1e78a400.i2c-bus: i2c bus 11 registered, irq 33
[    6.630000] aspeed-i2c-bus 1e78a440.i2c-bus: i2c bus 12 registered, irq 34
[    6.650000] aspeed-i2c-bus 1e78a480.i2c-bus: i2c bus 13 registered, irq 35

Also I've checked at Barreleye "v1.99.7-5" and sensors are shown there (so original subject was incorrect), and at Garrison "v1.99.7-5" sensors not shown (so thinking that all these p8 platforms similar the bug in configuration of linux kernel or .dts possibly could be found comparing difference of Barreleye hardware configuration and Garrison configuration for kernel 4.10). And latest OpenBMC "v1.99.7-16" or "v1.99.7-43" does the same.

yadron picture yadron on 29 Jun 2017

Continuing prev. msg., for Barreleye with 4.10 when everything working the dmesg output is very similar with above dmesg 4.10 output when nothing worked:

[    5.540000] aspeed-smc 1e620000.fmc: Using IRQ 18
[    5.540000] aspeed-smc 1e620000.fmc: DMA support disabled.
[    5.560000] aspeed-smc 1e620000.fmc: w25q256 (32768 Kbytes)
[    5.560000] aspeed-smc 1e620000.fmc: CE0 window [ 0x20000000 - 0x22000000 ] 32MB
[    5.570000] aspeed-smc 1e620000.fmc: CE1 window [ 0x22000000 - 0x26000000 ] 64MB
[    5.650000] aspeed-smc 1e630000.spi: Error applying setting, reverse things back
[    5.650000] aspeed-smc 1e630000.spi: n25q512ax3 (65536 Kbytes)
[    6.630000] i2c /dev entries driver
[    6.740000] at24 0-0050: 32768 byte 24c256 EEPROM, writable, 64 bytes/write
[    6.760000] aspeed-i2c-bus 1e78a040.i2c-bus: i2c bus 0 registered, irq 23
[    6.770000] aspeed-i2c-bus 1e78a100.i2c-bus: i2c bus 3 registered, irq 24
[    6.790000] aspeed-i2c-bus 1e78a140.i2c-bus: i2c bus 4 registered, irq 25
[    6.810000] aspeed-i2c-bus 1e78a180.i2c-bus: i2c bus 5 registered, irq 26
[    6.840000] aspeed-i2c-bus 1e78a1c0.i2c-bus: i2c bus 6 registered, irq 27
[    6.840000] lm75 0-004a: hwmon1: sensor 'lm75'
[    6.930000] aspeed-g4-pinctrl 1e6e2000.syscon:pinctrl@1e6e2000: pin B12 already requested by 1e660000.ethernet; cannot claim
 for 1e780000.gpio:445
[    6.940000] aspeed-g4-pinctrl 1e6e2000.syscon:pinctrl@1e6e2000: pin-153 (1e780000.gpio:445) status -22
yadron picture yadron on 29 Jun 2017
😕1 👍1

Fixed most sensors (apart from OCC sensors) by addition to config

arch/arm/boot/dts/aspeed-bmc-opp-customgarrison.dts

of these lines:

> @@ -0,0 +1,183 @@
233a234,238
> +
> +     w83795g@2f {
> +             compatible = "nuvoton,w83795g";
> +             reg = <0x2f>;
> +     };

Waiting how to fix OCC sensors (that is https://github.com/openbmc/openbmc/issues/1852 bug). I've tried turning it manually by "echo occ-hwmon 0x50 > /sys/bus/i2c/devices/i2c-4/new_device", but unsuccessfully yet.

-- after the above w83795 fix the "/sys/class/hwmon2" appeared and most sensors working (the list of working:

temperature/
ambient  OUTlet_Temp1  OUTlet_Temp2  RTC_temp1
FOUND/voltage/
P0V89_CPU0  P0V89_CPU2  P0V97_CPU0  P0V97_CPU2  P12V_1     P1V1_CPU1  P1V1_CPU3
P0V89_CPU1  P0V89_CPU3  P0V97_CPU1  P0V97_CPU3  P1V1_CPU0  P1V1_CPU2  P5V
tach/
FAN_TACH0  FAN_TACH1  FAN_TACH2  FAN_TACH3  FAN_TACH4
speed/
FAN_PWM0  FAN_PWM1  FAN_PWM2  FAN_PWM3  FAN_PWM4

), but not all of the expected.

The list of failed sensors:

temperature/INLet_Temp1
temperature/INLet_Temp2
powercap/
curr_cap  max_cap  min_cap  n_cap  system_power  user_cap

So there is one more question now:

The strange thing that OUTLet* sensors working well, but INLet* sensors not. Why?

Could you please check INLet* sensors?

For example, this is OutLet temp sensor:

cat /sys/class/hwmon/hwmon1/device/temp3_input
37750

curl -b cjar -k https://172.17.27.90/org/openbmc/sensors/temperature/OUTlet_Temp1

cat 1498827931.91/FOUND/temperature/OUTlet_Temp1/value.txt 
{
  "data": {
    "critical_lower": 0, 
    "critical_upper": 0, 
    "emergency_enabled": 0, 
    "error": 0, 
    "filename": "/sys/class/hwmon/hwmon1/device/temp3_input", 
    "offset": 0, 
    "scale": -3, 
    "threshold_state": "NORMAL", 
    "thresholds_enabled": 0, 
    "units": "C", 
    "value": 37750, 
    "warning_lower": 0, 
    "warning_upper": 0, 
    "worst_threshold_state": "NORMAL"
  }, 
  "message": "200 OK", 
  "status": "ok"

In older OpenBMC when INLet_Temp1 worked, it was: /sys/class/hwmon/hwmon1/device/temp3_input

Right now this value available too:

cat /sys/class/hwmon/hwmon1/device/temp3_input
38250

, then why this cmd doesn't work:

curl -b cjar -k https://172.17.27.90/org/openbmc/sensors/temperature/INLet_Temp1
{
  "data": {
    "description": "org.freedesktop.DBus.Error.FileNotFound: path or object not found: /org/openbmc/sensors/temperature/INLet_Temp1"
  }, 
  "message": "404 Not Found", 
  "status": "error" }

If using new xyz interface, then only these two sensors available:

curl -b cjar -k https://172.17.27.90/xyz/openbmc_project/senors/temperature/
{
  "data": [
    "/xyz/openbmc_project/sensors/temperature/ambient", 
    "/xyz/openbmc_project/sensors/temperature/RTC_temp1"
  ], 
  "message": "200 OK", 
  "status": "ok"
}

Why?

The version of OpenBMC that I'm testing (currently latest):

root@customgarrison:~# cat /etc/os-release
ID="openbmc-phosphor"
NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)"
VERSION="v1.99.7-70"
VERSION_ID="v1.99.7-70-gce19185-dirty"
PRETTY_NAME="Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) v1.99.7-70"
BUILD_ID="v1.99.7"
yadron picture yadron on 30 Jun 2017

@yadron Is this still a problem? If not, can this be closed?

gtmills picture gtmills on 23 May 2018

Garrison was removed here: https://gerrit.openbmc-project.xyz/#/c/10956/. Can we close?

gtmills picture gtmills on 1 Jun 2018

Closing since Garrison was removed.

gtmills picture gtmills on 29 Jun 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gkeishin picture gkeishin  路  4Comments
gkeishin picture gkeishin  路  9Comments
willenlee picture willenlee  路  10Comments
ya-mouse picture ya-mouse  路  5Comments
gkeishin picture gkeishin  路  5Comments