Netdata: Allow running multiple freeipmi jobs from the same netdata
Question/Issue:
There is no 'example' in https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin/example . I see an example for charts.py. Is that the boilerplate for a basic test module in Python?
OS / Environment
Ubuntu 14
Component Name
python.d.plugin
Expected results
A simple py module sample, with instructions.
I would like to write a plugin in python, to collect remote IPMI results.
Thanks for the help.
jimidak
All 57 comments
Hi @jimidak
How to write python module
https://docs.netdata.cloud/collectors/python.d.plugin/#how-to-write-a-new-module
to collect remote IPMI results.
https://docs.netdata.cloud/collectors/freeipmi.plugin/#freeipmiplugin
ilyam8
on 7 Feb 2019
Thanks so much! Let me take a crack at it, and reply back.
jimidak
on 7 Feb 2019
Would netdata/collectors/python.d.plugin/example/example.chart.py be a boilerplate for a module written in Python? Would a function/method under the Service class be where I do the "real work"?
I have read about the freeipmi plugin, and I will use that on hosts where I can install freeipmi and ipmimonitoring-dev, but I have some EXSi hosts and other appliances, where I can't do that, and so I wanted to write a module in Python, for remote collecting that IPMI. I have a Py script that makes calls to remote hosts, and gathers IPMI data... I'm just unsure how to get that in the python.d.plugin dir as a module like the example.charts.py
jimidak
on 7 Feb 2019
I have a Py script that makes calls to remote hosts, and gathers IPMI data
could you show it please
ilyam8
on 7 Feb 2019
right now it's just a simple py2 with user input, and a statically set vaule. I will make it py3 and non-interactive, and built it in a class, so I could import it, but this gives you the idea of what I'm currently doing.
#!/usr/bin/python
import os
cmd = 'ipmimonitoring -s 88 -h 10.117.6.190 -u root -p xxxx -D LAN_2_0'
value = int(raw_input("What is the ID you would like a status for? :" ))
if value == 88:
print os.system(cmd)
$ ./ipmi.py
What is the ID you would like a status for? :88
ID | Name | Type | State | Reading | Units | Event
88 | Status | Power Supply | Nominal | N/A | N/A | 'Presence detected'
0
Could be a bash script too...
$ for i in 88 89 90; do ipmimonitoring -s $i -h 10.117.6.190 -u root -p XXXX -D LAN_2_0; done
ID | Name | Type | State | Reading | Units | Event
88 | Status | Power Supply | Nominal | N/A | N/A | 'Presence detected'
ID | Name | Type | State | Reading | Units | Event
89 | Status | Power Supply | Nominal | N/A | N/A | 'Presence detected'
ID | Name | Type | State | Reading | Units | Event
90 | Pwr Consumption | Current | Nominal | 112.00 | W | 'OK'
and could awk and grep to get just what I want.
jimidak
on 7 Feb 2019
@jimidak
from freeimpi configuration i see that it can gather data from remote host
hostname HOST
username USER
password PASS connect to remote IPMI host
default: local IPMI processor
@jimidak
from freeimpi configuration i see that it can gather data from remote host
hostname HOST username USER password PASS connect to remote IPMI host default: local IPMI processor
Can you give me example syntax? Is it just hostname
./freeipmi.plugin hostname <host.network.net> username <user> password <abc123>
cc: @cakrit
Not much help from me on that :cry:
I did: ./freeipmi.plugin hostname 1.2.3.4 (random host)
I see from tcpdump
listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes
16:14:22.534459 IP 10.0.2.15.59169 > 1.2.3.4.623: UDP, length 23
16:14:23.035455 IP 10.0.2.15.59169 > 1.2.3.4.623: UDP, length 23
16:14:23.536378 IP 10.0.2.15.59169 > 1.2.3.4.623: UDP, length 23
16:14:24.537926 IP 10.0.2.15.59169 > 1.2.3.4.623: UDP, length 23
16:14:25.539576 IP 10.0.2.15.59169 > 1.2.3.4.623: UDP, length 23
16:14:27.041638 IP 10.0.2.15.59169 > 1.2.3.4.623: UDP, length 23
16:14:28.543680 IP 10.0.2.15.59169 > 1.2.3.4.623: UDP, length 23
So option works, it tries to (?) connect.
That doesn't seem to work.
What do you mean? What is output in your case?
ilyam8
on 7 Feb 2019
ipmi command works, but using the netdata per the options in -h, don't work, but it's hard from the freeipmi.plugin -h output, to know if I'm using the options correctly. I'd love an example.
# ipmimonitoring -h testhost1.oob.net -u admin -p abc123 -D LAN_2_0
Caching SDR repository information: /root/.freeipmi/sdr-cache/sdr-cache-testhost1.obb.net
Caching SDR record 156 of 156 (current record ID 156)
ID | Name | Type | State | Reading | Units | Event
2 | Intrusion | Physical Security | Nominal | N/A | N/A | 'OK'
12 | Fan1A | Fan | Nominal | 4680.00 | RPM | 'OK'
13 | Fan2A | Fan | Nominal | 4680.00 | RPM | 'OK'
14 | Fan3A | Fan | Nominal | 4680.00 | RPM | 'OK'
15 | Fan4A | Fan | Nominal | 4680.00 | RPM | 'OK'
16 | Fan5A | Fan | Nominal | 4680.00 | RPM | 'OK'
17 | Fan6A | Fan | Nominal | 4680.00 | RPM | 'OK'
18 | Inlet Temp | Temperature | Nominal | 25.00 | C | 'OK'
19 | Exhaust Temp | Temperature | Nominal | 37.00 | C | 'OK'
20 | Temp | Temperature | Nominal | 43.00 | C | 'OK'
21 | Temp | Temperature | Nominal | 43.00 | C | 'OK'
22 | OS Watchdog | Watchdog 2 | Nominal | N/A | N/A | 'OK'
23 | VCORE PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
24 | VCORE PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
25 | 3.3V PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
26 | 5V PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
27 | USB Cable Pres | Cable/Interconnect | Nominal | N/A | N/A | 'Cable/Interconnect is connected'
28 | VGA Cable Pres | Cable/Interconnect | Nominal | N/A | N/A | 'OK'
29 | Dedicated NIC | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
30 | Presence | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
31 | Presence | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
32 | Presence | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
33 | PLL PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
34 | PLL PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
35 | 1.1V PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
36 | M23 VDDQ PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
37 | M23 VTT PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
38 | FETDRV PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
39 | Presence | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
40 | VSA PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
41 | VSA PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
42 | M01 VDDQ PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
43 | M01 VDDQ PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
44 | M23 VTT PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
45 | M01 VTT PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
46 | NDC PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
47 | LCD Cable Pres | Cable/Interconnect | Nominal | N/A | N/A | 'OK'
48 | VTT PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
49 | VTT PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
50 | M23 VDDQ PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
51 | Presence | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
52 | Presence | Entity Presence | Critical | N/A | N/A | 'Entity Absent'
53 | Status | Processor | Nominal | N/A | N/A | 'Processor Presence detected'
54 | Status | Processor | Nominal | N/A | N/A | 'Processor Presence detected'
55 | Fan Redundancy | Fan | Nominal | N/A | N/A | 'Fully Redundant'
59 | Riser Config Err | Cable/Interconnect | Nominal | N/A | N/A | 'Cable/Interconnect is connected'
60 | Riser 3 Presence | Cable/Interconnect | Nominal | N/A | N/A | 'Cable/Interconnect is connected'
61 | 1.5V PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
62 | PS2 PG Fail | Voltage | Nominal | N/A | N/A | 'State Deasserted'
63 | PS1 PG Fail | Voltage | Nominal | N/A | N/A | 'State Deasserted'
64 | BP1 5V PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
66 | M01 VTT PG | Voltage | Nominal | N/A | N/A | 'State Deasserted'
67 | Presence | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
77 | vFlash | Module/Board | N/A | N/A | N/A | 'OEM Event = 0000h'
78 | CMOS Battery | Battery | Nominal | N/A | N/A | 'OK'
81 | Presence | Entity Presence | Nominal | N/A | N/A | 'Entity Present'
82 | Presence | Entity Presence | Critical | N/A | N/A | 'Entity Absent'
83 | Current 1 | Current | Nominal | 0.60 | A | 'OK'
84 | Current 2 | Current | Nominal | 0.00 | A | 'OK'
85 | Voltage 1 | Voltage | Nominal | 214.00 | V | 'OK'
86 | Voltage 2 | Voltage | Nominal | 212.00 | V | 'OK'
87 | PS Redundancy | Power Supply | Nominal | N/A | N/A | 'Fully Redundant'
88 | Status | Power Supply | Nominal | N/A | N/A | 'Presence detected'
89 | Status | Power Supply | Nominal | N/A | N/A | 'Presence detected'
90 | Pwr Consumption | Current | Nominal | 112.00 | W | 'OK'
91 | Power Optimized | OEM Reserved | N/A | N/A | N/A | 'OEM Event = 0001h'
130 | Drive 0 | Drive Slot | Nominal | N/A | N/A | 'Drive Presence'
131 | Cable SAS A | Cable/Interconnect | Nominal | N/A | N/A | 'Cable/Interconnect is connected'
132 | Cable SAS B | Cable/Interconnect | Nominal | N/A | N/A | 'Cable/Interconnect is connected'
139 | Power Cable | Cable/Interconnect | Nominal | N/A | N/A | 'Cable/Interconnect is connected'
140 | Signal Cable | Cable/Interconnect | Nominal | N/A | N/A | 'Cable/Interconnect is connected'
143 | Fan7A | Fan | Nominal | 4680.00 | RPM | 'OK'
144 | Fan1B | Fan | Nominal | 4440.00 | RPM | 'OK'
145 | Fan2B | Fan | Nominal | 4440.00 | RPM | 'OK'
146 | Fan3B | Fan | Nominal | 4440.00 | RPM | 'OK'
147 | Fan4B | Fan | Nominal | 4320.00 | RPM | 'OK'
148 | Fan5B | Fan | Nominal | 4440.00 | RPM | 'OK'
149 | Fan6B | Fan | Nominal | 4440.00 | RPM | 'OK'
150 | Fan7B | Fan | Nominal | 4440.00 | RPM | 'OK'
152 | A | Memory | Nominal | N/A | N/A | 'Presence detected'
153 | B | Memory | Nominal | N/A | N/A | 'Presence detected'
#
#
# ./freeipmi.plugin hostname testhost1.oob.net username admin password abc123
2019-02-07 14:40:06: freeipmi.plugin ERROR : MAIN : ipmi_monitoring_sensor_readings_by_record_id(): session timeout
2019-02-07 14:40:06: freeipmi.plugin FATAL : MAIN :freeipmi.plugin: data collection failed. # : Success
Hmmm, I see freeipmi isn't in the conf file...
# grep freeipmi /etc/netdata/netdata.conf
#
/usr/libexec/netdata/plugins.d/freeipmi.plugin
2019-02-07 17:55:49: freeipmi.plugin ERROR : MAIN : ipmi_monitoring_sensor_readings_by_record_id(): internal error (errno 2, No such file or directory)
2019-02-07 17:55:49: freeipmi.plugin FATAL : MAIN :freeipmi.plugin: data collection failed. # : Success
jimidak
on 8 Feb 2019
richard.glazier@netdata:~$ sudo /usr/libexec/netdata/plugins.d/freeipmi.plugin debug 1
2019-02-07 18:08:49: freeipmi.plugin ERROR : MAIN : update frequency 1 seconds is too small for IPMI. Using 5.
freeipmi.plugin: calling _init_ipmi_config()
freeipmi.plugin: calling ipmi_monitoring_init()
freeipmi.plugin: detecting IPMI minimum update frequency...
freeipmi.plugin: checking data collection speed iteration 1 of 10
2019-02-07 18:09:49: freeipmi.plugin ERROR : MAIN : ipmi_monitoring_sensor_readings_by_record_id(): internal error (errno 2, No such file or directory)
2019-02-07 18:09:49: freeipmi.plugin FATAL : MAIN :freeipmi.plugin: data collection failed. # : Success
Don't worry about the .netdat.conf file yet, the main issue to resolve is getting results from
./freeipmi.plugin hostname testhost1.oob.net username admin password abc123
The two additional options that may be needed are sdr-cache-dir PATH and sensor-config-file FILE.
From the error message, I suspect sdr-cache-dir PATH to be the culprit.
Do you know how to validate where freeipmi stores its cache? I read here the following...
--sdr-cache-directory=DIRECTORY
Specify an alternate directory for sensor data repository (SDR) caches to be stored or read from. Defaults to the home directory if not specified.
...but I'm not sure what is meant by 'home directory'
cakrit
on 8 Feb 2019
@jimidak if you can't make freeipmiplugin work i suggest to use ipmi_exporter + python custom module for scraping data from it. I can help you with it.
ilyam8
on 8 Feb 2019
We need to get this working, we have to understand why it doesn't. I will change the title because it's misleading.
cakrit
on 8 Feb 2019
it wasn't missleading. Better to open a new issue from this comment.
Now there is no connection between OP and title
ilyam8
on 8 Feb 2019
Difficult to more all comments to a new issue, perhaps @jimidak can edit his comment to more of a question regarding ipmi monitoring.
cakrit
on 8 Feb 2019
@jimidak if you can't make
freeipmipluginwork i suggest to use ipmi_exporter + python custom module for scraping data from it. I can help you with it.
Thank you very much!! I think that's all I need at this point, is a little help, and once the ball is rolling, all of this in the context of Netdata will make more sense. Should I write a py script the way I would want to get IPMI data back, and then bring it to you for help in getting it fed into a Netdata plugin?
jimidak
on 8 Feb 2019
Would you like me to open a new question, that is a little more focused, so you can close this one? Thank you both very much for the help.
jimidak
on 8 Feb 2019
I looked at the ipmi-exporter... Not sure I want to get in the weeds with complex Go scripts. I was hoping that the freeipmi-plugin would be able to do a remote call, as is referenced in its help output. I just figured I didn't have a correct config file or something simple. If it's suggested to not use the freeipmi with Netdata, for something else, I will probably just write scripts to grab IPMI data, since I know that works. I was hoping, if I got my scripts working with Netdata, that I could just contribute that to the Netdata PR.
jimidak
on 8 Feb 2019
I was hoping that the freeipmi-plugin would be able to do a remote call
Exactly it. ipmi_exporter does all heavy work. All metrics will be available at x.x.x.x::9290/metrics. Python will use this URL to get metrics.
- ipmi_exporte collects ipmi metrics
- python script (custom module) scrapes ipmi_exporter metrics endpoint, parses response and returns parsed data to netdata
Ok, let's postpone it. @cakrit is right, we need to investigate why freeipmiplugin is not working for you.
@cakrit any ideas how to debug it?
ilyam8
on 8 Feb 2019
Thanks for the help. I just re-imaged my VM and started from scratch. I can give verbose output, but here the short list of that I did.
- Ubuntu 16.04 VM, logged in as root
- ps -ef | grep -i ipmi; ps -ef | grep netdata (nothing)
- dpkg -l | grep -i -E 'ipmi|netdata' (nothing)
- ipmi TAB TAB (no output)
- apt-get update
- apt install freeipmi
- ipmi TAB TAB (I get output)
- ps -ef | grep -i ipmi (nothing started)
- apt install libipmimonitoring-dev
- bash <(curl -Ss https://my-netdata.io/kickstart.sh)
- no ipmi in /etc/netdata/netdata.conf
- there is a /usr/libexec/netdata/plugins.d/freeipmi.plugin
- ipmimonitoring -s "87,88,89" -h 10.117.6.190 -u admin -p abc123 -D LAN_2_0 (succeeds)
- /usr/libexec/netdata/plugins.d/freeipmi.plugin hostname 10.117.6.190 username admin abc123 (fails as root and non-root, by hostname and by IP)
jimidak
on 10 Feb 2019
for ipmimonitoring to work, I had to specify that it's ver. 2 of ipmi (-D flag). I don't see that that's an option in freeipmi.plugin
jimidak
on 10 Feb 2019
just for grins, I ran the netdata installer again, and this time I got a slew of ipmi stuff in /etc/netdata/netdata.conf!! I've never had any ipmi in there before, so that's progress. I uncommented the main 'freeipmi = yes', restarted netdata service, and tried again. It still doesn't work (same errors), but getting ipmi in the config file is new for me. Do I need to uncomment any of the other lines?
# grep ipmi /etc/netdata/netdata.conf
freeipmi = yes
[plugin:freeipmi]
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1024
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1024
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1024
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10240000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10240000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10240000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 10240000
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1
# dim freeipmi.plugin name = freeipmi.plugin
# dim freeipmi.plugin algorithm = absolute
# dim freeipmi.plugin multiplier = 1
# dim freeipmi.plugin divisor = 1
Also tried to run the plugin as netdata user, and failed as well (sudo su -s /bin/sh netdata)
jimidak
on 10 Feb 2019
Please read again this comment. Re-installing will not help, we need to configure netdata so that the freeipmi.plugin binary gives us data, just as ipmimonitoring does. i.e. we need to get what you were doing in https://github.com/netdata/netdata/issues/5344#issuecomment-461608092 working. Let me see if I can find any commands that would give us the location of your cache...
cakrit
on 11 Feb 2019
Let's see, I think I found something:
> sudo ipmi-sensors-config --diff
Caching SDR repository information: /root/.freeipmi/sdr-cache/sdr-cache-chris-msi.localhost
So if I was able to have ipmimonitoring working for my system, I believe I would need to pass the option --sdr-cache-directory=/root/.freeipmi/sdr-cache/ to freeipmi.plugin. sdr-cache-chris-msi.localhost is probably the cache itself in my case. Unfortunately I can't validate this, since freeipmi does NOT work on my system, but can you please try the same thing on your system to see if we can get the command line working?
Once we have all the options ironed out, you will have to put the correct options in netdata.conf, here:
[plugin:freeipmi]
command options =
On a different note, commented lines show what the default values are. Uncommenting something makes sense only if you want to change that default value.
cakrit
on 11 Feb 2019
Please read again this comment. Re-installing will not help, we need to configure netdata so that the
freeipmi.pluginbinary gives us data, just asipmimonitoringdoes. i.e. we need to get what you were doing in #5344 (comment) working. Let me see if I can find any commands that would give us the location of your cache...
Thanks. I just thought I'd start from scratch again on a clean VM, just so you could see my steps, in case I was missing something in my install/config.
jimidak
on 11 Feb 2019
Let's see, I think I found something:
> sudo ipmi-sensors-config --diff Caching SDR repository information: /root/.freeipmi/sdr-cache/sdr-cache-chris-msi.localhostSo if I was able to have
ipmimonitoringworking for my system, I believe I would need to pass the option--sdr-cache-directory=/root/.freeipmi/sdr-cache/tofreeipmi.plugin.sdr-cache-chris-msi.localhostis probably the cache itself in my case. Unfortunately I can't validate this, since freeipmi does NOT work on my system, but can you please try the same thing on your system to see if we can get the command line working?Once we have all the options ironed out, you will have to put the correct options in
netdata.conf, here:[plugin:freeipmi] command options =On a different note, commented lines show what the default values are. Uncommenting something makes sense only if you want to change that default value.
Thanks for this feedback! I am busy today, but will attend to this tonight. I will just say that ipmimonitoring works for remote, but not localhost, because it's a VM that doesn't support IPMI.
jimidak
on 11 Feb 2019
Yup, I'm on a VM, so I'm not going to cache any data...
r@netdata:~$ sudo ipmi-sensors-config --diff
Caching SDR repository information: /root/.freeipmi/sdr-cache/sdr-cache-netdata.localhost
ipmi_sdr_cache_create: internal IPMI error
r@netdata:~$
Again, I am not trying to get localhost IPMI; just remote host IPMI
jimidak
on 11 Feb 2019
Again, I am not trying to get localhost IPMI; just remote host IPMI
You're right, it shouldn't matter. I have great difficulty tracing this, because I don't have a server I can test it on. There's something we can try, without you giving me network access to whatever IPMI uses to collect info. That's to have you turn the IPMI debug flags on:
- git clone the netdata repo and cd to the directory
- edit file
collectors/freeipmi.plugin/freeipmi_plugin.cline 127 to make it like this:
unsigned int ipmimonitoring_init_flags = IPMI_MONITORING_FLAGS_DEBUG | IPMI_MONITORING_FLAGS_DEBUG_IPMI_PACKETS;
- run
sudo ./netdata-installer.sh - execute the freeipmi plugin binary again and do a pastebin of the output.
cakrit
on 12 Feb 2019
jimidak
on 12 Feb 2019
I suspect the version. You passed -D LAN_2_0 and netdata defaults to 1.5, without an option to modify it. I'd like to verify that this is it.
Please make line 80 of collectors/freeipmi.plugin/freeipmi_plugin.c as follows and try again:
int protocol_version = IPMI_MONITORING_PROTOCOL_VERSION_2_0;
or try ipmimonitoring -D LAN --debug to see if we get the same timeout result.
cakrit
on 13 Feb 2019
My ipmimonitoring works w -D and does not work without it. Not sure if that's what you are asking. I will add the line to the .c ipmi file and recompile
r@netdata:~# ipmimonitoring -s "87,88,89" -h 10.117.6.190 -u admin -p abc123 -D LAN_2_0
ID | Name | Type | State | Reading | Units | Event
87 | PS Redundancy | Power Supply | Nominal | N/A | N/A | 'Fully Redundant'
88 | Status | Power Supply | Nominal | N/A | N/A | 'Presence detected'
89 | Status | Power Supply | Nominal | N/A | N/A | 'Presence detected'
r@netdata:~# ipmimonitoring -s "87,88,89" -h 10.117.6.190 -u admin -p abc123
ipmi_ctx_open_outofband: session timeout
r@netdata:~#
This is it. You got the timeout without the -D option, because it defaulted to the1.5 version. The change I suggested will make the netdata plugin work as well. I will do a PR to add a config option to permit version selection in a little bit.
cakrit
on 13 Feb 2019
It worked!!! :) Thank you for being patient with me, and for ALL the great help!
r@netdata:~# /usr/libexec/netdata/plugins.d/freeipmi.plugin hostname 10.117.6.190 username admin password abc123
CHART ipmi.events '' 'IPMI Events' 'events' 'events' ipmi.sel area 90002 5
DIMENSION events '' absolute 1 1
CHART ipmi.sensors_states '' 'IPMI Sensors State' 'sensors' 'states' ipmi.sensors_states line 90001 5
DIMENSION nominal '' absolute 1 1
DIMENSION critical '' absolute 1 1
DIMENSION warning '' absolute 1 1
CHART ipmi.rpm '' 'System Fans read by IPMI' 'RPM' 'fans' 'ipmi.rpm' 'line' 90014 5
DIMENSION i61_n150_r2 'Fan7B i61' absolute 1 1000
DIMENSION i59_n149_r2 'Fan6B i59' absolute 1 1000
DIMENSION i57_n148_r2 'Fan5B i57' absolute 1 1000
DIMENSION i55_n147_r2 'Fan4B i55' absolute 1 1000
DIMENSION i53_n146_r2 'Fan3B i53' absolute 1 1000
DIMENSION i51_n145_r2 'Fan2B i51' absolute 1 1000
DIMENSION i49_n144_r2 'Fan1B i49' absolute 1 1000
DIMENSION i60_n143_r2 'Fan7A i60' absolute 1 1000
DIMENSION i58_n17_r2 'Fan6A i58' absolute 1 1000
DIMENSION i56_n16_r2 'Fan5A i56' absolute 1 1000
DIMENSION i54_n15_r2 'Fan4A i54' absolute 1 1000
DIMENSION i52_n14_r2 'Fan3A i52' absolute 1 1000
DIMENSION i50_n13_r2 'Fan2A i50' absolute 1 1000
DIMENSION i48_n12_r2 'Fan1A i48' absolute 1 1000
CHART ipmi.watts '' 'System Power read by IPMI' 'Watts' 'power' 'ipmi.watts' 'line' 90005 5
DIMENSION i119_n90_r2 'Pwr Consumption i119' absolute 1 1000
CHART ipmi.volts '' 'System Voltages read by IPMI' 'Volts' 'voltages' 'ipmi.voltages' 'line' 90012 5
DIMENSION i109_n86_r2 'Voltage 2 i109' absolute 1 1000
DIMENSION i108_n85_r2 'Voltage 1 i108' absolute 1 1000
CHART ipmi.amps '' 'System Current read by IPMI' 'Amps' 'current' 'ipmi.amps' 'line' 90013 5
DIMENSION i107_n84_r2 'Current 2 i107' absolute 1 1000
DIMENSION i106_n83_r2 'Current 1 i106' absolute 1 1000
CHART ipmi.temperatures_c '' 'System Celsius Temperatures read by IPMI' 'Celsius' 'temperatures' 'ipmi.temperatures_c' 'line' 90010 5
DIMENSION i15_n21_r2 'Temp i15' absolute 1 1000
DIMENSION i14_n20_r2 'Temp i14' absolute 1 1000
DIMENSION i1_n19_r2 'Exhaust Temp i1' absolute 1 1000
DIMENSION i4_n18_r2 'Inlet Temp i4' absolute 1 1000
BEGIN ipmi.events
SET events = 14
END
BEGIN ipmi.sensors_states
SET nominal = 77
SET warning = 0
SET critical = 2
END
BEGIN ipmi.rpm
SET i61_n150_r2 = 4440000
SET i59_n149_r2 = 4440000
SET i57_n148_r2 = 4440000
SET i55_n147_r2 = 4320000
SET i53_n146_r2 = 4440000
SET i51_n145_r2 = 4440000
SET i49_n144_r2 = 4440000
SET i60_n143_r2 = 4680000
SET i58_n17_r2 = 4800000
SET i56_n16_r2 = 4680000
SET i54_n15_r2 = 4680000
SET i52_n14_r2 = 4680000
SET i50_n13_r2 = 4680000
SET i48_n12_r2 = 4680000
END
BEGIN ipmi.watts
SET i119_n90_r2 = 112000
END
BEGIN ipmi.volts
SET i109_n86_r2 = 212000
SET i108_n85_r2 = 216000
END
BEGIN ipmi.amps
SET i107_n84_r2 = 0
SET i106_n83_r2 = 600
END
BEGIN ipmi.temperatures_c
SET i15_n21_r2 = 41000
SET i14_n20_r2 = 42000
SET i1_n19_r2 = 36000
SET i4_n18_r2 = 24000
END
BEGIN ipmi.events
SET events = 14
END
BEGIN ipmi.sensors_states
SET nominal = 77
SET warning = 0
SET critical = 2
END
BEGIN ipmi.rpm
SET i61_n150_r2 = 4440000
SET i59_n149_r2 = 4440000
SET i57_n148_r2 = 4440000
SET i55_n147_r2 = 4320000
SET i53_n146_r2 = 4440000
SET i51_n145_r2 = 4440000
SET i49_n144_r2 = 4440000
SET i60_n143_r2 = 4680000
SET i58_n17_r2 = 4800000
SET i56_n16_r2 = 4680000
SET i54_n15_r2 = 4680000
SET i52_n14_r2 = 4680000
SET i50_n13_r2 = 4680000
SET i48_n12_r2 = 4680000
END
BEGIN ipmi.watts
SET i119_n90_r2 = 112000
END
BEGIN ipmi.volts
SET i109_n86_r2 = 212000
SET i108_n85_r2 = 216000
END
BEGIN ipmi.amps
SET i107_n84_r2 = 0
SET i106_n83_r2 = 600
END
BEGIN ipmi.temperatures_c
SET i15_n21_r2 = 41000
SET i14_n20_r2 = 42000
SET i1_n19_r2 = 36000
SET i4_n18_r2 = 24000
END
BEGIN ipmi.events
SET events = 14
END
BEGIN ipmi.sensors_states
SET nominal = 77
SET warning = 0
SET critical = 2
END
BEGIN ipmi.rpm
SET i61_n150_r2 = 4440000
SET i59_n149_r2 = 4440000
SET i57_n148_r2 = 4440000
SET i55_n147_r2 = 4320000
SET i53_n146_r2 = 4440000
SET i51_n145_r2 = 4440000
SET i49_n144_r2 = 4440000
SET i60_n143_r2 = 4680000
SET i58_n17_r2 = 4800000
SET i56_n16_r2 = 4680000
SET i54_n15_r2 = 4680000
SET i52_n14_r2 = 4680000
SET i50_n13_r2 = 4680000
SET i48_n12_r2 = 4800000
END
BEGIN ipmi.watts
SET i119_n90_r2 = 112000
END
BEGIN ipmi.volts
SET i109_n86_r2 = 212000
SET i108_n85_r2 = 216000
END
BEGIN ipmi.amps
SET i107_n84_r2 = 0
SET i106_n83_r2 = 600
END
BEGIN ipmi.temperatures_c
SET i15_n21_r2 = 41000
SET i14_n20_r2 = 42000
SET i1_n19_r2 = 36000
SET i4_n18_r2 = 24000
END
2019-02-13 10:57:27: freeipmi.plugin ERROR : MAIN : ipmi_monitoring_sensor_readings_by_record_id(): internal error
2019-02-13 10:57:27: freeipmi.plugin FATAL : MAIN :data collection failed. # : Success
r@netdata:~#
It has been great tracing this with you. The proper solution is obviously a bit more involved. Please do us a favor and test PR #5384. The new command option is driver-type:
driver-type IPMIDRIVER
Specify the driver type to use instead of doing an auto selection.
The currently available outofband drivers are LAN and LAN_2_0,
which perform IPMI 1.5 and IPMI 2.0 respectively.
The currently available inband drivers are KCS, SSIF, OPENIPMI and SUNBMC.
So your new command will be /usr/libexec/netdata/plugins.d/freeipmi.plugin hostname 10.117.6.190 username admin password abc123 driver-type LAN_2_0.
To have netdata execute the plugin with these options of course, you'll need to give them in command options =, as mentioned in the documentation.
cakrit
on 13 Feb 2019
Yeah, my hunch early on was speaking ver. 2 of IPMI.
So should I blow away my netdata, do a fresh git clone, and run /usr/libexec/netdata/plugins.d/freeipmi.plugin hostname 10.117.6.190 username admin password abc123 driver-type LAN_2_0?
jimidak
on 13 Feb 2019
Yeah, my hunch early on was speaking ver. 2 of IPMI.
So should I blow away my netdata, do a fresh git clone, and run
/usr/libexec/netdata/plugins.d/freeipmi.plugin hostname 10.117.6.190 username admin password abc123 driver-type LAN_2_0?
nevermind, I just looked as the other link, and it looks like a PR that hasn't been accepted yet, so the driver-type option won't be there if I pull it down. How did you want me to test that new flag/option for freeipmi.plugin?
jimidak
on 13 Feb 2019
@jimidak
mkdir test
cd test
git clone https://github.com/cakrit/netdata --branch=freeipmiversion --depth=1
cd netdata
sudo ./netdata-installer.sh --install /opt
...
profit!
reinstall master branch, remove test/* when you are done with testing
ilyam8
on 13 Feb 2019
That worked!!
r@netdata:~/netdata/test/netdata# ./freeipmi.plugin hostname 10.117.6.190 username admin password abc123 driver-type LAN_2_0
CHART ipmi.events '' 'IPMI Events' 'events' 'events' ipmi.sel area 90002 5
DIMENSION events '' absolute 1 1
CHART ipmi.sensors_states '' 'IPMI Sensors State' 'sensors' 'states' ipmi.sensors_states line 90001 5
DIMENSION nominal '' absolute 1 1
DIMENSION critical '' absolute 1 1
DIMENSION warning '' absolute 1 1
CHART ipmi.rpm '' 'System Fans read by IPMI' 'RPM' 'fans' 'ipmi.rpm' 'line' 90014 5
DIMENSION i61_n150_r2 'Fan7B i61' absolute 1 1000
...
I will post in the PR as well....
jimidak
on 13 Feb 2019
It has been great tracing this with you. The proper solution is obviously a bit more involved. Please do us a favor and test PR #5384. The new command option is
driver-type:driver-type IPMIDRIVER Specify the driver type to use instead of doing an auto selection. The currently available outofband drivers are LAN and LAN_2_0, which perform IPMI 1.5 and IPMI 2.0 respectively. The currently available inband drivers are KCS, SSIF, OPENIPMI and SUNBMC.So your new command will be
/usr/libexec/netdata/plugins.d/freeipmi.plugin hostname 10.117.6.190 username admin password abc123 driver-type LAN_2_0.To have netdata execute the plugin with these options of course, you'll need to give them in
command options =, as mentioned in the documentation.
What is the "command options =" syntax in netdata.conf, for a plugin?
jimidak
on 19 Mar 2019
What is the "command options =" syntax in netdata.conf, for a plugin?
They are the options passed to the plugin executable, in this case freeipmi.plugin. You can see them when you execute it with -h and they are also shown here. So you can run it via the command line to ensure you have the options you need to get the correct data and then add those options to that config parameter, so netdata executes it the same way.
cakrit
on 19 Mar 2019
Thank you @cakrit ! I was running -h on the command line to know what options there were, and running with various options for freeipmi.plugin, and it was working, but I can't get the same results when putting those options in the netdata.conf file. I have tried to add the options one line at a time, and also tried comma sperated after the "command options =", and other ways as well, but none seem to work. I restart netween between each try. I can't find an example syntax for netdata.conf.
As an example, freeipmi.plugin -h shows option hostname HOST and username USER and driver-type IPMIDRIVER. On the command line that works for me.
How would that look in the nedata.conf file? I have tried....
[plugin:freeipmi]
update every = 1
command options =
hostname host.network.net
username useraccount
driver-type LAN_2_0
and also with "=" between the option and value
and also like this
[plugin:freeipmi]
update every = 1
command options = hostname host.network.net, username useraccount, driver-type LAN_2_0
Also tried double quoting string values.
I've been looking in the documentation and files themselves, but can't find netdata.conf example syntax.
jimidak
on 19 Mar 2019
[christopher@chris-msi netdata]$ /usr/libexec/netdata/plugins.d/freeipmi.plugin hostname host.network.net username useraccount driver-type LAN_2_0
command options = hostname host.network.net username useraccount driver-type LAN_2_0
Can you also do me a favor? Please execute the command with debug, i.e.
/usr/libexec/netdata/plugins.d/freeipmi.plugin debug hostname host.network.net username useraccount driver-type LAN_2_0
And tell me what you see here:
host.network.net: =====================================================
host.network.net: IPMI 1.5 Get Channel Authentication Capabilities Request
host.network.net: =====================================================
It should have been IPMI 2.0 or something like that instead of 1.5, but perhaps I don't get it because I don't have a host to try it on.
cakrit
on 19 Mar 2019
So I should be able to add the values to netdata.conf, and then run again, and it should work?
ie.
锘匡豢[plugin:freeipmi]
update every = 10
command options = hostname 10.117.6.190 username admin password abc1 driver-type LAN_2_0
then
/usr/libexec/netdata/plugins.d/freeipmi.plugin
Is that right?
jimidak
on 19 Mar 2019
Actually, netdata uses the options to execute freeipmi.plugin with them. So from the command line you'll always need to pass the options. The executable itself just parses the arguments passed to it.
After updating the configuration, you restart netdata and it should show you the charts.
cakrit
on 19 Mar 2019
I guess I don't understand netdata architecture well enough yet. So what I want to do is:
I have one host (_host1_), that I am graphing data for, on default port 19999. I want remote IPMI data from one remote host also graphed on _host1_'s graph. What is the best way to accomplish that?
Thanks for the help.
jimidak
on 19 Mar 2019
Ok, I have my remote IPMI host now being graphed in my local 19999! yea! Now I'm messing around with getting more than one remote host monitored....
jimidak
on 19 Mar 2019
Hm... I don't think we're able to monitor multiple hosts as it is. I'm checking with @ktsaou just in case I missed something, but it looks like we need to extend the plugin to support this. I'll verify and open a new feature request. You could of course run another netdata instance and push the metrics from both instances to a master netdata (see streaming and replication), but that's obviously an overkill for this.
cakrit
on 20 Mar 2019
Thanks for getting back!! I tried to fudge a way to get X remote hosts, with no luck. I do have 1 remote host IPMI data being pulled into the localhost graph, under the IPMI section, and I can ignore certain metrics in the conf file, and it get's reflected in the graph. Cool stuff.
jimidak
on 20 Mar 2019
For a 2nd host, do this please:
cd /usr/libexec/netdata/plugins.d/
sudo cp freeipmi.plugin freeipmi2.plugin
sudo chown root:netdata freeipmi2.plugin
sudo chmod 4750 freeipmi2.plugin
Update netdata.conf, adding the following:
[plugin:freeipmi2]
update every = ...
command options = ...
Restart netdata
If it works well, I'll add it to the docs.
cakrit
on 21 Mar 2019
For a 2nd host, do this please:
cd /usr/libexec/netdata/plugins.d/ sudo cp freeipmi.plugin freeipmi2.plugin sudo chown root:netdata freeipmi2.plugin sudo chmod 4750 freeipmi2.pluginUpdate
netdata.conf, adding the following:[plugin:freeipmi2] update every = ... command options = ...Restart netdata
If it works well, I'll add it to the docs.
Thx, i was searching for this, but it is not working.. i see only one IPMI in the menu.
svdblom
on 9 Jun 2019
I looked at the code again, it's because even the second one would send exactly the same chart.
There's no workaround that will work then, we just need to change the code, to support additional hosts.
Modifying the title and putting this on the backlog.
cakrit
on 10 Jun 2019
Related issues
ldelossa
路
107Comments
ScrumpyJack
路
66Comments
ktsaou
路
116Comments
titpetric
路
64Comments
UltimateByte
路
68Comments