Raspiblitz: Telegraf as bonus for collecting system metrics

[frennkie]

Telegraf is a nice solution to gather system metrics that can be collected locally or remotely in a InfluxDB and this can then be displayed in Grafana dashboard. https://medium.com/@dorian599/iot-raspberry-pi-container-and-system-monitoring-with-influxdb-telegraf-and-grafana-a1767c38c109

sudo -i 

curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
DISTRIB_ID=$(lsb_release -c -s)
echo "deb https://repos.influxdata.com/debian ${DISTRIB_ID} stable" | tee /etc/apt/sources.list.d/influxdb.list
apt-get update
apt-get install -y telegraf
sudo usermod -aG video telegraf
setcap 'cap_net_admin,cap_net_raw+ep' $(which ping)

Get a RaspiBlitz specific telegraf.conf from somewhere and put in /etc/telegraf/telegraf.conf

systemctl restart telegraf

[PatrickScheich]

Hi, I would like to contribute to this topic. How can I do so?

I had created a telegraf.conf for my own raspiblitz and a matching grafana dashboard

I'm quite keen with programming, etc... but haven't yet worked on a github project
so I might need some pointers there

kindest regards, patrick

[rootzoll]

@PatrickScheich In the next week we will produce some introduction tutorials on how to start contributing. But basically you can start small with the following:

• get it working on your own raspiblitz
• edit the /mnt/hdd/app-data/custom-installs.sh in a way that it contains all needed setup shell commands so that it will reinstall the service sucessfully if you start your RaspiBlitz with a fresh sd card (update/recovery) - check /home/admin/raspiblitz.recover.log for debug output after recovery.
• then you can can post that shell script code in this issue here .. we can point you to the next steps from there

[PatrickScheich]

Great, I will take a look into this.

Unfortunately telegraf needs a bunch of individual decisions on config settings like

• ip and port of the backend influxdb
• name of the target database
• credentials for the database

I will assume that proper entries will exists in raspiblitz.conf

[rootzoll]

Check if you can set those parameters on the install commands you can add to custom-installs.sh so that they dont need interaction ... in the next step when turning custom-installs.sh into an install script those parameters then get written into the raspiblitz.conf and the install script gets called using those config values during the recover process.

[PatrickScheich]

here we go, it took a bit of time...

attached you find the files that should go to /mnt/hdd/app-data.
plus a snippet for raspiblitz.conf
beside the telegraf part I added some code that autocopies my ssh-key into /root/.ssh/authorized_keys

app-data-for-custom-install.zip

The Grafana dashboards are a bit tricky
In my installation they are working fine, but they are not yet generic
(with proper variable setup for datasource name and hostname of the raspiblitz)
I surely can provide the JSON model for reference if helpful. Let me know.

[PatrickScheich]

Screenshots of Grafana dashboards

Quick Summary:

System and services uptime and connections/peers:

Blockchain and Mempool:

Basic Computer Metrics:

[frennkie]

Coming along nicely... @PatrickScheich your plan is to run the influxdb and Grafana locally on the RaspiBlitz, correct?

I already have a online VPS running both and I also have more than one active Raspiblitz. So it would be great it you take this into consideration for the Grafana dashboard.. 😊

I surely can provide the JSON model for reference if helpful. Let me know.

Having a nice would definitely be great.

[PatrickScheich]

your plan is to run the influxdb and Grafana locally on the RaspiBlitz, correct?

@frennkie no, only _telegraf_ will run on the raspiblitz for metric collection.
influxDB and Grafana will be located elsewhere

[frennkie]

Can you share the telegraf config file and the Grafana JSON?

[PatrickScheich]

sure,

• the telegraf.conf is in the zip file
• the grafana JSON is attached here grafana.json.txt

[frennkie]

1. You hardcoded your node name - this should be a variable (very easy to do - see below)

1. You used a panel (in the Quick Summary) that's marked a deprecated in my Grafana installation - would make sense to use the current version.

[frennkie]

Can you check this updated version on your Grafana: grafana.json.txt

I also added the telegraf user to the lndadmin group using:

sudo usermod telegraf -a -G lndadmin

and changed the lnddir here:

[[inputs.exec]]
  interval                    = "60s"
  commands                    = ["/usr/local/bin/lncli --lnddir=/mnt/hdd/app-data/lnd getinfo" ]
  name_override               = "ln-info"
  data_format                 = "json"

[frennkie]

The installation of telegraf that you put into custom-installs.sh would be best go into a bonus script in the ~/config.scripts directory. There are many examples (and now there is even a Youtube video explaining this) how to do it. Could you work on a PR?

[PatrickScheich]

Could you work on a PR?

Hmm, I had not worked with this feature of github yet.
I used to work locally with git and VSC but have no idea I how to collaborate

(and now there is even a Youtube video explaining this)

Maybe you could point me to the video, I will check it first

There are many examples

can you name some, I will look into them and try to adopt

thanks for the hint with Grafana variables

[frennkie]

https://youtu.be/ZVtZepV3OfM

This was recorded this week.

Regarding git and GitHub ... There are many resources... Just try one or two tutorials and see what works best for you. Definitely a must have skill - IMHO.. 😉

[PatrickScheich]

@frennkie I opened the PR