Lizardfs: Request: Send email to warn about running out of critical disk space

[biocyberman]

In my experience with Lizardfs and communication with the support team, the problem of lacking critical disk space is a serious threat:
Meta data corruption
Master nodes fail to start (I experienced this)

This disk usage is not the same as disk space for chunks. And even space for chunks also cause troubles as I saw here an there in the list of issues here at this git repo.

So a rather proactive prevention is to check periodically and send email to administrators to warn about disk space problem.

A poor man solution would be to have a bash script run by a cron job that checks disk space and send email with send mail. But if this can be built into LizardFS itself, it will be more convenient and make Lizardfs a more thoughtful solution, which can take care of some ugly and likely situations.

[psarna]

I think that sending e-mails should be a responsibility of external monitoring systems like nagios, zabbix etc. LizardFS exposes a lot of information through lizardfs-admin, and these can be used to feed monitoring scripts.

What we could potentially do is making it possible to register user-defined scripts for certain LizardFS events (like entering out-of-space state), but it's not a priority now. My personal opinion is that external monitoring systems with the help of lizardfs-admin are sufficient for the job.

[guestisp]

Probably would be better to make lizard able to send SNMP traps in case of events

Running external scripts is not standard in any way. SNMP traps are a well defined standard and any trap receiver can then run any arbitrary script, if needed

[biocyberman]

@psarna I am not very familiar with lizardfs-admin in this aspect. To be clear, I am more concerned about disk space, and RAM* usage say in percentage for meta data. This information is not visible via CGI page, and can be overlooked. Information about disk usage for chunkservers is also important, but relatively less vital than for meta data. If I had to use lizardfs-admin with external tool, it would be the same as using native system tools, no gain.

As an extension for this feature, it is also very useful when LizardFS can also report a hard disk failure or a mountpoint failures (i.e. for chunkservers)

*Note: About RAM: I see 100 times or more increase of RAM usage when I use snapshots (from 340MB jumping to 34GB) . This much use of RAM is unexpected for me and not desirable at all. One of the uraft nodes choked because it has only 33GB of RAM. I had to reduce number of snapshots.

[4Dolio]

I agree with @psarna that such checks are the job of an administrator to be aware of and monitor for and alert on when critical. I recently encountered a 12 hour regression of metadata due to master lacking disk space and my HA doing a transition into a shadow which lacked current metadata.. but this was al entirely my fault and not the fault of lfs directly. My notification and HA implanentation needs to be more robust, i can not expect lfs to save me from doing something stupid.

[biocyberman]

@4Dolio Would you not like to make it less busy to administrating the cluster? And it will be come harder when the cluster growth to dozens of nodes.

[4Dolio]

@biocyberman I do not have a strong opinion either way. As an admin I already monitor and alert many other non LFS disks and mem use. I prefer a tool, LFS in this case, stick to doing what it does and not try to solve tertiary or tangent aspects of how anyone might leverage such a tool. So... Shrugs..

[biocyberman]

Fair enough for you. In my opinion however, lacking this feature is to leave LSF open to data loss or corruption. It then passively depends on the system admin with their own implementations of resource monitoring scripts to take care of the problem. Some will have to learn the hard way.

[guestisp]

Fair enough for you. In my opinion however, lacking this feature is to leave LSF open to data loss or corruption. It then passively depends on the system admin with their own implementations of resource monitoring scripts to take care of the problem. Some will have to learn the hard way.

+1

It's the same with the lack of a proper scrub procedure.

[onlyjob]

IMHO this is redundant to proper monitoring solution which should be used to monitor all sensitive systems anyway. Monitoring available disk space is a generic task that would be best to leave to more specialised tools like Zabbix, etc.

[Blackpaw]

Monitoring available disk space is a generic task that would be best to leave to more specialised tools like Zabbix, etc.

Agree.