Lizardfs: cgiserv view hard disks timed out

[biocyberman]

OS: Ubuntu 16.04
Lizardfs: 3.10

It has been working fine since some days ago when I first installed the system. Now I don't know what happened but I can't view disks:

http://myhost:9425/mfs.cgi?sections=HD

Traceback (most recent call last):
  File "/usr/share/mfscgi/mfs.cgi", line 1423, in 
    s.connect((hostip,port))
  File "/usr/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 110] Connection timed out

I restarted the cgiserv service, it did not help.
I will do something more later but not now because the server is busy.

[onlyjob]

A duplicate of #407 ?

[biocyberman]

That can well be it, even though I don't know where to look for client log. I am indeed using btrfs. What should I do then? Changing file system is not an option for now.

[onlyjob]

You should be able to check syslog for kernel messages and for chunkserver messages.
Btrfs is not a safe option for chunkserver storage as you need to run latest kernel (ideally) as well as to do regular maintenance as described in https://github.com/lizardfs/lizardfs/issues/417#issuecomment-229556206

[biocyberman]

@onlyjob Further experimenting with my setup I found that the hanging happens whenever I start the chunkserver which has a very full disk (100 % used of 9TB volume). This RAID 5 disk volume is formatted in ext4 actually. I have chunk directory under /mnt/mydisk/lizardfs, while I still have other non-LFS under /mnt/mydisk/. So it is not the same case you described.

[onlyjob]

Actually it is almost the same issue -- poor behaviour of nearly full underlying file system.
Setting HDD_LEAVE_SPACE_DEFAULT to higher value in mfschunkserver.cfg should help.

Also RAID5 is harmful and provides no benefits to LizardFS. Eventually it would be better to configure LizardFS to use HDDs directly.

[biocyberman]

I removed some non-LFS data from /mnt/mydisk and free space is now about 50%. The problem still remains.

RAID5 is harmful and provides no benefits to LizardFS. Eventually it would be better to configure LizardFS to use HDDs directly.

You raised a fundamental problem that I don't find any good documentation for: how to prepare hard drive and filesystem for LFS.

How harmful is RAID5? I would think it only provide least protection for data safety. And even though not as good as RAID6 or RAID10 for data safety, it is still better than none. And read/write speed is generally higher than RAID6, while give more usable space. So I think RAID5 still gives a little bit of protection. In particular, in case file goal is 1, RAID5 or other RAID is better than non-RAID.

Moreover, should I use LFS directly to dozens of disks on a server? I've never tried having dozens of /dev/sdXX there and mount them.

[onlyjob]

I removed some non-LFS data from /mnt/mydisk and free space is now about 50%. The problem still remains.

Which probably means that the storage is not causing the problem...

how to prepare hard drive and filesystem for LFS

Just format the file system (_ext4_, _xfs_, etc.), create _fstab_ record and add corresponding entry to _mfshdd.cfg_ for each disk.

How harmful is RAID5?

Quite harmful. It gives illusion of protection as degraded RAID-5 is much more vulnerable than single non-redundant disk. RAID-5 is not maintainable because removing HDD for replacement exposes data to unacceptable risks. RAID-5 exposes HDDs to symmetric load and considering that usually HDDs are not diversified among vendors and models it is likely that catastrophic failure of second HDD in RAID-5 happens during recovery after failure. IMHO RAID-5 is unreliable and it should never be used.

in case file goal is 1

Replicated file system probably should never use goal=1 unless data is not valuable in which case is shouldn't matter if it is on non-redundant disks.

should I use LFS directly to dozens of disks on a server?

By all means. Yes. Unlike RAID-5, failure of standalone HDD will have only little effect on performance.
Besides load is distributed among standalone HDDs when RAID-5 reads from all disks (minus one) therefore your I/O would be N times better on standalone disks than on RAID-5 (where N is number of HDDs).

[biocyberman]

@onlyjob I am trying to reverse the setup to the one you recommended: Serving separate disks to LFS directly. There is a problem:

One of the RAID volumes contains 10TB of chunks data, which appears to chunkserver service as one HDD, something like: /mnt/RAIDvolume/lizardfs/chunkXX. Obviously this amount of data is bigger than any single disk. And I don't want to copy my data via lizardfs-client mount because of slow speed. So, suppose I backup /mnt/RAIDvolume/lizardfs/ to another server with plain ssh-rsync command; how do I split that huge directory into those single disks? I guess there is not any simple way at the moment.

Will I have to backup via lizardfs-client mount anyway?

[4Dolio]

I have a set of scripts that does exactly this. It will source from X number of disks and copy or move them to Y number of target disks. And can also do so for 1/z number of chunk files. Ill try to share it soon...

[biocyberman]

@4Dolio That will be great. Would you be able to put it up today on github? :)

[4Dolio]

The proper way to do this would be to add to the cluster a new chunkserver with say 3*4TB disks (12TB) and increase the goal so that when you add that server the cluster will just replicate the data to the new disks. Then tear down the RAID and set back up as individual disks and let the cluster rebalance/replicate back to them again.

Sigh, you asked for it... in all it's glory...

• BTW, I am not sure that anyone should actually use this code. I use this script because I feel like it is faster than letting the cluster do it on it's own, I only use it to manually rebalance between old/new disks on a single server, not between chunk servers, I could be completely wrong and perhaps inter-disk rebalancing is awesome... I recommend using native natural replication to do your rebalancing between servers or disks...

Some explanations:

• I call my disks or devices or hdds "Bricks"
• I run multiple clusters, some chunk servers service multiple clusters with the same disks
  
  
  
  • To avoid conflict my brick mountpoints are something like /mfsbrick.XX.{00..23} (24 disks) or /LFSX820{b..k} (10 disks)
  
  
  • Next is a sub-folder matching the name of the cluster (Even if only servicing one cluster having a sub-folder is a good ideas because the chunk service can not initialize a new empty set of 00..FF folders if the disk did not get mounted because the sub-folder will not exist when not mounted)
  
  
  • Within the clusters sub-folder are folders 00..FF (which has changed to "chunks{00..FF} which this script does not do currently(I'm running older Lizard still))
  
  
• There is a ratio you give the script to tell it roughly what % of chunks you want to act on..
  
  
  
  • The source chunk files are sorted by size to help try and keep things even.
  
  
  • Ratio 1 will move/copy every chunk file (Equal sized Brick to a replacement, silly to use this script)
  
  
  • Ratio 2 will move/copy every other chunk file ( Big Brick to a smaller Brick )
  
  
  • To redistribute from 8 bricks to a new 9th I would use a ratio of 9 and then run the script 8 times in parallel, sourcing each of the 8 original bricks and copying 1/9th of their chunk files to the new disk.
  
  
• Once the ReDistributions have finished you need to carefully restart/reload the chunk service in a particular way. The ChunkService will delete duplicate chunk files as it scans a brick, so...
  
  
  
  • Comment out the original bricks and uncomment/activate only the new disk(s) and restart the service.
  
  
  • Once it finishes scanning, enable to original brick(s) again and reload the service, this scan can be rather slow as the duplicate chunks are removed from their original bricks while the scan progresses. The result is that the copied bricks stay on their new device and the originals are purged from their source device(s).
  
  

#!/bin/bash

Debug=1; # unset Debug or comment out # test -z $Debug && echo NoDebugging ; test ! -z $Debug && echo Debugging
Verbo=1; # Be slightly noisy to indicate activity
if   [[ -z `which bc` ]] ; then echo " sudo apt-get install bc # we need bc in order to do math, please install it, exiting." ; exit ; fi
if   [[ `hostname -s` == "ChunkServer00"    ]] ; then B=/mfsbrick.00. # {00..20}
elif [[ `hostname -s` == "ChunkServer02" ]] ; then B=/mfsbrick.02. # {00..20}
elif [[ `hostname -s` == "ChunkServer820"  ]] ; then B=/LFSX820      # {b..k}
else echo " This host is not recognized as a Chunk Server with multiple brick devices. exiting." ; exit
fi
Operations="info copy move"
if ! [[ "$Operations" =~ "$1" ]] ; then echo -e " First option must be an Operation type such as: $Operations" ; exit ; fi
Opp="$1"
Clusters="mfs-data lfs-data-ssd lfs-data-test"
if ! [[ "$Clusters" =~ "$2" ]] ; then echo -e " Second option must be cluster name which is known, choose from:\n$Clusters" ; exit ; fi
C="$2"
# Ratios need to be able to have decimals in order to have enough fine grain control.  The following should work
# last=0;ratio=5.5;for r in {0..100};do this=$( echo "$r / $ratio" | bc ) ; echo -e "\t$r\t$ratio\t"$this ; if [[ $last -ne $this ]] ; then echo -n CP ; fi; last=$this ; done | paste - - - - -
## As en example if our sources are 18..20 with 136k chunks each and our targets are {00..17} with 75k each, then
## moveing 53k from each source to each target will result in 53*3 = 159/18 = 75k+8.8k=83.8k at targets and 136k-53k=83k at sources.
## 83.8 ~ 83k is fairly closely ballanced, so 136/53 = 2.566 as our ratio value for a chunk move process...
## Please for the love of math come up with a better explanation and formula for figuring this all out...
if [ -z $3 ] ; then echo -e " Third option must be the ratio value for how many of the source files to act upon
  1=every file, 2=every other file, 2.5= every 2.5 files. \n" ; exit ; fi
ratio=$3 ; ratiovalue=1 ; ratiovaluelast=1 # 1 will trigger on the first object, 0 will not?
if [ -z $4 ] ; then echo -e " Fourth option is the source bricks as a CSV list such as:\n  00,01,02,03"     ; exit ; fi
IFS=, read -a Sources <<<"$4" ; SourcesC=$((${#Sources[@]}-1))
if [ -z $5 ] ; then echo -e " Fifth option is the destination bricks as a CSV list such as :\n  18,19,20" ; exit ; fi
IFS=, read -a Targets <<<"$5" ; TargetsC=$((${#Targets[@]}-1)) ; TF=${Targets[0]} ; TL=${Targets[TargetsC]} ; TC=$TF ; Ti=0
test ! -z $Debug && ( echo -e "\tSources=${Sources[@]} ($SourcesC)\n\tTargets=${Targets[@]} ($TargetsC)\n\tFirstTarget=$TF\tLastTarget=$TL" ) # Debug the sources and targets arrays
test ! -z $Debug && ( echo -e " Debugging is enabled, No actual opperations will be executed.\n  Modify script to disable debugging by commenting out variable." )
test   -z $Debug && (
 echo -en "\t"`date`"\n\t\t From source bricks ${Sources[@]}\n\t\t  To Target bricks ${Targets[@]}\n\t\t   Opperation Type:$Opp 1/$ratio of chunk files by size"
 echo -e "\n !! Debugging is disabled !! Press any key to begin ReDistribution of Chunks..." ; read foo )

#root@ChunkServer02:~# ./ReDistChunks.sh info lfs-data 2.566 18,19,20 00,01,02,03,04,05,06,07,08,09,10,11,12,13,14,15,16,17 | head -n30 # as an example
for source in ${Sources[@]} ; do
 echo -en "\n\t"`date`" $Opp from $B$source/$C: "
 for d in {0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F}{0,1,2,3,4,5,6,7,8,9,A,B,C,D,E,F} ; do
  cd $B$source/$C/$d ; echo -en "$d " ; for sc in `ls -1S` ; do
   ratiovaluenow=$( echo "$ratiovalue / $ratio" | bc )
   if [ $ratiovaluelast -ne $ratiovaluenow ] ; then # If ratiovaluenow value it not equal to ratiovaluelast because the quotient of the division operation has changed then we do the $Opp
    test ! -z $Verbo && ( echo -n . ) # when verbos print . when executing Opp for each file
    if [[ "$Opp" == info ]] ; then
     test ! -z $Debug && ( echo -e "\t\t$Opp $B$source/$C/$d/$sc " ) # Debug $Opp Operation
     test   -z $Debug && ( ls -1 $B$source/$C/$d/$sc 2> /dev/null ) # Execute info $Opp Operation
    elif [[ "$Opp" == copy ]] ; then
     test ! -z $Debug && ( echo -e "\t\t$Opp $B$source/$C/$d/$sc $B$TC/$C/$d/" ) # Debug $Opp Operation
     test   -z $Debug && ( cp -a $B$source/$C/$d/$sc $B$TC/$C/$d/ 2> /dev/null ) # Execute copy $Opp Operation
    elif [[ "$Opp" == move ]] ; then    
     test ! -z $Debug && ( echo -e "\t\t$Opp $B$source/$C/$d/$sc $B$TC/$C/$d/" ) # Debug $Opp Operation
     test   -z $Debug && ( mv $B$source/$C/$d/$sc $B$TC/$C/$d/ 2> /dev/null ) # Execute move $Opp Operation
    fi
    TC=${Targets[$Ti]};if [ $Ti -eq $TargetsC ] ; then Ti=-1 ; fi ;Ti=$(($Ti+1)) # Incriment and loop a Target(index) value with which to reference a TargetArrayElement
   else # the ratiovaluenow vs ratiovaluenow quotient did not change so we skip this $Opp
#    test ! -z $Verbo && ( echo -n - ) # when verbos print - when we skip a file
    if [[ "$Opp" == info ]] ; then
     test ! -z $Debug && ( echo -e "\tNo\t$Opp $B$source/$C/$d/$sc $B$TC/$C/$d/" ) # Debug Skipped $Opp Operation
    elif [[ "$Opp" == copy ]] ; then
     test ! -z $Debug && ( echo -e "\tNo\t$Opp $B$source/$C/$d/$sc $B$TC/$C/$d/" ) # Debug Skipped $Opp Operation
    elif [[ "$Opp" == move ]] ; then    
     test ! -z $Debug && ( echo -e "\tNo\t$Opp $B$source/$C/$d/$sc $B$TC/$C/$d/" ) # Debug Skipped $Opp Operation
    fi
   fi ; ratiovalue=$(($ratiovalue+1)) ; ratiovaluelast=$ratiovaluenow # Incriment the ratio counter and ratiovaluelast to ratiovaluethis
  done
 done
done
echo;echo -e "\tDone" `date`
##

[biocyberman]

Now you say it :) I agree that the native replication process is easy and peaceful. I did not want to use it because I don't see any way to make the replication complete faster. It could take weeks or months to do the replication twice (one migrate out, one migrate back in). Besides, the chunkserver need to access to its own data at the speed equivalent to local drives.

In short, I will try to adapt your script to my case. You came to my rescue. Thank you.

Update:

@4Dolio I think more about the "native" way you said, and decided I will do this way:

1. Copy the chunks to the backup server ( I did)
2. Shutdown the old chunkserver
3. Configure the backup server as a temporary chunkserver.
4. Tear down the RAID on the old chunk server and remake a new chunkserver.
5. Initiate migration of the data from backup server back into the LFS.

This approach is the middle way that avoid the trouble of manually handing the chunks while reduce the time of the backup step.

This topic is going out of the original problem, so I will close it here.