Lizardfs: are ec goals trustworthy? - question

[eleaner]

Dear all

Considering 3.13.rc and 3.12 versions, would use of ec goals be a good idea?

I seem to spot once upon a time a thread that spurious problems are reported.
I am considering to replace my btrfs raid1 with a lizard with ec target and would not like to regret it.

[onlyjob]

746 is still affecting 3.13 and so your EC chunks will be removed/deleted if you upgrade to 3.13.0\~rc1. EC is fine on current 3.12 release (and _maybe_ fine on 3.13.0\~rc1 on new clusters).

It is a good idea to replace terrible Btrfs RAID but your real concern should be #805 (health of this project).

[eleaner]

Thanks, @onlyjob
aye to all you say.
The problem is that there is hardly anything to compete with lizard - being free and opensource.
So I assume my best bet is 3.12
Second-best is a new cluster on 3.13.0~rc1
Everything on this level is either bought or/and ceases to exist (a conspiration theory right here)
Moose is going strong, but ec is available only in pro version (too steep price)
BTRFS might be terrible (I agree with points you mentioned in another post) but has been running stable for two years now. I am mostly pissed off with poor resilience and still factor x2 on the storage.

[onlyjob]

RozoFS could be an alternative and MooseFS is awesome even without EC chunks because it is also free from most LizardFS bugs. MooseFS is well and actively maintained.

EC chunks are for limited applications (e.g. long term archival of large write-once files). EC chunks have significant costs in number of chunks (i.e. RAM on master and performance of chunkservers in regards to number of chunk files), I/O, required number of chunkservers and configuration of goals for careful planning of data placement.
Also changing goal of EC chunks is very I/O demanding.

Stable and reliable behaviour of your storage system is more important than storage efficiency. At the moment EC chunks require good understanding of caveats and implementation. Maintenance overhead is significant.

Having used EC goals for some years, I think that cost of extra HDDs for replicated storage is less than cost/time of maintenance of LizardFS EC storage.

I would go ahead with MooseFS because it is mature and well maintained.

---

If you are using Btrfs RAID then you don't have much data at all therefore EC chunks should not be your priority. If you can put your data on two servers then you can have much greater resilience to hardware failures, you can spread I/O, etc. Capacity growth will not be limited by configuration of your RAID.

[eleaner]

thank you @onlyjob
so now you've got me thinking

How safe is actually goal 2 (I am not comparing to BTRFS - obviously superior)

Imagine two machines with six HDD each. One chunk server for each HDD.
Let's say I use goal 2 with chunks evenly distributed between machines. So is the HDD space.
So I am protected from one hardware failure and up to 6 drives but they have to be on the same machine.
Technically I am safe in 100% from the failure of only one HDD. Failure of any second one at the same time increases the probability of my data being busted.

Considering that drives are not equal sizes an alternative could be to use ecc2.2 or ecc3.3 or maybe even ecc4.4 increasing my safety zone

But this comes with LizardFS risks as well as maintenance overhead you mentioned above.

What setup would you choose?

[njhurst]

Running a few hundred TB on lizard, I've not seen any of the issues others have had with EC. I'm running {5,3} and have not (yet) lost any data. That's deterministically safe up to 3 drives lost, with a 60% overhead (compared to 1 drive and 100% for 2 goal). But it's just one data point (well if you trust me, 3 data points). I also perform periodic drive removals (I just turn one random chunkserver off for a week every week) to prove the system's safety.

[eleaner]

@njhurst which version of lizard are you running, if I may ask?

[njhurst]

12, after everyone's warning I checked out a git branch and have sat there ever since. I have also contemplated backporting from 13, but really with everything working smoothly it's probably not worth it.

[onlyjob]

@eleaner, I would have chosen MooseFS.
LizardFS is currently unmaintained and I do not feel comfortable recommending unmaintained software...

Just two servers are not enough for EC goals. If you consider server as a unit of failure then your data will remain available with 200% storage requirements for redundancy. That's the same requirement as for as RAID-1 or ec(2,2) with one chunkserver per server -- very simple and straightforward replicated setup.

Replicated chunks are best for performance.
With 6 chunkservers you could use ec(4,2) which needs 150% of storage but that will be much slower and somewhat more difficult to setup and it won't protect against failure of a server...