Lizardfs: 3.13.0~rc1: epic fail? Growing amount of missing chunks; "replication status: IO error"

[onlyjob]

After upgrading chunkservers to 3.13.0~rc1 I'm afraid I'm not getting away without massive data loss: mfsmaster logs replication status: IO error all the time and as replication progresses, CGI's _Chunks_ view report __growing__ (!) number of missing chunks in _ec_, and _xor_ goals.

Bloody hell... :( :( :(

[njhurst]

Oh no! :(:(:( Thanks for the warning and being the sacrifical one. I'll wait for now. I hope you have a backup.

[onlyjob]

Thank you for kind words, @njhurst. No, there is no backup. Where would you backup 100+ TiB? Systems like LizardFS are meant to protect from disasters, not cause them... Unforgivable...

Now I have 100_000+ missing chunks... I'm guessing that 3.13 destroyed chunks that did not finish goal change and had excessive chunks (mix of replicated and EC chunks).

Most of the damage occurred in the most precious data with goals _ec(2,2)_ and _ec(3,2)_ - those files should have been protected by 2 redundant chunks (RAID-6 level of safety).
All lost files were readable. No hardware failure was involved.

Replicated goals were not affected as far as I can tell... Maybe safe upgrade could be to change all EC goals to replicated ones, wait till no EC chunks are left and then upgrade...

[Blackpaw]

Any chance of a snapshot?

Sent from my Windows 10 phone

From: Dmitry Smirnov
Sent: Monday, 27 August 2018 8:34 PM
To: lizardfs/lizardfs
Cc: Subscribed
Subject: Re: [lizardfs/lizardfs] 3.13.0~rc1: epic fail? Growing amount ofmissing chunks; "replication status: IO error" (#746)

Thank you for kind words, @njhurst. No, there is no backup. Where would you backup 100+ TiB? Systems like LizardFS are meant to protect from disasters, not cause them... Unforgivable...
Now I have 100_000+ missing chunks... I'm guessing that 3.13 destroyed chunks that did not finish goal change and had excessive chunks (mix of replicated and EC chunks).
Most of the damage occurred in the most precious data with goals ec(2,2) and ec(3,2) - those files should have been protected by 2 redundant chunks (RAID-6 level of safety).
All lost files were readable. No hardware failure was involved.
Replicated goals were not affected as far as I can tell... Maybe safe upgrade could be to change all EC goals to replicated ones, wait till no EC chunks are left and then upgrade...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[onlyjob]

Yes I've managed to retrieve some files from snapshots though only some...

I've managed to recover some missing chunks from older (recently replaced) HDD by connecting it to chunkserver 3.12 (chunkserver 3.13 rapidly deletes valid _ec_ chunks).

[onlyjob]

Here is a quick summary of devastating upgrade from 3.12.0: tibibytes of data destroyed; 100_000+ missing chunks; 80_000+ files damaged. Almost all data in EC goals is gone either due to direct or collateral damage.

Pattern of damage is not enough parts available:

~~
chunk 0: 00000D9CA2DCB53A_00000001 / (id:14966398432570 ver:1)
copy 1: 192.168.0.130:9422:wks part 4/4 of ec(2,2)
not enough parts available
~~

~~
chunk 0: 00000D9CA2DCB6EA_00000001 / (id:14966398433002 ver:1)
copy 1: 192.168.0.250:9622:stor part 1/4 of ec(2,2)
not enough parts available
~~

Before upgrade I had fully replicated files with _ec(2,2)_ goals and most of them are gone despite having no undergoal files prior to upgrade.

I also have significant loss (at least 50%) in _ec(3,2)_ chunks of which some were fully replicated and some were in progress of changing goal from _std:3_ to _ec(3,2)_ so there were enough replicas to avoid data loss.

This is how damaged _ec(3,2)_ files look, according to lizardfs fileinfo:

~~
chunk 0: 0000000000954D73_00000001 / (id:9784691 ver:1)
copy 1: 192.168.0.204:9422:pool part 4/5 of ec(3,2)
copy 2: 192.168.0.250:9622:stor part 1/5 of ec(3,2)
not enough parts available
~~

~~
chunk 0: 0000000000954B9D_00000001 / (id:9784221 ver:1)
copy 1: 192.168.0.2:9422:pool part 4/5 of ec(3,2)
copy 2: 192.168.0.3:9622:pool part 3/5 of ec(3,2)
not enough parts available
chunk 1: 0000000000954BA5_00000001 / (id:9784229 ver:1)
copy 1: 192.168.0.2:9422:pool part 3/5 of ec(3,2)
copy 2: 192.168.0.204:9422:pool part 4/5 of ec(3,2)
not enough parts available
chunk 2: 0000000000954BAC_00000001 / (id:9784236 ver:1)
copy 1: 192.168.0.130:9422:wks part 2/5 of ec(3,2)
not enough parts available
chunk 3: 0000000000954BB8_00000001 / (id:9784248 ver:1)
copy 1: 192.168.0.2:9422:pool part 4/5 of ec(3,2)
copy 2: 192.168.0.3:9622:pool part 3/5 of ec(3,2)
copy 3: 192.168.0.4:9422:wks part 1/5 of ec(3,2)
copy 4: 192.168.0.204:9422:pool
copy 5: 192.168.0.250:9422:stor part 2/5 of ec(3,2)
copy 6: 192.168.0.250:9522:stor part 5/5 of ec(3,2)
copy 7: 192.168.0.250:9622:stor
chunk 4: 0000000000954BBC_00000001 / (id:9784252 ver:1)
copy 1: 192.168.0.2:9422:pool part 3/5 of ec(3,2)
copy 2: 192.168.0.3:9622:pool
copy 3: 192.168.0.4:9422:wks part 2/5 of ec(3,2)
copy 4: 192.168.0.204:9422:pool part 4/5 of ec(3,2)
copy 5: 192.168.0.250:9422:stor part 5/5 of ec(3,2)
copy 6: 192.168.0.250:9522:stor part 1/5 of ec(3,2)
copy 7: 192.168.0.250:9622:stor
chunk 5: 0000000000954BC4_00000002 / (id:9784260 ver:2)
copy 1: 192.168.0.2:9422:pool part 1/5 of ec(3,2)
copy 2: 192.168.0.3:9622:pool
copy 3: 192.168.0.130:9422:wks part 2/5 of ec(3,2)
copy 4: 192.168.0.204:9422:pool part 4/5 of ec(3,2)
copy 5: 192.168.0.250:9422:stor
copy 6: 192.168.0.250:9522:stor part 3/5 of ec(3,2)
copy 7: 192.168.0.250:9622:stor part 5/5 of ec(3,2)
~~

Snapshots were useless to recover data unless snapshots had different goals. In the aftermath I'll probably make an _std:1_ goal to use exclusively on snapshots an pin it to slow-ish chunkserver.

3.13.0~rc1 is very unsafe for EC chunks. It deletes valid copies causing massive replication of remaining data. Beware...

[eleaner]

Hi Guys. Now you scared me shitless.
I just started my adventure with LizzardFS and obviously with 3.13.0~rc1
200k chunks and I don't see any major problems yet, maybe except #765

What is the safest way forward? the whole point on using lizzard was to use ec instead of btrfs/zfs parity
is there a way to downgrade lizard version to a working one?

[onlyjob]

If you are already on v3.13.0~rc1 then you might be safe. Between 3.12 and 3.13 they've made a very unsafe change to convert EC chunks made by earlier LizardFS versions: e76c38623. Unless there are other issues affecting EC chunks, this particular one is about _upgrade_ to 3.13.0~rc1.

[eleaner]

@onlyjob
Yes. I started in v3.13.0~rc1
So I hope I am safe

[creolis]

We updated to 3.13.0rc1 in order to get proper bandwidth limit handling - but we did not notice this ticket prior to updating.

We also use EC goals (EC6,3 and EC7,2) with 660280 chunks (404057 fs objects).
Now 3 days into the update we start losing chunks in EC6,3 for no apparent reason,
we lost 5 chunks (3 files) so far.

Looking into the issue I stumbled upon your ticket and now I'm not sure how to proceed -
not sure if downgrading would be an option to consider since I don't know if the recalculation is still running and if we have to expect lost chunks adding up leaving it on 3.13.0rc1.

To be honest I would love to see at least some sign of life from skytech here ... at least a heartbeat showing that they acknowledge our findings and issues.

@onlyjob: how did you ultimately proceed?

[onlyjob]

@creolis, I think downgrade is the only option to save your data. It is especially important to avoid upgrading chunkservers (or downgrade them ASAP). I don't know if anything else could be done. From my memory, ~rc chunkservers were aggressively removing valid EC chunks.

I've lost terabytes of data due to this bug and ultimately moved away from LizardFS.
IMHO current governance of LizardFS can not be trusted and even if they'd care to repair the trust it would take a lot of time, expertise and communication with community.
Skytech is hopeless. It's been almost a year and they couldn't care less... :(

Knowing no better alternatives, I recommend to use MooseFS instead of LizardFS.

[creolis]

sigh I hate to accept this ... but data loss is the only thing I can't cope with and I have a hard time trusting a FS that allows this, even if we're talking about a RC.

For me the real issue is two things:
1) I wonder why 3.13.0-rc1 is still online and the preferred link if you hit "Download" on lizardfs.com, without the slightest note or warning that there is the chance to loose data on existing EC goals.

2) I really don't know why an absolute showstopper like this is not handled as a priority. If your users loose data, you get a reputation problem, even if you're working on an updated branch. Just dedicate enough time to 3.13.0-rc2 to prevent data loss. Management features that do not work? I'll survive that. Lost Performance? I can deal with that. Unstable chunkservers? I'll watchdog them with a shell script and restart them if necessary. Data Loss? I can't deal with this.

[onlyjob]

Regarding __1__ I'd be very reluctant to install software from source to production infrastructure, not to mention pre-release. To some extent Debian users are protected from this regression because I could not upload such broken release knowing severity of the problems. Official Debian package is not a panacea but it is better/safer because at least package maintainer double checked the release.

As for __2__, it seems there are nobody left to care. I think either all developers are left (or they were pulled away from the project). You can see from the history of commits that senior developers stopped committing a while ago, then there were no commits at all, then (after a while) a new (junior?) developer started to work on simple issues.
IMHO under current governance there is no hope for this project: #805.

Indeed data loss is the worst but there are other severe issues I've listed in the milestone: https://github.com/lizardfs/lizardfs/milestone/2. Notably #662 causes a lot of grief in CI because various git commands randomly fail. I'm not sure what else might be affected by #780 and #672 but it feels very insecure. #754 leaves even less confidence and #742/#743 show how much worse the quality of ~rc1 has become comparing to previous releases.

IMHO priorities of this project drifted too far away from quality (towards features?), causing so much damage to trust that I'd be surprised if it ever recovers... :(

[zicklag]

LizardFS is now under new management ( see https://github.com/lizardfs/lizardfs/issues/805#issuecomment-528899790 ) so hopefully LizardFS will start to get back on track again.