Lizardfs: fragile master data; master can't use metadata after OS upgrade.

[onlyjob]

We seems to have a very serious problem. My cluster is running master from jessie-backports and so is shadow. Recently I've upgraded shadow to soon-to-be-released "Stretch" (Debian v9) and immediately shadow failed with endless loop re-fetching data from master over and over again.
I stopped it as I thought may be it might work if I stop master, transfer metadata to shadow and re-start shadow as master but that failed too with "inconsistent" errors! This is bad as whoever decides to upgrade will end up losing master and therefore compromising access to all data in cluster.
Strange thing is that _jessie-backports_ follows _testing_ suite so LizardFS was re-built in backports without any modifications (only different library versions). The only thing that changed in Lizardfs packaging in _testing_ is re-build with newer googletest.
Both master and shadow are on the same hardware architecture (_x86-64_ a.k.a. _amd64_) and it seems that currently metadata is not compatible even with the same version of LizardFS built for the same architecture... :( This is bad... Would I be able to safely upgrade OS on my LizardFS master?

[Zorlin]

Is it common to upgrade between various versions of distros? I don't trust distro-upgrades since going from Ubuntu 6.06 to Ubuntu 6.10 completely broke sudo...

In most environments I would never upgrade, always clean install.

Does a cleanly installed Stretch work as a shadow with the original master?

[onlyjob]

Debian distr-upgrade is a supported feature for over 15 years and it works very reliable and usually (always?) tested thoroughly. I routinely dist-upgrade my servers and some of them survived several releases as well as hardware upgrades (sometimes until none of the original hardware is left)...
The concern here is how on earth LizardFS manages to produce incompatible/unusable data that even clean install is not able to use?? Metadata should never be sensitive to any versions of dependency chain.
Clean install does not work as it can't use metadata generated in "stable". :(

[onlyjob]

Ubuntu is an invasive derivative as it seems to break/modify too much of what we rarely see broken in Debian...

[Zorlin]

Okay, +1 this as a critical issue.

[psarna]

We'll try to reproduce it, but we definitely need more context:

1. Logs from both master and shadow server - especially the first one in shadow that caused the "inconsistent" error would help.
2. I need to know what "transfer metadata to shadow" means. Is it the whole /var/lib/lizardfs directory? Is it only the metadata.mfs file? If it is the former, can you try cleaning /var/lib/lizardfs on shadow first and then using only metadata.mfs file from master?
3. Can you run mfsmetadump executable on affected OS with metadata.mfs file extracted from valid master? Does it produce errors? What kind of errors?
4. Additionally, cleaning /var/lib/lizardfs directory of your shadow, restarting it and providing all the logs it produced since its start would also help. By logs I mean all the syslog stuff AND any leftover changelog.mfs.* files produced in shadow's /var/lib/lizardfs directory.

[psarna]

UPDATE:

This already merged patch https://github.com/lizardfs/lizardfs/commit/e77895af3be6d09f4fb115147b06417dcc1f0f3d checks for a very annoying bug in Judy library and disables Judy during compilation if it is discovered. and it was introduced in 3.10.6. I can see that your packages are 3.10.4, so I assume the patch is not here.

Please try a) using 3.10.6 on Debian 9 and then trying to start shadow or b) applying this patch on your 3.10.4, rebuilding and checking again. This bug in Judy used to mess with loading/storing ids in metadata, which might just cause your "inconsistency" problems.

Yes, we do our best to be compatible through new OS releases, but we are not clairvoyant - things need to be tested first (like with this Judy bug). Thanks a lot for pointing out the problems with LizardFS on Debian 9, I hope https://github.com/lizardfs/lizardfs/commit/e77895af3be6d09f4fb115147b06417dcc1f0f3d solves the issue.

[onlyjob]

Thanks but unfortunately _libjudy_ is not the culprit and re-building with e77895a did not help...
We need to fix the problem on 3.10.4 as at the current stage of freeze new releases will not be accepted to _Stretch_.

When I start shadow master from scratch (with empty metadata dir) it throws endless some changes lost: [...], download metadata again and Master temporarily refused to produce a new metadata image.

I'll submit logs with "inconsistent" errors once I get to the machine and I'll try mfsmetadump as well.

... what "transfer metadata to shadow" means. Is it the whole /var/lib/lizardfs directory?

Yes.

[psarna]

I tested 3.10.4 from Debian repo and our newest master from Github repo and 3.10.4 indeed produces "Master temporarily refused to produce a new metadata image". Our master branch works fine, so what is left to be done is find which patch fixes the issue.

[psarna]

This issue occurs on GCC >= 6.2.0 with certain optimizations enabled, but not on earlier version of the compiler. Debian 9's default GCC version is 6.3.0 and that's why it has the symptoms.

This patch should be enough to solve the issue (at least it did in our testing environment): http://cr.skytechnology.pl:8081/#/c/2870/. It can be cherry-picked to 3.10.4 without conflicts. Please test it and let us know the result.

Meanwhile, we will research why GCC compiled the code like that and how it relates to the standard.

EDIT: Merged version from github: https://github.com/lizardfs/lizardfs/commit/422175eb1aa8c2a1e4d0727ee3fe8190e1ae1340

[onlyjob]

422175e fixed the issue -- thank you very much!