Lizardfs: 3.12.0: data corruption in apt-cacher-ng

Created on 10 Mar 2018 · 10Comments · Source: lizardfs/lizardfs

Ever since upgrade to 3.12.0 my apt-cacher-ng is malfunctioning: installing or upgrading packages is no longer possible because downloaded (.deb) files fail integrity check. Changing cache location from LizardFS mount to local file system fixed the problem. 3.11.3 (and all previous releases back to 2.6) never had this problem.

I don't have synthetic reproducer for this problem yet and I am very worried about integrity of the data. I've tested integrity of some old files and found no problems so corruption must be happening on write.

Please investigate this as a highest priority.

bug

Source

onlyjob

Most helpful comment

Hi!
Good news, we finally replicated the issue with appropriate apt-cacher-ng settings! As we thought, this is problem with sendfile() when the file is read and written at the same spot at the same time. The simplest workaround for now is to add option
-o cacheexpirationtime=0
in LizardFS mount.
This basically reverts the cache to 3.11 settings.
We will address this case and make the next version resilient to such a situation.

Another good news is that it should not trigger in most situations - sendfile() is not POSIX compliant and sports this interesting feature "Note that a successful call to sendfile() may write fewer bytes than requested;" (it is possible that this is what causes the issue - sendfile() must be called again, but the process that reads data works in a different context and reads from cache, not the real file), and is generally not used in portable software. But it is faster indeed than read and write.

So, thank you very much again for finding this interesting case, we will add a patch and a test to prevent this.

For now please let us know if the workaround works for you.

pkali on 29 Mar 2018

🎉2

All 10 comments

Hello Dmitry,

Thank you very much for your message and concern.

We have been looking at the issue for about a week now and are not able to replicate the issue, but we have some insight on possible reasons.

First of all apt-cacher-ng is as we learnt is a quite sensitive software, we reckon it is due to, among others, using sendfile(). There are several error reports on Debian tracker referring to .deb data corruption in this app.
What we think has happened is that some bug or misconfiguration in apt-cacher-ng is being triggered by our new and intelligent caches. Unfortunately on our soft/hardware we could not spot the broken package, therefore I would like to ask you for details on your OS on machine with apt-cacher-ng, where it was writing data, what were goal / label settings for the space and other non default settings of your cluster.

We will replicate your set-up as closely as possible and then find the commit that causes the behaviour and then change/fix it or propose relevant configuration settings.

In summary - we currently think it is not strictly a LizardFS bug, but clash of apt-cache-ng and LizardFS caches that is possibly solvable by changing settings in one or the other software, but will be sure after replicating it.

Thanks,

Pawel Kalinowski

pkali on 23 Mar 2018

Thanks for looking into the problem.

First of all apt-cacher-ng is as we learnt is a quite sensitive software

I've never noticed its sensitivity in years up until LizardFS 3.12.0...

There are several error reports on Debian tracker referring to .deb data corruption in this app.

Could you be more specific please? What bug reports?

Anyway I doubt they are relevant or applicable as problem can be easily isolated by changing cache location between LizardFS mount and local file system (ext4).

I'm using Debian v9 "Stretch" on amd64 in a small network with up to dozen PCs.
All PCs have APT proxy pointed to one box where "/var/cache/apt-cacher-ng" is on LizardFS mount. Goal is set as $std {stor pool cacher} so it is a simple replicated goal with three replicas. Hardware is healthy and no errors were reported. I've completely removed cache multiple times and problem happens every time rendering apt-cacher-ng on LizardFS useless...

onlyjob on 24 Mar 2018

Thank you!

With regards to apt-cacher-ng corruptions I refered to bugs like
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=677983
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=644959
https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1259191
https://bugs.launchpad.net/ubuntu/+source/apt-cacher-ng/+bug/1162876
https://askubuntu.com/questions/119298/apt-get-using-apt-cacher-ng-fails-to-fetch-packages-with-hash-sum-mismatch
https://ubuntuforums.org/showthread.php?t=1491239

These all are old reports and these issues should be long gone, but we suspect clash between cashes as I mentioned. This report seems to be leading us in a right direction: https://github.com/moby/moby/issues/9592
Not directly related, but we hope to get it in days.

pkali on 26 Mar 2018

Those _very old_ bugs are most certainly irrelevant.

As far as I'm concerned this is a serious bug as no access pattern should be able to trigger corruption. Clearly something happened in 3.12 that caused this...

onlyjob on 26 Mar 2018

Hi Dmitry,
We do try to replicate this issue, to no avail. Tried 3.12, few versions of 3.11 and the latest sources on Debian 9 and Ubuntu 16.4 with different caching settings of LizardFS.

We have added test apt-cacher-ng here: http://cr.skytechnology.pl:8081/#/c/3269/
As we were not using apt-cacher-ng got the fresh version from official repos. For the same reason if you see how this test could be made closer to your usage, please let us know!

At this point we have nothing to hold on. Cache analysis shown that it is very remotely probable that when .deb is being saved and read at the same time there could be bytes read out of the currently saved stream, but when saving ends, the file would be perfectly fine anyways, no data corruption at all.

Our idea for the next steps when we have this issue replicated would be to start with the last release version behaving normally and then apply one commit after another till the issue appears. This way we would know what changes in code cause the issue and then we could devise a solution. (Un)fortunately so far no corrupted files whatsoever.

Do you think you could try such a procedure on your system? Narrowing it to the single commit would definitely help understand what is going wrong in your setup.

pkali on 28 Mar 2018

I'm afraid I have no capacity to isolate (bisect) the problem... I've inspected freshly downloaded .deb files in apt-cacher-ng/cache and found garbage in some of them...
Server where it is happening have ECC RAM and no hardware error were reported...

onlyjob on 29 Mar 2018

So, thank you very much again for finding this interesting case, we will add a patch and a test to prevent this.

For now please let us know if the workaround works for you.

pkali on 29 Mar 2018

🎉2

Excellent news, @pkali, thank you very much for diligent troubleshooting. I will apply workaround and confirm shortly.

onlyjob on 30 Mar 2018

Workaround works perfectly - thank you. I have had no further incidents ever since I introduced -o cacheexpirationtime=0. Patch would be much appreciated.

onlyjob on 22 May 2018

👍1

Thank you, Pawel, for the summary of acng issues. It helped us to resolve similar problem with corrupted files (despite we do not use LizardFS). We haven't found any proofs but our impression about acng is that they prioritize serving files and may not complete caching. There is a maintenance cron job (in Ubuntu 16.04 daily) "acngtool maint" that fixes incomplete/corrupted files in cache. Therefore, we execute /etc/cron.daily/apt-cacher-ng for data integrity.