Openlibrary: Homepage often appears to use VERY outdated cache

[cdrini]

Carousels sometimes (at lease a few times a day) appear non-functional. On these cases, the sponsorship carousel also does not appear at all, and the graphs at the bottom of the page do not function. The JS/HTML reveals what looks like a caching issue; there's old JS in the HTML file that's getting called. This JS doesn't appear on the page at all when it loads correctly.

This error is thrown in the console:

ReferenceError: Carousel is not defined                                 openlibrary.org:394:3
    <anonymous> (index):394
    jQuery 4
        c
        fireWith
        ready
        L

The referenced line is:

<script type="text/javascript">
window.q.push(function() {
  Carousel.add('#categories_carousel', 6, 5, 4, 3, 2);
});
</script>

I'm also seeing similar code in a few places; e.g.

    <script type="text/javascript">
          window.q.push(function() {
            Carousel.add(
              '.carousel-public_domain', // dom element to bind slick
              6, 5, 4, 3, 2, 1, { // num books per responsive breakpoint
              });
          });
    </script>

Code like this was removed quite a while ago and shouldn't be here; that's what makes me think it's a cache issue.

Here's the full HTML file (in a zip): openlibrary_homepage.zip

Evidence / Screenshot (if possible)

Relevant url?

https://openlibrary.org/

Expectation

• carousels should have arrows and be interactible
• sponsorship carousel should appear

Details

• Logged in (Y/N)? Y
• Browser type/version? FF69
• Operating system? Win10

Proposal & Constraints

• This looks like a caching issue, where the page is _super_ out of date.

Stakeholders

@jdlrobson

[jdlrobson]

Agreed. Carousel. has not been supported for a long time. I think fixing this should be a top priority as those JS errors will be breaking all features of the site built using JS.

[cdrini]

Although I think I'll put this as P1; P0 implies "stay up late to fix this" :P

[cdrini]

Apparently this is also causing the authors page to display blank sometimes (cc @BrittanyBunk)

[tfmorris]

I see author pages display zero works and works pages display zero editions on a relatively regular basis, but always assumed that those were due to search failures.

[BrittanyBunk]

@cdrini yes, that's what I was talking about yesterday.

[cdrini]

I investigated the memcaches themselves with the following code (based off https://github.com/internetarchive/openlibrary/wiki/Getting-Started#memcache ):

import memcache
def check_cache(key, query):
  for i in range(3,6):
    mc_url = 'ol-mem%d:11211' % i
    value = memcache.Client([mc_url]).get(key)
    print(mc_url + ' ' + (str(query in value) if value else 'None'))

check_cache('home.homepage-', 'Carousel.add')
check_cache('home.homepage-', 'sponsorship')

Got results like:

ol-mem3:11211 False
ol-mem4:11211 None
ol-mem5:11211 False

ol-mem3:11211 True
ol-mem4:11211 None
ol-mem5:11211 True

Got matches on Carousel.add once; deleted the entry on the corresponding client and it seemed to fix it. Got sponsorship _not_ matching once; deleting it correctly showed None for a bit, but then _it came back still incorrect_. :/ Deleted it ~3 times; that seemed to fix it.

I'm not convinced these aren't being generated outdated though. Also none of this would address author pages that have been cached. So the root cause of this is still unclear. We might have to fully clear the caches one at a time...? But that still doesn't explain how we got here in the first place.

[cdrini]

They appeared outdated again :(

>>> check_cache('home.homepage-', 'Carousel.add')
ol-mem3:11211 True
ol-mem4:11211 None
ol-mem5:11211 False
>>> check_cache('home.homepage-', 'sponsorship')
ol-mem3:11211 False
ol-mem4:11211 None
ol-mem5:11211 True

[cdrini]

So it looks like they're somehow getting inserted into memcache outdated. So perhaps there is outdated code running hooked up to the production memcaches?

It also appears the ol-mem3 is the one that's been wrong (based on short observation). And ol-mem4 appears to always be None for these two fields.

Tried checking the status of ol-mem3 (with ssh ol-mem3 sudo /etc/init.d/memcached status) and noticed a few warnings; the other memcaches did had no warnings. These might be related?

Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 03:26:31 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 17:46:33 systemd-memcached-wrapper[4526]: accept4(): Too many open files
Nov 14 17:46:33 systemd-memcached-wrapper[4526]: accept4(): Too many open files

[cdrini]

KUBERNETES

[cdrini]

Maybe it's those k8s instances? I think they were hooked to prod memcache. Gitlab says they're still running, but its deleted the logs, so I can't find their urls to check if their still there. I'll get confirmation they're unused and then knock them down. 🎳

[tfmorris]

This is probably obvious, but clearly no dev or QA machines should have write access to any part of the production infrastructure (including caches).

[cdrini]

Ok, pods stopped (thanks Tracey!); the caching issue should be fixed now. (I'll monitor and re-open if necessary).

@tfmorris Agree; that is the way dev.openlibrary.org currently works, though, unfortunately :/