Openlibrary: Related books carousel breaking on some book pages with AttributeError

[mekarpeles]

e.g. https://openlibrary.org/works/OL17829905W/Seveneves

Related books renders with "Unable to render the page"

Visiting https://openlibrary.org/works/OL17829905W/Seveneves?debug=true the following error is provided:
at /works/OL17829905W

[jdlrobson]

Is this a php error or js error? Can you print any related stack traces?
I think this is php but am not 100% sure.

[mekarpeles]

This is a python + memcached problem

[jdlrobson]

Is this still a problem now we upgraded to slick?

[mekarpeles]

Still an issue: see same example provided with ?debug=true

[tfmorris]

Why was this tagged Python 3? It seems to be fixed now, whatever the issue was.

@mekarpeles Are you OK with closing?

[jdlrobson]

Closing.. :)

[cdrini]

Hmmm, I'm still getting this issue on the provided link. @jdlrobson @tfmorris Can you provide screenshots of what the carousels on that page look like for you guys?

[jdlrobson]

I'm seeing the problem again now!

[jdlrobson]

[jdlrobson]

The issue appears to be there are no books in the carousel. The carousel should not try to render nothing.

[jdlrobson]

Investigating this more this seems to be something to do with memcached/caching
The problematic line is:

books = generic_carousel(query=query, subject=subject, work_id=work_id, _type=_type, sorts=sorts, limit=limit) if not test else []

in openlibrary/templates/home/custom_ia_carousel.html which calls out to the generic_carousel function in openlibrary/plugins/openlibrary/home.py
According to https://openlibrary.org/books/OL26418025M/Seveneves?debug=true

This is out of my expertise area and I can't seem to replicate the problem. Who would be a better person to assign this to?

I can't seem to replicate this locally by importing the page.

[brad2014]

@mek Can you opine on the correct owner and priority of this?

[tfmorris]

I can confirm that the problem still exists. Allthough it's targeting the memcache, the problem is actually in the JSON encoding code (probably being triggered by bad data that's not properly escaped).

Since the error is occurring trying to set a memcache entry, we should probably just catch the error and not cache anything as a fallback.

Because this is querying the IA API (bad idea!) for the related books and it depends on availability (also a bad idea), the problem is not deterministic and may go away when the problematic book isn't returned in the results.
https://archive.org/search.php?query=openlibrary_work:(*)%20AND%20collection:(inlibrary)%20AND%20loans__status__status:AVAILABLE%20AND%20(subject:%22Survival%22%20OR%20subject:%22Exploration%22%20OR%20subject:%22Disasters%22)%20AND%20!openlibrary_work:(OL17829905W)

I don't have time to debug it right now, but the offending value that can't be JSON encoded is below. Note it's Python encoded and contains Python objects, so isn't directly usable to feed to a JSON encoder, but I wanted to make sure we have a record of the problematic value(s).

[<Storage {'title': '100 most dangerous things on the planet', 'url': '/books/OL24950597M/100_most_dangerous_things_on_the_planet', 'cover_url': '//covers.openlibrary.org/w/id/6941465-M.jpg', 'key': '/books/OL24950597M', 'authors': [<Storage {'name': 'Anna Claybourne', 'key': u'/authors/OL1392098A'}>], 'ocaid': '100mostdangerous00clay', 'borrow_url': '/books/OL24950597M/100_most_dangerous_things_on_the_planet/borrow'}>, <Storage {'title': 'Nature gone wild!', 'url': '/books/OL3509104M/Nature_gone_wild!', 'cover_url': '//covers.openlibrary.org/w/id/8233535-M.jpg', 'key': '/books/OL3509104M', 'authors': [<Storage {'name': 'Walter G. Oleksy', 'key': u'/authors/OL20210A'}>], 'ocaid': 'naturegonewild00olek', 'borrow_url': '/books/OL3509104M/Nature_gone_wild!/borrow'}>, <Storage {'title': 'FAMOUS DISASTERS (National Enquirer)', 'url': '/books/OL7649521M/FAMOUS_DISASTERS_(National_Enquirer)', 'cover_url': '//covers.openlibrary.org/w/id/8268669-M.jpg', 'key': '/books/OL7649521M', 'authors': [<Storage {'name': 'J. Wagenvoord', 'key': u'/authors/OL2749004A'}>], 'ocaid': 'famousdisasters0000unse', 'borrow_url': '/books/OL7649521M/FAMOUS_DISASTERS_(National_Enquirer)/borrow'}>, <Storage {'title': 'Surviving natural disasters', 'url': '/books/OL26453119M/Surviving_natural_disasters', 'cover_url': '//covers.openlibrary.org/w/id/8179038-M.jpg', 'key': '/books/OL26453119M', 'authors': [<Storage {'name': 'Jen Green', 'key': u'/authors/OL38756A'}>], 'ocaid': 'survivingnatural0000gree', 'borrow_url': '/books/OL26453119M/Surviving_natural_disasters/borrow'}>, <Storage {'title': 'The handbook of the weather', 'url': '/books/OL26497711M/The_handbook_of_the_weather', 'cover_url': '//covers.openlibrary.org/w/id/8251529-M.jpg', 'key': '/books/OL26497711M', 'authors': [<Storage {'name': 'Gerrie McCall', 'key': u'/authors/OL1436105A'}>], 'ocaid': 'handbookofweathe0000mcca', 'borrow_url': '/books/OL26497711M/The_handbook_of_the_weather/borrow'}>, <Storage {'title': 'Catastrophe', 'url': '/books/OL21486239M/Catastrophe', 'cover_url': '//covers.openlibrary.org/w/id/4651806-M.jpg', 'key': '/books/OL21486239M', 'authors': [<Storage {'name': 'A. C. Waltham', 'key': u'/authors/OL6131805A'}>], 'ocaid': 'catastropheviole0000walt', 'borrow_url': '/books/OL21486239M/Catastrophe/borrow'}>, <Storage {'title': 'Portent', 'url': '/books/OL7287033M/Portent', 'cover_url': '//covers.openlibrary.org/w/id/1069423-M.jpg', 'key': '/books/OL7287033M', 'authors': [<Storage {'name': 'James Herbert', 'key': u'/authors/OL27138A'}>], 'ocaid': 'portent00jame', 'borrow_url': '/books/OL7287033M/Portent/borrow'}>, <Storage {'title': 'Survival', 'url': '/books/OL22569669M/Survival', 'cover_url': '//covers.openlibrary.org/w/id/8284103-M.jpg', 'key': '/books/OL22569669M', 'authors': [<Storage {'name': 'Russel L. Honore', 'key': u'/authors/OL6480981A'}>], 'ocaid': 'survivalhowbeing0000hono', 'borrow_url': '/books/OL22569669M/Survival/borrow'}>, <Storage {'title': 'Major disasters.', 'url': '/books/OL1001927M/Major_disasters.', 'cover_url': '//covers.openlibrary.org/w/id/3858573-M.jpg', 'key': '/books/OL1001927M', 'authors': [], 'ocaid': 'majordisastersit00staf', 'borrow_url': '/books/OL1001927M/Major_disasters./borrow'}>, <Storage {'title': 'Disaster: major American catastrophes', 'url': '/books/OL5211173M/Disaster_major_American_catastrophes', 'cover_url': '//covers.openlibrary.org/w/id/7159791-M.jpg', 'key': '/books/OL5211173M', 'authors': [<Storage {'name': 'A. A. Hoehling', 'key': u'/authors/OL381205A'}>], 'ocaid': 'disastermajorame00hoeh', 'borrow_url': '/books/OL5211173M/Disaster_major_American_catastrophes/borrow'}>, <Storage {'title': 'Calamities of the world', 'url': '/books/OL5088504M/Calamities_of_the_world', 'cover_url': '//covers.openlibrary.org/w/id/7133559-M.jpg', 'key': '/books/OL5088504M', 'authors': [<Storage {'name': 'Hubert S. Banner', 'key': u'/authors/OL1971416A'}>], 'ocaid': 'calamitiesofworl00hube', 'borrow_url': '/books/OL5088504M/Calamities_of_the_world/borrow'}>, <Storage {'title': '7th sigma', 'url': '/books/OL24839168M/7th_sigma', 'cover_url': '//covers.openlibrary.org/w/id/7922123-M.jpg', 'key': '/books/OL24839168M', 'authors': [<Storage {'name': 'Steven Gould', 'key': u'/authors/OL34414A'}>], 'ocaid': '7thsigma00goul', 'borrow_url': '/books/OL24839168M/7th_sigma/borrow'}>, <Storage {'title': 'World of disasters', 'url': '/books/OL3299834M/World_of_disasters', 'cover_url': '//covers.openlibrary.org/w/id/1408338-M.jpg', 'key': '/books/OL3299834M', 'authors': [<Storage {'name': 'Ned Halley', 'key': u'/authors/OL451430A'}>], 'ocaid': 'worldofdisasters0000hall', 'borrow_url': '/books/OL3299834M/World_of_disasters/borrow'}>, <Storage {'title': "Jungle survivor's guide", 'url': "/books/OL7512000M/Jungle_survivor's_guide", 'cover_url': '//covers.openlibrary.org/w/id/4628589-M.jpg', 'key': '/books/OL7512000M', 'authors': [<Storage {'name': 'Rory Storm', 'key': u'/authors/OL1608930A'}>], 'ocaid': 'junglesurvivorsg0000stor', 'borrow_url': "/books/OL7512000M/Jungle_survivor's_guide/borrow"}>, <Storage {'title': '100 disasters that shook the world', 'url': '/books/OL26586607M/100_disasters_that_shook_the_world', 'cover_url': '//covers.openlibrary.org/w/id/8280666-M.jpg', 'key': '/books/OL26586607M', 'authors': [<Storage {'name': 'Nigel Cawthorne', 'key': u'/authors/OL4861216A'}>], 'ocaid': '100disastersthat0000cawt', 'borrow_url': '/books/OL26586607M/100_disasters_that_shook_the_world/borrow'}>, <Storage {'title': 'Frozen fire', 'url': '/books/OL24746594M/Frozen_fire', 'cover_url': '//covers.openlibrary.org/w/id/6800267-M.jpg', 'key': '/books/OL24746594M', 'authors': [<Storage {'name': 'James Houston', 'key': u'/authors/OL4281262A'}>], 'ocaid': 'frozenfire00hous', 'borrow_url': '/books/OL24746594M/Frozen_fire/borrow'}>, <Storage {'title': 'The nightworld', 'url': '/books/OL26378602M/The_nightworld', 'cover_url': '//covers.openlibrary.org/w/id/8345933-M.jpg', 'key': '/books/OL26378602M', 'authors': [<Storage {'name': 'Jack Blaine', 'key': u'/authors/OL7399756A'}>], 'ocaid': 'isbn_9780062133809', 'borrow_url': '/books/OL26378602M/The_nightworld/borrow'}>, <Storage {'title': 'Natural disasters', 'url': '/books/OL22463778M/Natural_disasters', 'cover_url': '//covers.openlibrary.org/w/id/8335633-M.jpg', 'key': '/books/OL22463778M', 'authors': [<Storage {'name': 'Patrick L. Abbott', 'key': u'/authors/OL27821A'}>], 'ocaid': 'naturaldisasters00patr', 'borrow_url': '/books/OL22463778M/Natural_disasters/borrow'}>, <Storage {'title': 'The sixth seal', 'url': '/books/OL1416249M/The_sixth_seal', 'cover_url': '//covers.openlibrary.org/w/id/3938822-M.jpg', 'key': '/books/OL1416249M', 'authors': [<Storage {'name': 'Mary Wesley', 'key': u'/authors/OL259696A'}>], 'ocaid': 'sixthseal00wesl', 'borrow_url': '/books/OL1416249M/The_sixth_seal/borrow'}>, <Storage {'title': 'How to survive the end of the world as we know it', 'url': '/books/OL24061777M/How_to_survive_the_end_of_the_world_as_we_know_it', 'cover_url': '//covers.openlibrary.org/w/id/8065004-M.jpg', 'key': '/books/OL24061777M', 'authors': [<Storage {'name': 'James Wesley Rawles', 'key': u'/authors/OL245398A'}>], 'ocaid': 'howtosurviveendo00rawl', 'borrow_url': '/books/OL24061777M/How_to_survive_the_end_of_the_world_as_we_know_it/borrow'}>, <Storage {'title': 'Great Disasters of the 20th Century', 'url': '/books/OL11468279M/Great_Disasters_of_the_20th_Century', 'cover_url': '//covers.openlibrary.org/w/id/8327281-M.jpg', 'key': '/books/OL11468279M', 'authors': [<Storage {'name': 'Margot Keam Cleary', 'key': u'/authors/OL3639401A'}>], 'ocaid': 'greatdisastersof0000clea', 'borrow_url': '/books/OL11468279M/Great_Disasters_of_the_20th_Century/borrow'}>, <Storage {'title': 'Snowstruck', 'url': '/books/OL24219991M/Snowstruck', 'cover_url': '//covers.openlibrary.org/w/id/6563500-M.jpg', 'key': '/books/OL24219991M', 'authors': [<Storage {'name': 'Jill A. Fredston', 'key': u'/authors/OL3000724A'}>], 'ocaid': 'snowstruckingrip00fred', 'borrow_url': '/books/OL24219991M/Snowstruck/borrow'}>, <Storage {'title': 'Catastrophe!', 'url': '/books/OL4460956M/Catastrophe!', 'cover_url': 'https://archive.org/services/img/catastrophewhenm00newy', 'key': '/books/OL4460956M', 'authors': [<Storage {'name': u'Encyclop\xe6dia Britannica, Inc.', 'key': u'/authors/OL6850721A'}>], 'ocaid': 'catastrophewhenm00newy', 'borrow_url': '/books/OL4460956M/Catastrophe!/borrow'}>, <Storage {'title': "World's worst historical disasters", 'url': "/books/OL16856277M/World's_worst_historical_disasters", 'cover_url': '//covers.openlibrary.org/w/id/8265843-M.jpg', 'key': '/books/OL16856277M', 'authors': [<Storage {'name': 'Chris McNab', 'key': u'/authors/OL5139270A'}>], 'ocaid': 'worldsworsthisto0000mcna', 'borrow_url': "/books/OL16856277M/World's_worst_historical_disasters/borrow"}>, <Storage {'title': "World's Greatest Disasters", 'url': "/books/OL8572733M/World's_Greatest_Disasters", 'cover_url': 'https://archive.org/services/img/worldsgreatestdi0000robi', 'key': '/books/OL8572733M', 'authors': [<Storage {'name': 'Joyce Robins', 'key': u'/authors/OL2865204A'}>], 'ocaid': 'worldsgreatestdi0000robi', 'borrow_url': "/books/OL8572733M/World's_Greatest_Disasters/borrow"}>, <Storage {'title': 'Disasters', 'url': '/books/OL24856338M/Disasters', 'cover_url': '//covers.openlibrary.org/w/id/6833499-M.jpg', 'key': '/books/OL24856338M', 'authors': [<Storage {'name': 'Ned Halley', 'key': u'/authors/OL451430A'}>], 'ocaid': 'disasters00hall', 'borrow_url': '/books/OL24856338M/Disasters/borrow'}>, <Storage {'title': 'Hellbenders', 'url': '/books/OL7418406M/Hellbenders', 'cover_url': '//covers.openlibrary.org/w/id/219469-M.jpg', 'key': '/books/OL7418406M', 'authors': [<Storage {'name': 'James Axler', 'key': u'/authors/OL1385953A'}>], 'ocaid': 'hellbenders00jame', 'borrow_url': '/books/OL7418406M/Hellbenders/borrow'}>, <Storage {'title': 'Exploring Natural Disasters', 'url': '/books/OL8636551M/Exploring_Natural_Disasters', 'cover_url': '//covers.openlibrary.org/b/id/6497359-M.jpg', 'key': '/books/OL8636551M', 'authors': [<Storage {'name': 'Stella Sands', 'key': u'/authors/OL2865597A'}>], 'ocaid': 'exploringnatural00sand', 'borrow_url': '/books/OL8636551M/Exploring_Natural_Disasters/borrow'}>, <Storage {'title': 'Icequake!', 'url': '/books/OL26424941M/Icequake!', 'cover_url': '//covers.openlibrary.org/w/id/8133060-M.jpg', 'key': '/books/OL26424941M', 'authors': [<Storage {'name': 'Spencer, John R.', 'key': u'/authors/OL560523A'}>], 'ocaid': 'icequakechronicl0000spen', 'borrow_url': '/books/OL26424941M/Icequake!/borrow'}>, <Storage {'title': 'Rage', 'url': '/books/OL25110368M/Rage', 'cover_url': '//covers.openlibrary.org/w/id/7026058-M.jpg', 'key': '/books/OL25110368M', 'authors': [<Storage {'name': 'Matthew J. Costello', 'key': u'/authors/OL236071A'}>], 'ocaid': 'isbn_9780345529367', 'borrow_url': '/books/OL25110368M/Rage/borrow'}>, <Storage {'url': , 'ocaid': None, 'authors': [], 'key': '/books/OL24770917M', 'title': None}>, <Storage {'title': 'Lost Valley', 'url': '/books/OL9897552M/Lost_Valley', 'cover_url': '//covers.openlibrary.org/w/id/6807072-M.jpg', 'key': '/books/OL9897552M', 'authors': [<Storage {'name': 'Dudley Bromley', 'key': u'/authors/OL1659445A'}>], 'ocaid': 'lostvalley00brom', 'borrow_url': '/books/OL9897552M/Lost_Valley/borrow'}>, <Storage {'title': 'Natural Disasters', 'url': '/books/OL7658536M/Natural_Disasters', 'cover_url': '//covers.openlibrary.org/w/id/1263619-M.jpg', 'key': '/books/OL7658536M', 'authors': [<Storage {'name': 'Madeline Boskey', 'key': u'/authors/OL2703800A'}>], 'ocaid': 'naturaldisasters0000bosk', 'borrow_url': '/books/OL7658536M/Natural_Disasters/borrow'}>, <Storage {'title': 'Eyewitness to disaster', 'url': '/books/OL4886328M/Eyewitness_to_disaster', 'cover_url': '//covers.openlibrary.org/w/id/6703770-M.jpg', 'key': '/books/OL4886328M', 'authors': [<Storage {'name': 'Dan Perkes', 'key': u'/authors/OL1295278A'}>], 'ocaid': 'eyewitnesstodisa00perk', 'borrow_url': '/books/OL4886328M/Eyewitness_to_disaster/borrow'}>, <Storage {'title': 'The Associated Press Library of Disasters', 'url': '/books/OL10528342M/The_Associated_Press_Library_of_Disasters', 'cover_url': '//covers.openlibrary.org/w/id/8318443-M.jpg', 'key': '/books/OL10528342M', 'authors': [<Storage {'name': 'Grolier Educational', 'key': u'/authors/OL2789599A'}>], 'ocaid': 'associatedpressl0000asso', 'borrow_url': '/books/OL10528342M/The_Associated_Press_Library_of_Disasters/borrow'}>, <Storage {'title': 'The great disasters', 'url': '/books/OL4874146M/The_great_disasters', 'cover_url': 'https://archive.org/services/img/greatdisasters00malo', 'key': '/books/OL4874146M', 'authors': [<Storage {'name': 'William Edward Maloney', 'key': u'/authors/OL1661272A'}>], 'ocaid': 'greatdisasters00malo', 'borrow_url': '/books/OL4874146M/The_great_disasters/borrow'}>, <Storage {'title': 'Disaster!', 'url': '/books/OL7511587M/Disaster!', 'cover_url': '//covers.openlibrary.org/w/id/7908452-M.jpg', 'key': '/books/OL7511587M', 'authors': [<Storage {'name': 'Ellen Leroe', 'key': u'/authors/OL455017A'}>], 'ocaid': 'disasterthreerea00elle', 'borrow_url': '/books/OL7511587M/Disaster!/borrow'}>, <Storage {'title': 'Survive! Could you?', 'url': '/books/OL1405437M/Survive!_Could_you', 'cover_url': '//covers.openlibrary.org/w/id/8341113-M.jpg', 'key': '/books/OL1405437M', 'authors': [<Storage {'name': 'Judy Donnelly', 'key': u'/authors/OL612797A'}>], 'ocaid': 'survivecouldyouh00judy', 'borrow_url': '/books/OL1405437M/Survive!_Could_you/borrow'}>, <Storage {'title': 'Disaster!', 'url': '/books/OL26435027M/Disaster!', 'cover_url': '//covers.openlibrary.org/w/id/8148000-M.jpg', 'key': '/books/OL26435027M', 'authors': [<Storage {'name': u'Ren\xe9 Schmidt', 'key': u'/authors/OL7409962A'}>], 'ocaid': 'disaster0000schm', 'borrow_url': '/books/OL26435027M/Disaster!/borrow'}>, <Storage {'title': 'About disasters', 'url': '/books/OL1578674M/About_disasters', 'cover_url': '//covers.openlibrary.org/w/id/8149225-M.jpg', 'key': '/books/OL1578674M', 'authors': [<Storage {'name': 'Joy Wilt Berry', 'key': u'/authors/OL329907A'}>], 'ocaid': 'aboutdisasters0000berr', 'borrow_url': '/books/OL1578674M/About_disasters/borrow'}>, <Storage {'title': 'Horror in paradise', 'url': '/books/OL24209814M/Horror_in_paradise', 'cover_url': '//covers.openlibrary.org/b/id/6512705-M.jpg', 'key': '/books/OL24209814M', 'authors': [<Storage {'name': 'James Cornell', 'key': u'/authors/OL1594712A'}>], 'ocaid': 'horrorinparadise00corn', 'borrow_url': '/books/OL24209814M/Horror_in_paradise/borrow'}>, <Storage {'title': 'Preppers Home Defense Security Strategies To Protect Your Family By Any Means Necessary', 'url': '/books/OL26004230M/Preppers_Home_Defense_Security_Strategies_To_Protect_Your_Family_By_Any_Means_Necessary', 'cover_url': '//covers.openlibrary.org/w/id/7545849-M.jpg', 'key': '/books/OL26004230M', 'authors': [<Storage {'name': 'Jim Cobb', 'key': u'/authors/OL7309309A'}>], 'ocaid': 'preppershomedefe0000cobb', 'borrow_url': '/books/OL26004230M/Preppers_Home_Defense_Security_Strategies_To_Protect_Your_Family_By_Any_Means_Necessary/borrow'}>]

[xayhewalo]

Giving this a work in progress label. @mekarpeles @brad2014 Let me know if you agree or disagree