Openlibrary: Eligibility API errors when invalid isbn - 'NoneType' object has no attribute 'get'

[brierjon]

Relates to 2194 and possibly 2517

Ungraceful response for item not found, but apparently attempting to be added.
When I went back to test the URL it resolves with no error - this is a temporary error, but one which needs a more graceful response.

Evidence / Screenshot (if possible)

'NoneType' object has no attribute 'get'

/opt/openlibrary/deploys/openlibrary/openlibrary/openlibrary/catalog/add_book/__init__.py in add_cover, line 222

GET https://dev.openlibrary.org/sponsorship/eligibility/9781482217865

Relevant url?

https://dev.openlibrary.org/sponsorship/eligibility/9781482217865?debug=true

relevant server log: 2019-11-20/122838184699

Steps to Reproduce

1. Send valid ISBN, but not found in OpenLibrary and is found on Amazon.com and not found on Better World Books

• Actual: error thrown
• Expected: response such as "not found, but acquisition in progress" check back in x minutes or this book is 58th in line for import.

Details

• Logged in (Y/N)?
• Browser type/version? N/A
• Operating system? N/A
• Environment (prod/dev/local)? prod and dev tested
  

Proposal & Constraints

Related files

Stakeholders

[brierjon]

Experienced the same issue where the error occurs and the book is listed on both Amazon.com and Better World Books by that ISBN.

Error Log: 2019-11-21/130952174436
URL: https://openlibrary.org/sponsorship/eligibility/9781682827352

[brierjon]

Also fails similarly for the following ISBN13:

9781594859410
9781631632952

[tfmorris]

Apologies! It looks like this fell through the cracks in our triaging processing, but all three example ISBNs above now return valid results from the API.

We've changed several related things recently including the version of Amazon API, handling of ISBN normalization and 10/13 conversion, etc, so it was probably fixed as a side effect of one of those.

Please feel free to reopen if you find there's still and problem.

[brierjon]

@tfmorris still seeing valid ISBNs reporting as invalid in response. May need a few more lookup endpoints for improved handling.

https://openlibrary.org/sponsorship/eligibility/9789535111887
Tried also with dev and debug=true, but nothing but {"status": "error", "reason": "Invalid ISBN 13"} was returned.

Not Found on:

• Better World Books - not found: https://www.betterworldbooks.com/search/results?q=9789535111887
• Amazon.com lookup - not found: https://www.amazon.com/s?k=9789535111887&ref=nb_sb_noss

Found on:

• Amazon.co.uk lookup - found! https://www.amazon.co.uk/s?k=9789535111887&ref=nb_sb_noss
• Crossref lookup - found! https://search.crossref.org/?q=9789535111887&type-name=Book - metadata may have compatible license as Wikidata tools also are using the Crossref API for imports. Might add as fallback check limiting api call to book isbn.

Book being referenced - Hardcover:
https://www.intechopen.com/books/anopheles-mosquitoes-new-insights-into-malaria-vectors
Published: July 24th 2013
DOI: 10.5772/3392

[tfmorris]

Reopening per OP's request (not sure why you don't have permission to do it yourself @brierjon).

Having said that, the team may just end up closing it again since you appear to be attempting to use an internal API which is undocumented for external use (and a very unreliable on at that, as you've discovered).

What are you trying to do? Do none of the documented APIs support your use case? If you're trying to determine whether the ISBN is "valid", a simple checksum is the best you can do. If you want to determine if it's in use, there are much bigger databases (WorldCat, Amazon, etc) that you could check against.

[brierjon]

Thanks for opening, this issue may need a better title as it is for ongoing work on a graceful handling of valid ISBNs in the sponsorship query related to #3168 and the graceful handling process and the comment on 2517. The same core issue is present as the response is not correct for valid ISBN values.

My usecase is in line with the the sponsorship API for the functional purpose of determining if a collection of ISBN values is available for sponsorship on OL. I'm working on a donation campaign for specific subject areas. The ISBN passed are valid unless there is was a data error in the source, but a robust API will handle that gracefully.

Functionally no valid ISBN should receive a response that states the ISBN is invalid regardless if queried externally or within OL. It should state ineligible or unable to determine eligibility as current sources do not offer the book for sale. Here I was pointing out the availability differs across different Amazon domains and Crossref would be a possible endpoint for Importbot to pull metadata.

The Sponsorship API is not stated or enforced with access control as an internal API, it is functionally just an undocumented API as it is exposed to the Internet. When there is no access control mechanism an API is public in the technical sense, it just may not be reliable. In reporting valid ISBNs receiving an error response from the API demonstrates the verification and response logic is not yet valid regardless of use internally or more broadly.

[BrittanyBunk]

@brierjon The book you are looking for is not in the OL catalog. I don't believe the book will be available for sponsorship unless it's there, because I see that the bot only takes up books once they're in the catalog. If you want it to be available for sponsorship, you'll need to add the book in.

[brierjon]

@BrittanyBunk it is correct that not all books with valid ISBN are in OL, but the issues is mainly that the API needs to respond with an accurate error statement not generic error. The invalid ISBN 13 is not accurate, but could not locate ISBN 13 is accurate. Even better would be could not locate ISBN 13 the book is not yet found in open library or an open library partner 1. xyz, 2. zzz, etc.

[BrittanyBunk]

@brierjon I see. Maybe you could give a before and after picture so that people understand what you mean, it's really difficult to go from the title to what you're saying now.