Openlibrary: save_many aka HTTPError: 400 Client Error when using OL-Client

[cclauss]

Related to #4285

@seabelis reported via Slack...

Evidence / Screenshot (if possible)

Relevant url?

Steps to Reproduce

1. Go to Colab.
2. Do ...

dupes = 'OL...W'  #@param { type: "string" }
into_olid = 'OL...W'  #@param { type: "string" }

works = [ol.Work.get(olid) for olid in dupes.split(',')]
for work in works:
  editions = work.editions
  print(f"{work.olid}: {len(editions)} edition(s)")

Run

Then

save_set = []
for work in works:
  editions = work.editions
  for ed in editions:
    ed.work_olid = into_olid
    save_set.append(ed)
response = ol.save_many(save_set, 'moving edition(s) to primary work')
response.raise_for_status()
response

save_set = [.Edition' {'_work': None, 'work_olid': 'OL1168059W', 'olid': 'OL7362672M', 'description': None, 'notes': None, 'identifiers': {'librarything': ['23835']}, 'title': 'In Front of Your Nose, 1945-1950 (Collected Essays, Journalism and Letters of George Orwell, Vol 4)', 'subtitle': None, 'pages': None, 'authors': [.Author' {'olid': 'OL118077A', 'identifiers': {}, 'name': 'George Orwell', 'bio': 'George Orwell, originally born as Eric Arthur Blair, was an English novelist and journalist. His work is marked by keen intelligence and wit, a profound awareness of social injustice, an intense, revolutionary opposition to totalitarianism, a passion for clarity in language and a belief in democratic socialism. ([Source][1].)\r\n\r\n[1]:http://en.wikipedia.org/wiki/George_Orwell', 'personal_name': 'George Orwell', 'wikipedia': 'http://en.wikipedia.org/wiki/George_Orwell', 'death_date': '21 January 1950', 'alternate_names': ['Eric Arthur Blair', 'George Orwell', 'Orwell, George, 1903-1950', 'George Orwel', "George ' 'Orwell", 'GEORGE ORWELL'], 'created': {'type': '/type/datetime', 'value': '2008-04-01T03:28:50.625462'}, 'photos': [9160343, 6388323], 'last_modified': {'type': '/type/datetime', 'value': '2020-11-22T12:05:19.599246'}, 'latest_revision': 17, 'birth_date': '25 June 1903', 'revision': 17, 'type': {'key': '/type/author'}, 'remote_ids': {'viaf': '95155403', 'wikidata': 'Q3335', 'isni': '0000000121441305'}}>, .Author' {'olid': 'OL2660467A', 'identifiers': {}, 'name': 'Sonia Orwell', 'bio': None, 'last_modified': {'type': '/type/datetime', 'value': '2008-04-29 13:35:46.87638'}, 'type': {'key': '/type/author'}, 'id': 10034588, 'revision': 1}>], 'publisher': None, 'publish_date': 'June 1971', 'cover': None, 'publishers': ['Harcourt'], 'languages': [{'key': '/languages/eng'}], 'last_modified': {'type': '/type/datetime', 'value': '2010-08-05T22:21:59.091298'}, 'series': ['Collected Essays, Journalism, and Letters of George Orwell (4)'], 'contributions': ['Ian Angus (Editor)'], 'created': {'type': '/type/datetime', 'value': '2008-04-29T13:35:46.876380'}, 'isbn_13': ['9780151185498'], 'isbn_10': ['0151185492'], 'latest_revision': 4, 'type': {'key': '/type/edition'}, 'revision': 4}>]

• Actual:

Error using client:
HTTPError                                 Traceback (most recent call last)
<ipython-input-5-d841aca49599> in <module>()
      6     save_set.append(ed)
      7 response = ol.save_many(save_set, 'moving edition(s) to primary work')
----> 8 response.raise_for_status()
      9 response
/usr/local/lib/python3.6/dist-packages/requests/models.py in raise_for_status(self)
    939 
    940         if http_error_msg:
--> 941             raise HTTPError(http_error_msg, response=self)
    942 
    943     def close(self):

___HTTPError: 400 Client Error: Bad Request for url: https://openlibrary.org/api/save_many___

• Expected:

Details

• Logged in (Y/N)?
• Browser type/version?
• Operating system?
• Environment (prod/dev/local)? prod
  

Proposal & Constraints

Related files

Stakeholders

[seabelis]

Issue started 2020-03-12 following Python 3 upgrade.

[cdrini]

Can repro this when trying to merge works on staging: http://staging.openlibrary.org/works/merge?records=OL520614W,OL18391608W

If you copydocs these works into your local environment, you should be able to test it out there.

[cdrini]

Here's the callstack; my suspicions lie on the marshal function and the urllib.request.Request call.

https://github.com/internetarchive/openlibrary/blob/9c194ce8444ff1a215cb0175cfcd3bb49978de7f/openlibrary/api.py#L151-L152

https://github.com/internetarchive/openlibrary/blob/9c194ce8444ff1a215cb0175cfcd3bb49978de7f/openlibrary/api.py#L136-L149

https://github.com/internetarchive/openlibrary/blob/9c194ce8444ff1a215cb0175cfcd3bb49978de7f/openlibrary/api.py#L45-L57

[cdrini]

curl 'http://staging.openlibrary.org/api/save_many' -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.7,de-DE;q=0.3' --compressed -H 'Referer: http://staging.openlibrary.org/works/merge?records=OL520614W,OL18391608W' -H 'Opt: "http://openlibrary.org/dev/docs/api"; ns=42' -H '42-comment: Merge works' -H 'Content-Type: text/plain;charset=UTF-8' -H 'Origin: http://staging.openlibrary.org' -H 'Connection: keep-alive' -H 'Cookie: donation-identifier=MC45OTk4NjA2NzUzNjkwOTE=; session=/people/ScarTissue%2C2020-12-31T16%3A46%3A49%2Caa1b6%24850d439b25b5dd4036642581d153f933' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' --data-raw '[{"created":{"type":"/type/datetime","value":"2009-12-08T04:21:26.300832"},"subjects":["Available light photography"],"latest_revision":8,"key":"/works/OL520614W","title":"Available light","authors":[{"type":{"key":"/type/author_role"},"author":{"key":"/authors/OL36372A"}}],"type":{"key":"/type/work"},"last_modified":{"type":"/type/datetime","value":"2020-12-31T16:47:01.801"},"covers":[9643483,8358090],"revision":8},{"location":"/works/OL520614W","key":"/works/OL18391608W","type":{"key":"/type/redirect"}},{"publishers":["Petersen Pub Co"],"source_records":["amazon:0822740427"],"title":"Available Light","created":{"type":"/type/datetime","value":"2019-02-06T08:39:49.822817"},"last_modified":{"type":"/type/datetime","value":"2019-02-06T08:39:49.822817"},"covers":[8358090],"isbn_13":["9780822740421"],"full_title":"Available Light","isbn_10":["0822740427"],"publish_date":"Nov 06, 1980","key":"/books/OL26687783M","authors":[{"key":"/authors/OL36372A"}],"latest_revision":1,"works":[{"key":"/works/OL520614W"}],"type":{"key":"/type/edition"},"revision":1}]'

await fetch("http://staging.openlibrary.org/api/save_many", {
    "credentials": "include",
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0",
        "Accept": "*/*",
        "Accept-Language": "en-US,en;q=0.7,de-DE;q=0.3",
        "Opt": "\"http://openlibrary.org/dev/docs/api\"; ns=42",
        "42-comment": "Merge works",
        "Content-Type": "text/plain;charset=UTF-8",
        "Pragma": "no-cache",
        "Cache-Control": "no-cache"
    },
    "referrer": "http://staging.openlibrary.org/works/merge?records=OL520614W,OL18391608W",
    "body": "[{\"created\":{\"type\":\"/type/datetime\",\"value\":\"2009-12-08T04:21:26.300832\"},\"subjects\":[\"Available light photography\"],\"latest_revision\":8,\"key\":\"/works/OL520614W\",\"title\":\"Available light\",\"authors\":[{\"type\":{\"key\":\"/type/author_role\"},\"author\":{\"key\":\"/authors/OL36372A\"}}],\"type\":{\"key\":\"/type/work\"},\"last_modified\":{\"type\":\"/type/datetime\",\"value\":\"2020-12-31T16:47:01.801\"},\"covers\":[9643483,8358090],\"revision\":8},{\"location\":\"/works/OL520614W\",\"key\":\"/works/OL18391608W\",\"type\":{\"key\":\"/type/redirect\"}},{\"publishers\":[\"Petersen Pub Co\"],\"source_records\":[\"amazon:0822740427\"],\"title\":\"Available Light\",\"created\":{\"type\":\"/type/datetime\",\"value\":\"2019-02-06T08:39:49.822817\"},\"last_modified\":{\"type\":\"/type/datetime\",\"value\":\"2019-02-06T08:39:49.822817\"},\"covers\":[8358090],\"isbn_13\":[\"9780822740421\"],\"full_title\":\"Available Light\",\"isbn_10\":[\"0822740427\"],\"publish_date\":\"Nov 06, 1980\",\"key\":\"/books/OL26687783M\",\"authors\":[{\"key\":\"/authors/OL36372A\"}],\"latest_revision\":1,\"works\":[{\"key\":\"/works/OL520614W\"}],\"type\":{\"key\":\"/type/edition\"},\"revision\":1}]",
    "method": "POST",
    "mode": "cors"
});

[cdrini]

Error you were asking for from work merge:

{
  "error": "bad_data",
  "message": "Bad JSON: Expecting value: line 1 column 1 (char 0)"
}

[SouthGoingZax]

Per our off-issue discussion, that's the same error seen in https://github.com/internetarchive/openlibrary/issues/4285#issuecomment-749400874, so it makes sense that the PR there resolves this issue too.

[cclauss]

Was this issue closed by #4338 ?