Dataverse: Native API file upload - internal server error

[gboeing]

I've been uploading files to my dataset for a couple days using a simple Python script and the native API endpoints. It was all working fine until this afternoon. Now, every file upload post request responds with status code 500 and an internal server error.

[pdurbin]

@gboeing thanks for the bug report. I'm sorry to hear you were getting 500 errors. Since you closed this issue I assume they stopped but if you have any more details to share that would help us reproduce the problem, I'd be happy to read more.

[gboeing]

Thanks. I got it mostly sorted out. I was using python requests package to upload the files. requests.post() takes a files argument that specifies a dictionary of files to upload. At some point I had absent-mindedly changed the dictionary's key from "file" to "each_file_name.zip". Per the requests documentation:

files -- (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.

So I thought I could change the key. But apparently the server expects the key to consistently be "file" because that started causing the 500 errors. It took a long time to trace it back to that change because the 500 errors are a bit opaque and didn't provide a message as to why it was erroring.

[pdurbin]

@gboeing ok, when you say "file" you're probably referring to this example from http://guides.dataverse.org/en/4.6.1/api/native-api.html#datasets

curl -H "X-Dataverse-key:$API_TOKEN" -X POST -F '[email protected]' -F 'jsonData={"description":"My description.","categories":["Data"]}' "https://example.dataverse.edu/api/datasets/:persistentId/add?persistentId=$PERSISTENT_ID"

Dataverse APIs shouldn't return 500 errors if you simply didn't use the right word ("file") in this context. I'm going to re-open this issue so we remember to investigate.

[gboeing]

Yes indeed, that was the example I based my script on.

[gboeing]

@pdurbin actually, I'm getting more 500 errors, now through the web interface.

See screenshot below. I click the circled download button, a spinner spins for 60 seconds, disappears, and my console displays the 500 error. I was successfully able to download another file on this page, but not the circled one. I tried multiple browsers and computers.

[gboeing]

@pdurbin and another example of more 500 errors. Here I open my dataset, click the "edit" button, then click the "metadata" link (see red arrow in screenshot below). A spinner spins for 60 seconds, disappears, and my console displays the 500 error.

[pdurbin]

@gboeing from that last screenshot it sounds like dataset in question is the one you mentioned at https://help.hmdc.harvard.edu/Ticket/Display.html?id=248690 . Let's let this GitHub issue be about the 500 error that results from passing something other than "file" to the native file upload. It sounds like we might need two more GitHub issues for the other 500 errors you've encountered. (Yuck.) If it's ok with you, let's use that 248690 ticket to gather the stack traces so we can zero in on the lines of code that are throwing exceptions. As of this writing https://dataverse.harvard.edu is running 4.6.1 build 69-1ed82ae. It's useful for us to know the 1ed82ae part so that we can find the line numbers in question.

[gboeing]

@pdurbin I have moved those other two errors (occurring on the web page) to new GitHub issues #3774 and #3775

[pdurbin]

@gboeing thanks for creating those issues! Much appreciated!

[gboeing]

@pdurbin just wondering if anything happened with this issue?

[pdurbin]

@gboeing not that I know of but I see https://help.hmdc.harvard.edu/Ticket/Display.html?id=248690 is still open. Let me follow up with the owner of that issue to see if there is any news. I assume you're only asking about this issue and not #3774 and #3775 but please let me know if you mean all three. Thanks for your patience and Happy New Year!

[pdurbin]

@gboeing I brought this up in backlog grooming today and some comments have been added to #3774 and #3775 but we discussed this issue as well.