Tripal: Pubmed importer fails because of 3/second request limit by NCBI API

[dsenalik]

BUG/ERROR report

Issue description

Pubmed importer has two major and one minor issues.

1. Multiple errors because requests are placed to the NCBI API more frequently than the 3/second limit for requests without an NCBI API key.
   Example of error: Warning: fopen(http://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?rettype=null&retmode=xml&retstart=0&retmax=1&db=Pubmed&query_key=1&WebEnv=NCID_1_17067917_130.14.18.48_9001_1582429467_1388293548_0MetA0_S_MegaStore&id=32073426): failed to open stream: HTTP request failed! HTTP/1.1 429 Too Many Requests in tripal_pub_PMID_fetch() (line 316 of .../sites/all/modules/tripal/tripal_chado/includes/loaders/tripal_chado.pub_importer_PMID.inc).
   ERROR: Could not perform PubMed query.

2. Publication importing loop exits after first iteration because the value of count($pubs) is reset by an array merge

3. Minor text fix, change http: to https: in API url and comments

A pull request is provided to fix these issues

[par12005]

Thanks for this pull request! Do you think it might be worth it to add an option to provide an NCBI API key for the importer? Then users could make up to 10 requests/second. Perhaps the option could go somewhere under admin/tripal/loaders?

Otherwise I was able to load 840 publications in ~15 minutes with no fatal errors.

[laceysanderson]

The original PR has been merged -Thank you @dsenalik!

Regarding @par12005 suggestion for an option to provide an API key for NCBI -we would welcome this contribution with the option on the admin/tripal/loaders/pub page. This addition would need to be optional but would help speed up large uploads. The 7,985 set I uploaded as a test took well over 2 hrs on my desktop.

[par12005]

@laceysanderson, I've submitted a PR which allows you to add an API key, if you wouldn't mind taking a look!

[laceysanderson]

Nice! I'll try to take a look at it tonight

[bradfordcondon]

merged, so closing. thanks all.