Tripal: TripalImporter errors related to NCBI importing

[risharde]

The following error is seen in /var/log/php-fpm/php-errors.log on a massive scale. In our case, the log files quickly filled our VM disk allocation which also causes other issues. This is on our live server.

As seen, it seems to be related to the TripalImporter where TaxonomyImporter.inc file contains the problematic code. The job being executed is importing NCBI data.

[27-Jul-2020 16:39:47 America/New_York] PHP Warning: feof() expects parameter 1 to be resource, boolean given in /var/www/Drupal/sites/all/modules/tripal/tripal_chado/includes/TripalImporter/TaxonomyImporter.inc on line 685 pid 16026
[27-Jul-2020 16:39:47 America/New_York] PHP Warning: fread() expects parameter 1 to be resource, boolean given in /var/www/Drupal/sites/all/modules/tripal/tripal_chado/includes/TripalImporter/TaxonomyImporter.inc on line 686 pid 16026
[27-Jul-2020 16:39:47 America/New_York] PHP Warning: feof() expects parameter 1 to be resource, boolean given in /var/www/Drupal/sites/all/modules/tripal/tripal_chado/includes/TripalImporter/TaxonomyImporter.inc on line 685 pid 16026

• Tripal Version: 3.1

[bradfordcondon]

Thank you @risharde

Do you have an example taxonomy ID or import job that is failing for you?

If we look at the line in question, we can see that fopen() is failing with your built URL and returning false instead of a file resource. Also possible that your fpm isnt set up to allow fopen on a remote URL?

https://github.com/tripal/tripal/blob/dd88e9fa9b497fe046913e97110b4f4721431edc/tripal_chado/includes/TripalImporter/TaxonomyImporter.inc#L676-L688

We can make an easy PR to to check that our request was successful from NCBI. However this wont fix your actual NCBI request, ~i wonder if its a valid taxon ID or if the API has changed~. I get back XML at https://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=taxonomy&id=1 for example so i'd imagine the problem is your php configuration.

edit: see this issue

https://stackoverflow.com/questions/9038694/fopen-is-not-working-on-my-server

check that allow_url_fopen is set in your php.ini. since you're using fpm you might have to find the php-fm.conf or equivalent config file instead

[laceysanderson]

I wonder if its a valid taxon ID or if the API has changed

I confirmed that the URL format + parameters is still correct for a valid taxon ID: https://www.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=taxonomy&id=362247.

I also confirmed the importer works without error on my setup (traditional server, not vm/docker/etc) using the same taxon id.

[par12005]

@bradfordcondon @laceysanderson I remember we worked on a similar issue in the Tripal Core Dev meetings a while back.

The problem is that the NCBI API now limits the number of requests per second (details here, under the section titled "Coming in December 2018: API Keys"). If >3 requests are made in one second without an API key, then NCBI will refuse the request, resulting in fopen() returning a boolean (false). This results in the following while loop continuing infinitely, and tossing warnings each time it calls feof() and fread(). On our server this filled up the php error log and ate all of our disk space within a matter of minutes.

I have submitted a PR #1074 which adds an optional NCBI API key and also performs some checks before trying to iterate over the boolean returned by fopen(). Most of the changes were based off of the work we did on #1033. Let me know what you guys think!

[bradfordcondon]

thank you @par12005 you're right it's entirely possible that Risharde's PHP is configured correctly and he's running into the API limit. I'll try to get a review on your request sooner than later. perhaps @risharde can also try it out. Thank you!

[laceysanderson]

@risharde I've merged the PR. As such I'm going to close out this issue. If you still see the errors when testing on production comment back here and I'll reopen but 🤞 you're good to go!