Tripal: OBO importer tests run out of memory at > 2GB in Travis

[bradfordcondon]

The travis build is inconsistently failing, resulting in half passes/half fails. its really annoying and it makes it easy to ignore actual failures in the build/test.

I previously modified it to not test on the GOslim OBO, but thats not enough. https://travis-ci.org/tripal/tripal/jobs/474957282

As the below excerpt from the log shows, the tripal_obo_test vocabulary exhausts memory (sometimes).

While I'd prefer we not remove the OBO loader test, it might be unavoidable. Maybe instead we need to cover the individual methods, and get by without a functional test of the whole loader. Alternatively we need to somehow account for limited memory environments.

Building cvterm paths for vocabulary: tripal_obo_test
Clearing the cvtermpath table for this vocabulary...
Clearing completed.
Retrieving relationships...
Processing terms...
.PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 12288 bytes) in /home/travis/build/tripal/drupal/includes/database/database.inc on line 2227
PHP Stack trace:
PHP   1. {main}() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/phpunit:0
PHP   2. PHPUnit\TextUI\Command::main() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/phpunit:53
PHP   3. PHPUnit\TextUI\Command->run() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/TextUI/Command.php:151
PHP   4. PHPUnit\TextUI\TestRunner->doRun() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/TextUI/Command.php:198
PHP   5. PHPUnit\Framework\TestSuite->run() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:529
PHP   6. PHPUnit\Framework\TestSuite->run() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/Framework/TestSuite.php:776
PHP   7. PHPUnit\Framework\TestSuite->run() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/Framework/TestSuite.php:776
PHP   8. Tests\tripal_chado\TaxonomyImporterTest->run() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/Framework/TestSuite.php:776
PHP   9. PHPUnit\Framework\TestResult->run() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/Framework/TestCase.php:798
PHP  10. Tests\tripal_chado\TaxonomyImporterTest->runBare() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/Framework/TestResult.php:645
PHP  11. Tests\tripal_chado\TaxonomyImporterTest->runTest() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/Framework/TestCase.php:840
PHP  12. Tests\tripal_chado\TaxonomyImporterTest->testImportExistingTaxonomyLoader() /home/travis/build/tripal/drupal/sites/all/modules/tripal/vendor/phpunit/phpunit/src/Framework/TestCase.php:1145
PHP  13. TaxonomyImporter->run() /home/travis/build/tripal/drupal/sites/all/modules/tripal/tests/tripal_chado/loaders/TaxonomyImporterTest.php:36
PHP  14. TaxonomyImporter->updateExisting() /home/travis/build/tripal/drupal/sites/all/modules/tripal/tripal_chado/includes/TripalImporter/TaxonomyImporter.inc:254
PHP  15. feof() /home/travis/build/tripal/drupal/sites/all/modules/tripal/tripal_chado/includes/TripalImporter/TaxonomyImporter.inc:486
PHP  16. _drupal_error_handler() /home/travis/build/tripal/drupal/sites/all/modules/tripal/tripal_chado/includes/TripalImporter/TaxonomyImporter.inc:486
PHP  17. _drupal_error_handler_real() /home/travis/build/tripal/drupal/includes/bootstrap.inc:2591
PHP  18. _drupal_log_error() /home/travis/build/tripal/drupal/includes/errors.inc:75
PHP  19. watchdog() /home/travis/build/tripal/drupal/includes/errors.inc:210
PHP  20. module_invoke() /home/travis/build/tripal/drupal/includes/bootstrap.inc:2012
PHP  21. dblog_watchdog() /home/travis/build/tripal/drupal/includes/module.inc:934
PHP  22. InsertQuery_pgsql->execute() /home/travis/build/tripal/drupal/modules/dblog/dblog.module:164
PHP  23. DatabaseConnection_pgsql->query() /home/travis/build/tripal/drupal/includes/database/pgsql/query.inc:96
PHP  24. DatabaseStatementBase->execute() /home/travis/build/tripal/drupal/includes/database/pgsql/database.inc:106
PHP  25. DatabaseStatementBase->execute() /home/travis/build/tripal/drupal/includes/database/database.inc:2227
Percent complete: 37.93 %. Memory: 56,453,824 bytes.
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 12288 bytes) in /home/travis/build/tripal/drupal/includes/database/database.inc on line 2227

some notes on the above failure:

• it ran out of memory at 1GB. thats pretty insane, considering its loading a 120 line OBO file.
• Travis should give us 8GB memory https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system.

[bradfordcondon]

right ok problem is obvious now: we need to up the mem limit for PHP within the travis container. 2GB should be better.

lets jsut try upping the mem limit in the before script:

# up memory limit of PHP
  - echo "memory_limit=2G" >> ~/.phpenv/versions/$(phpenv version-name)/etc/conf.d/travis.ini

[bradfordcondon]

we're running out of memory on the OBO test with 2GB memory. loading a 120 line OBO.

@spficklin this is definitely a memory leak issue yeah?

` Building cvterm paths for vocabulary: tripal_obo_test Clearing the cvtermpath table for this vocabulary... Clearing completed. Retrieving relationships... Processing terms... .PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 12288 bytes) in /home/travis/build/tripal/drupal/includes/database/database.inc on line 2227 PHP Stack trace:

[bradfordcondon]

im proposing we disable the tests for the OBO file until this is resolved as it makes dealing with all other PRs a pain.

[spficklin]

I'm not sure if that's a memory leak per se. I don't know why it's consuming 2GB of RAM.

[laceysanderson]

I had to re-run tests on 7.x-3.x multiple times today until they passed. I consider this issue critical since it will show in our readme if our tests on master are failing. @spficklin can we disable the OBO test until we figure out why it's consuming so much memory?

[spficklin]

Sure. No complaints from me.

[bradfordcondon]

PR merged, renaming issue to describe current status, which is that the OBO tests cause the test env to run out of memory.

[bradfordcondon]

ok we've got two things going on here.

importer test fails with too many organisms

The taxonomy importer test, that checks the import_existing flag is what runs out of memory.

https://github.com/tripal/tripal/blob/85b3c2988e5ebc8083c67b34ed8fa1ef7efd253e/tests/tripal_chado/loaders/TaxonomyImporterTest.php#L18-L47

it does this because the taxonmy importer does no error checking from NCBI when downloading from NCBI taxon.
We run out of memory because we're spamming 2gb worth of this error somehow
fread() expects parameter 1 to be resource, boolean given TaxonomyImporter.inc:487 [warning]

if we add a check that we got an openable file back (and not an error message), we'll be good

some rollback isnt working.

why do we know this? because we have 30 organisms, which is why the importer test is failing: NCBI is limited to a set number of trnasactions/second, and we exceed it!

Could not look up Trevor+Barton+Sr.+Darion+Kassulke+DDS+Dulce+Strosin+Jr.+
fopen(http://www.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=taxonomy&term=chinese++chestnut): failed to open stream: HTTP request failed! HTTP/1.1 429 Too   [warning]
Many Requests

So, unresolved: why do we have so many test organisms in the database? what rollback os failing?

resolved, hopefully: why are we running out of memory?

[bradfordcondon]

I added a sleep failsafe to prevent sending more than 3 requests/second to NCBI, thus fixing the taxonomy Importer.

I'm at a loss why we still have organisms in our database....

[almasaeed2010]

Here is why we have organisms, stocks and projects in our DB (and their corresponding entities) https://github.com/tripal/TripalTestSuite/issues/136#issuecomment-457186961

I've implemented the solution suggested in that issue and will submit a PR soon.

[bradfordcondon]

ok. this is resolved on both sides. big thanks to @almasaeed2010 for providing the other half of the fix.