Tripal: Tripal Collections are very slow to create for Drupal Search API Views

[laceysanderson]

BUG/ERROR report

Follow-up Issue for #351.

System information

• Tripal Version: 7.x-3.x
• Drupal Version: n/a
• PostgreSQL Version: n/a
• PHP Version: n/a

Issue description

Currently creating a Tripal Collection from a Drupal Search API view is very slow (e.g. 5 min to create a collection with 400 members).

This is due to needing to lookup the bundle for each item in the collection in order to create it.
https://github.com/tripal/tripal/blob/7.x-3.x/tripal/views_handlers/tripal_views_handler_area_collections.inc#L413-L421

  // First, determine the bundle for each entity in our resultset.
  $entities = array();
  $results = unserialize($form_state['values']['results']);
  foreach($results as $r) {
    // Retrieve the bundle for the current entity.
    // ASSUMPTION: all search results are TripalEntities.
    $wrapper = entity_metadata_wrapper('TripalEntity', $r->entity);
    $bundle = $wrapper->getBundle();

Additionally, we need to loop through all bundles to get the list of fields attached to them.
https://github.com/tripal/tripal/blob/7.x-3.x/tripal/views_handlers/tripal_views_handler_area_collections.inc#L450-L458

  foreach($entities as $bundle_name => $bundle_col) {
    $field_ids = array();
    foreach (field_info_instances('TripalEntity', $bundle_name) as $field) {
      $field_ids[] = $field['field_id'];
    }
    $bundle_col['fields'] = $field_ids;
    $collection->addBundle($bundle_col);
  }

NOTE: This is not an issue for non-Drupal Search API views, since they are created specific to a single bundle (no need to look it up) and fields are looked up during loading of the view (and cached) rather then during collection creation. We can't look up fields during loading of the view since we don't know which bundles our results include.

Steps to reproduce

1. Set up the Drupal Search API as indicated in this tutorial: http://tripal.info/tutorials/v3.x/searching/search-api
2. Edit the search view created in the tutorial to add Global: Tripal Content Data Collections to the footer.
3. On the search page, create a tripal collection.

[spficklin]

So, doing a bit of digging, I don't think the problem is the lookup of the bundle. I think the problem is that the $form_state['values']['results'] comes pre-populated with the full entity for every match! This means that if you have 400 results, the Search API will generate 400 entities, which means it must run through all of those entities and generate values for all the fields (or at least those indexed). I tried a bit of digging down into the SearchAPI to try to find out if we can tweak this behavior in anyway, but I didn't make it too far.

[spficklin]

Okay, I have figured out the culprit, and I think I understand why @laceysanderson and I had different records for entities when testing PR #351. When highlighting is turned on for the search index, then that filter will fully load the entities and all of the fields of type 'fulltext', This I believe is what is causing the slowness. It takes time to load all the indexed fields for 400 entities. If the entity fields have been cached then that will go faster. If highlighting is turned off then only the entity ID is returned as a result in the code @laceysanderson mentioned above. And if highlighting is turned off then I think creating a data collection will go much faster.

@bradfordcondon can you try turning off the "Highlighting" checkbox (Home » Administration » Configuration » Search API, find the index and click the 'Filters' tab) and then return to the view, repeat the search, save the data collection and see if it still takes 5 minutes?

So, I think the only way to fix this would be to update the Search API to only load the entities for those that appear on the page and not all of the results. Then at most it only loads 25 or so entities. We'd have to change their code though :-/

[spficklin]

So, I guess I should ask. If we can't fix this very easily. Should we remove the functionality for now? I would rather not offer it than have something be extremely slow....

[laceysanderson]

If we know the slowness is specific to the "Highlighting", I would be more tempted to add a Tripal Notice in the Views UI Settings for the Tripal Collections handler warning admin of this and telling them to turn off highlighting.

I need this feature for KnowPulse and we don't use the highlighting (my preference is to display results in a table). So please don't take it away! ;-p

[spficklin]

@laceysanderson do you have slowness creating large collections when you have highlighting turned off? Also, just curious, how do you format your search results without highlighting?

I'm good with your suggested approach, BTW.

[spficklin]

I just added a PR that does what you asked for @laceysanderson.

[bradfordcondon]

hi @spficklin @laceysanderson my highlighting was never turned off so i dont think this was the cause of my slow collection creation.

[laceysanderson]

Darn.

So we figured out why @spficklin had entities vs. me simply having their ids 👍 Loading the full entities was likely a point of slowness, although Drupal seems to use impressive caching to make this less of a problem.

However, it's still way to slow even with "highlighing" turned off :-( I'll do some testing today and see what I can figure out.

[spficklin]

Just checking on this @laceysanderson any progress?

[laceysanderson]

No progress :-( Upgrading KnowPulse to Tripal3 is taking a lot of my time :-(

[spficklin]

I don't think we have time to fix this one before the stable release. This isn't a bug, just a problem that we can address later. I'm taking off the release blocker.

[bradfordcondon]

i'm adding this to the 3.2 project. I think collections are ham-stringed by this issue (and also by the field picking which we can discuss as well) and its worth addressing.

[laceysanderson]

This is only a problem for people using the Drupal Search API views. KnowPulse no longer uses this functionality and I simply do not have the time to work on this. I suggest closing it out.

[laceysanderson]

Closing this out as I simply do not have the time/need to work on it. I would be happy to collaborate if someone else needs the functionality though!