Ckan: Harvesting and DataPusher issues

[amiedes]

Introduction

Hi, I've been working on a CKAN setup that was using harvesting, DataPusher and DataStore to download and import into our app several datasets periodically everyday. We faced several issues during the process which I already asked in the mailing list.

We got no answer so I'd like to leave our case and solutions documented here so it's easy to find for other people facing the same issues.

Expected behavior

Once a day we run the harvesting process, we expect:

• Dataset and resources to be downloaded and created or updated in CKAN
• Be uploaded to DataStore when the process has finished, and their default views to be created

Problem

1. There's a race condition in CKAN/DataPusher code that for some cases causes a resource datastore_active_fag be set to false at the end of the import process. This causes the resource views depending on DataStore to not be created.

   • What gets set to false is the value in the Solr document, in Postgres it's OK.
   • This occurs more often if the dataset has lots of resources.
   • The resources affected by this vary between each execution of the harvesting process.
   • Although the flag is set to false, the data is correctly indexed in the datastore db.

2. The harvesting process is capable of detecting which datasets/resources are new and which ones are being updated if you map correctly the attributes you read from the RDF. This is important because otherwise every time you run the harvester you'll create new tables in datastore but the old ones won't be deleted. The problem is that right now DataPusher is only triggered when a new resources is added, so when you try to update one the datastore_active_fag will be set to false, and since the resubmit is not triggered it won't be updated to true later and the resource views will become missing.

   • Relevant parts of code here and here.

Solutions

To solve problem 1:

• A periodic task that checks the value of all datastore_active flags in postgres, and updates the values in Solr if they're different.
• A periodic tasks that recreates the resource views if they're missing

To solve problem 2:

• Run task datapusher submit_all everyday after the harvesting

Setup description

• CKAN 2.8.2
• Relevant extensions installed: DataStore, DataPusher, ckanext-harvest, ckanext-dcat
• Running on Docker with the following containers:
  
  
  
  • web server
  
  
  • app server (apache)
  
  
  • cron
  
  
  • supervisor
  
  
  • datapusher server (apache)
  
  
  • postgresql
  
  
  • solr
  
  
  • redis
  
  

Related

• https://github.com/ckan/ckan/issues/3245
• https://github.com/ckan/datapusher/issues/100

[DerekW24]

I've experienced similar when uploading via api larger (500k + rows) datasets. I solved this by using the xloader plugin. Its much faster than datapusher, and perhaps should be the default on new installs. Datasets that were problematic to load with datapusher loaded fine with xloader. By default with xloader it loads all into text fields, but I understand you can change the datatype after.
https://github.com/ckan/ckanext-xloader

[amiedes]

We read about xloader once we were already deep into the Datapusher setup, and decided to discard it for the moment since we did not have experience with it and we weren't sure if it was stable enough. But for the next time we'll consider giving it a try.

[jqnatividad]

xloader was meant to be a "drop-in" replacement for datapusher and is production-ready.

Datapusher was the single largest source of production support post-install issues for CKAN implementations I've been involved in and highly recommend you use it instead of datapusher.

https://github.com/ckan/ckan/issues/4415
https://ckan.org/2017/11/20/express-loader-for-ckan-datastore/