Dataverse: Scheduled harvests aren't running

[jggautier]

On Harvard Dataverse 4.8.6, scheduled harvesting jobs aren't running when scheduled. Harvesting works only when pressing the "Run Harvesting" button.

[djbrooke]

@matthew-a-dunlap thanks for picking this up. #3672, #3669, and #3985 represent some previous work in this area.

[matthew-a-dunlap]

Talking more with @jggautier it looks like this predates 4.8.6. Will investigate more after my appointment today.

[matthew-a-dunlap]

I'm also assuming that #4548 has fixed at least some of this issue. @kcondon @pdurbin, have we tried running harvesting post #4548 for the clients stuck in production (as in, running the same harvests on a different box)?

[kcondon]

@matthew-a-dunlap The #4548 failure was a data failure, the scheduling one is a timer issue I believe. You can try out scheduling locally and ask Julian about specific harvesting jobs that are not running or what the pattern looks like.

[matthew-a-dunlap]

I have tried out scheduling locally on develop and 4.8.6 and harvesting seems to run as expected. Maybe there is a server timer configuration issue? Or some other variable in play that I do not have replicated on my local? @jggautier @kcondon

[landreev]

Yes, it appears to be turned off in production right now. (can't remember honestly when we turned it off or why...)
I'm turning it back on (yes, this is the JVM timer option that needs to be present on the designating primary server).

[matthew-a-dunlap]

@landreev Great! Thanks much for looking. I'm sure it was just one less thing to have in play when fixing everything else.

[landreev]

OK, turned it on.

/opt/dvn/app/glassfish4/glassfish/bin/asadmin create-jvm-options "-Ddataverse.timerServer=true"

Will keep an eye on the actual harvests, once they start running.

[jggautier]

I've been keeping track of Harvard Dataverse harvesting schedules in this Google sheet.

[matthew-a-dunlap]

@jggautier Good to know! Yea its hard to know what's happening without a "next scheduled harvest" column.

[matthew-a-dunlap]

Scheduled Harvests are now running in production now that the jvm timer is on. There are still failures IN those harvests, but they may be failing due to issues fixed in our dev branch.