Hyrax: jobs should be idempotent

[hackartisan]

Descriptive summary

AttachFilesToWorkJob, e.g. adds files repeatedly if re-run. Jobs need to be reviewed for idempotence and updated where they are not idempotent.

Rationale

When background jobs fail it should be possible to restart them and get a successful result. Sidekiq, for example, re-queues failed jobs automatically.

[atz]

This is absolutely critical.

[jcoyne]

"Add a thing to a list" is not an idempotent operation. I don't know how we would make a job be idempotent if the operation itself is not.

[hackartisan]

"Ensure the thing is in the list" can be idempotent; I think that would be acceptable logic.

If it can't be made idempotent maybe it shouldn't be a job?

[jcoyne]

@hackmastera but this is a list, not a set, you can have duplicates.

[hackartisan]

@jcoyne maybe it's worth investigating whether we have an actual use case for duplicates. I'm willing to sacrifice that flexibility.

[jcoyne]

I don't think it matters whether we have a use case as we are using a linked list to model the data. The operation is "Create a new file set" (create is not idempotent), "create a new bitstream" (not idempotent), attach bitstream to the FileSet (idempotent!) and then "attach it to a linked list" (also not idempotent).

[hackartisan]

If it can't be made idempotent it maybe shouldn't be run as a job.

If we want to keep jobs for some parts of this process, maybe there is a way to do the object creation outside of a job and then the links as jobs. "attach to linked list" can be idempotent; if we accept a constraint that an object can only be there once, we can check for it before adding.

@jcoyne do you think non-idempotent jobs are a non-issue?

[cjcolvar]

:+1: I was thinking the same thing. Maybe we can assume the unique constraint as the default but allow overriding similar to AddFileToFileSet in hydra-works: https://github.com/projecthydra/hydra-works/blob/master/lib/hydra/works/services/add_file_to_file_set.rb

[mbklein]

Dredging this back up (because we're currently running into something similar).

“Add a thing to a list” is not an idempotent operation, but “apply these changes to this (new or existing) object” _has_ to be an idempotent operation. That might mean changing something other than the specific job referenced in this issue, but the feature as written is (still) broken.

[mbklein]

I also have questions about this issue being labeled as enhancement and technical debt when IMO it falls pretty solidly in bug territory.

[no-reply]

thanks @mbklein. I agree that this is a significant bug. There are two scopes here:

• [ ] make AttachFileToWorkJob idempotent, since it has known issues
• [ ] audit all delayed jobs for idempotency.

I fear there's a mess of latent problems waiting to be found in the second scope, and I'd rather expend my own effort on finishing up the dry-transactions actor stack replacement and the job rework that would come with that.

The first scope seems clearly like it belongs in 2.x backport level priority.

[no-reply]

I don't see the benefit in any case of adding multiple FileSets, generated from the same UploadedFile to the same Work.

• If a user intends to have multiple FileSets with the same content, they can upload the file repeatedly.
• If a user intends to have the same FileSet attached to a work repeatedly, that can be done at the command line.

Contrary to the discussion here, it is not currently possible to add duplicate list entries using the AttachFilesToWorkJob; instead, running the job repeatedly results in wholly new FileSets.

I think the appropriate constraint is: each UploadedFile should result in a single FileSet. This isn't in conflict with any properties of linked lists, works, etc...

[mbklein]

I agree with the above assessments. I'm excited about the dry-transactions work, but I'm still concerned about how we evaluate the done-ness of asynchronous jobs, especially if we have steps and jobs that depend on outputs/artifacts from other async jobs. This ticket probably isn't the place to address all of that.

[tahirpoduska]

We are seeing this issue affect our site, additional FileSets are created for the same uploaded file and causes other duplication based on our customizations. We have switched to using 'perform_now' for 'AttachFilesToWorkJob' and 'IngestJob' calls in the actors.

[no-reply]

@tap87 I'm surprised to be seeing this as a current issue, since #3408 should have closed it, months ahead of your comment.

Can someone update about the status on recent hyrax versions

[no-reply]

Note that this is open to resolve the remaining issues in #3408. The multiple attachment issue described here should be resolved in the latest on the 2.x branch as of Nov, 2019

[jlhardes]

Closing this because there are no use cases or known issues associated with this to solve.