Hyrax: Ldp::Conflict occurring on all new works once it starts

[elrayle]

Descriptive summary

In Hyrax 2.1.0, I am able to create a bunch of works. At some point, creating a new work starts getting an Ldp::Conflict, "Can't call create on an existing resource" error. After that, all new works get this error. I am able to clear the problem by using ActiveFedora::Cleaner. This is not a viable solution if this occurs in production.

Steps to reproduce the behavior

Unfortunately, I do not have a sense of what starts triggering the Ldp::Conflict error to start. Once it does, you can see it with every attempt to create a new work.

Related work

https://github.com/samvera/ldp/issues/91

[elrayle]

Discussions from Slack:

@no-reply

I doubt that this conflict thing is actor stack related, but instead has to do with minter state.

if the minter state is emitting an id that a work already has (i.e. it's always one behind), i'd expect Conflict on every #create call.

could confirm by bumping the minter state counter in the db manually
i.e. take #seq and bump it to some arbitrary number 10_000. https://github.com/samvera/noid-rails/blob/master/db/migrate/20160610010003_create_minter_states.rb#L9
notably, I haven't run into this in prod in over a year of hosting a Hyrax 1.0 app running AF::Noid. I wonder whether we have an issue that's new in Hyrax 2.x with Noid::Rails, or else an issue that's db specific

Several replies about databases in use which included MySql, Postgres, and sqlite.

[elrayle]

I tried bumping the seq number in MySql db. No effect.
I restarted rails after bumping seq number. No effect.

We finally cleared the error by...

• when ldp::conflict happened, we dumped the URL
• deleted the resource from Fedora
• deleted the tombstone from Fedora

Now we can create works again.

[elrayle]

We are able to reproduce the Ldp::Conflict problem by...

• Dashboard -> Works -> New Work
• upload word doc file (without soffice properly installed)
• save

This gets an error about soffice not working.

• Dashboard -> Works -> New Work
• upload txt file
• save

This gets Ldp::Conflict error.

[elrayle]

From @no-reply in Slack...

oh, that's interesting... there would be an ActiveRecord transaction rollback when the actor stack fails. I wonder if that is the issue.
i.e. minter state rollback once, then each time you try to create with the stack, you fail with Conflict and rollback again.

[elrayle]

From @jcoyne in Slack...

The rollback does seem likely.
I would love for the minter to live in its own isolated transaction. I wonder if that’s possible.

[fereira]

The server upon which I encountered the issue is for an organization in Kenya (it's an AWS EC2 instance) that is chomping at the bit to get into production. I have since installed LibreOffice 6.0 on that machine and would rather not break it again. I think I can reproduce the issue on my laptop though by renaming soffice and trying to upload a Word document. That will generate a stack trace that indicates that it can't find soffice. Any subsequent attempt to upload any documents (even text files or different work types) will produce the ldp::Conflict error. I made a local change to lib/ldb/resource.rb to add #{subject} in the create method when it raises the Ldb::Conflict exception. That displays the URL in fedora is causing the conflct. Removing that resource and its tombstone will "fix" the app so that works can be uploaded.

[no-reply]

I haven't been able to reproduce locally, and I'm still not clear on whether you're able to successfully mint new ids outside of the actor stack.

Could you post the output of: [::Noid::Rails::Service.new.minter.mint, ::Noid::Rails::Service.new.minter.mint]

Also, that LibreOffice is implicated suggests that derivatives jobs are running inline. Have you setup a production job queuing system (you can check with: ActiveJob::Base.queue_adapter). See also: https://github.com/samvera/hyrax/wiki/Background-Workers. This is unrelated to the minter issue, but may be a serious issue in your production configuration besides.

[coblej]

@no-reply In case it helps ... I just encountered this error on my workstation in our Hyrax-based app. I attempted to create an object through the UI but, due to an error in some new actor code I had just written, an exception was thrown. I corrected the code error and tried again. This time, I got the Ldp::Conflict error. I opened a rails console and typed the command you listed above; namely, [::Noid::Rails::Service.new.minter.mint, ::Noid::Rails::Service.new.minter.mint]. I got ["5999n337h", "rf55z7713"]. (I actually then ran the command again and got two different id's, but I think that's probably irrelevant to the issue at hand.) I then restarted my rails server and again attempted to add a new work through the UI. That time, it succeeded. So, in response to the question you posed above, at least in this case, when the Ldp::Conflict occurred, I was able to successfully mint new ids outside of the actor stack and that seems to have caused the problem to go away.

[no-reply]

@coblej's experiment seems to support the conclusion that this is due to the db transaction rollback.

[wickr]

Had the same issue just now with local dev on ScholarsArchive/Hyrax 2.1.0. I was modifying our default indexer, triggered a code error, and then kept getting the LDP::Conflict on saves and creating new works. Running the NOID minter from the console gave me 2 values as well and resolved the LDP errors.

[straleyb]

We have been able to clear the problem by deleting the "broken" work through the interface. It shows up with no metadata.

But yes, we have been experiencing this for a while in Scholars Archive. Only when there is a rollback when a work fails to "create" fully. Per discussions from Slack, this has been a pain in the butt to track down, but hasnt effected our production instance yet. Only our staging instance once and plagued my development environment.

[revgum]

Can confirm [::Noid::Rails::Service.new.minter.mint, ::Noid::Rails::Service.new.minter.mint] causes the commit to happen and clears the issue. It returned two IDs.

[RudyOnRails]

Also confirming. This bug popped up today during Samvera Newcomers Call demo of customizing actor stack. Was able to rails console: [::Noid::Rails::Service.new.minter.mint, ::Noid::Rails::Service.new.minter.mint] and then ingest works again.

[elrayle]

To work around this and clear the error...

• Update samvera/ldp to show the URI when the error occurs. See PR samvera/ldp#98
• In command line, delete fedora object with the problem URI:
  curl -X DELETE <url>
• In command line, delete the fedora tombstone for the problem URI:
  curl -X DELETE <url>/fcr:tombstone

[jayanthdeejay]

We were getting this error and after deleting the fedora objects as mentioned above, I created a new work with an image file but the work view page after saving the work has 2 pdf files sets from another work appeared instead of the image uploaded. When I tried to open the filesets, the application breaks with "ArgumentError in Hyrax::FileSetsController#show Nil location provided. Can't build URI."

I deleted the work that I created above and tried to create a new work, the application breaks with Ldp::Conflict.

[no-reply]

@jayanthdeejay:

after deleting the fedora objects as mentioned above

Does this mean as discussed in https://github.com/samvera/hyrax/issues/3128#issuecomment-415067912?

Were you able to clear the issue with [::Noid::Rails::Service.new.minter.mint, ::Noid::Rails::Service.new.minter.mint] as confirmed in https://github.com/samvera/hyrax/issues/3128#issuecomment-409656956?

[jayanthdeejay]

@no-reply Tried both. Didn't work. We recently ingested about 1600 objects. I believe I still could use the script to ingest more but not with the create new work form. I may have to few weeks before I could test this as my manager will be presenting this sandbox soon with a ton of stakeholders. I'll get back to you if ingest works and the work creation from UI doesn't work.

Also, the #3198 comment works once. I delete the relevant object. Add new work, which will for some reason has filesets attached totally unrelated to the work that I just added. Cant delete that file set. If I try again to delete using 3198. This time I fail to create new work again. Something like. that. It was going in circles. create, can't create wrong file set and so on....

[vantuyls]

@no-reply do we have a way forward on this?

[fereira]

This is becoming a big issue for me. I am a co-developer for a knowledge management system for an organization in Kenya that is based on Samvera/Hyrax (2.1.0). There were some issues early on related to using the inline job queue but have since switched to Sidekiq. I made modification to the LDP gem which displays the LDP::Conflict error such that the resource which is causing the confict is displayed. The organization in Kenya recently hired a consultant to add content (there are about 200 documents already loaded) and he's seeing the LDP::Conflict error every day. Deleting the resource which is causing the conflict is not an option as they are legitimated resources that ingested correctly. I've seen conflicts for resources (when creating a new work) that are other other works, filesets, and even collections. Using the "minter" from rails console only temporily allows a new work to be added until another is attempted which has a conflict. I created a simple script which does the minter commands ([::Noid::Rails::Service.new.minter.mint, ::Noid::Rails::Service.new.minter.mint]) and I think the IT support person has run that liberally. Why is it running the minter.mint service twice? What happens if, say, someone executed several times? I think that the minter_states table me be "out of wack". What can be to get things back in sync? I haven't yet found in the code where the minter.mint service is being called when creating a new work but could it check to see if a resource already exists, and if so, mint another id until it has one that won't cause a conflict?

[fereira]

Digging a bit deeper into how Hyrax mints a new id for a resource I think I can see what is going on. Keeping in mind that my ruby coding skill are still quite marginal, from what I can tell, when creating a new resource (Collection, Work, FileSet) it uses the Hyrax::Noid class which uses the ::Noid::Rails::Service.new service. That gem, in lib/noid/rails/minter/base.rb has the following:

def mint
          Mutex.new.synchronize do
            loop do
              pid = next_id
              return pid unless identifier_in_use?(pid)
            end
          end
        end

That appears to check to see if a id is already in use (identifier_in_use), which has the following in lib/noid/rails/config.rb

def identifier_in_use
        @identifier_in_use = lambda do |_id|
          false
        end
      end

As far as I can tell, this will always return false, which will then use LDP with an id which does already exist. My ruby skills are not up to par to know how the behavior should be changed such that "indentifier_in_use" actually checks the minter_state (whether in a DB or file implementation) to return the correct value.

[elrayle]

@jcoyne Can you comment on identifier_in_use always returning false? Is that expected? Is it overwritten somewhere else to do a real check?

def identifier_in_use? https://github.com/samvera/noid-rails/blob/master/lib/noid/rails/minter/base.rb#L47-L49

def identifier_in_use https://github.com/samvera/noid-rails/blob/master/lib/noid/rails/config.rb#L25-L32

[jcoyne]

@elrayle the idea is that you can set a lambda in the configuration to answer that question for you.

for example:

config.identifier_in_use = lambda { |id| ActiveFedora::Base.exists?(id) }

[elrayle]

@jcoyne Is there any particular reason why there isn't a default noid config initializer in Hyrax generator code?

[jcoyne]

@elrayle I don't know of any reasons.

[elrayle]

At last, I believe we have a fix for the LDP Conflict and Gone problems. @fereira was right on with the problem being that identifier_in_use always returns false. @jcoyne was right on with what we need to get it to do instead.

Who is affected?

All Hyrax apps unless you took additional steps to configure noid ‘identifier_in_use’ lambda or use an id generator other than noid-rails

The FIX for current apps...

Add to your hyrax app... /config/initializers/noid_rails.rb

::Noid::Rails.config.identifier_in_use = lambda do |id|
  ActiveFedora::Base.exists?(id) || ActiveFedora::Base.gone?(id)
end

A better FIX for the future

There are two approaches we can take with this...

OPTION 1: Add the described fix to the README as a required step for hyrax to work out of the box with a link to other alternatives.

OPTION 2: Add the described fix to config/intializers/hyrax.rb as a default implementation using noid. If a site wants to use something other than noid, they would want to comment out or remove the code from the hyrax.rb initializer.

@no-reply @vantuyls How would you like to proceed?

[vantuyls]

my reading of this suggests that Option 2 is a good way forward. @no-reply @elrayle - what are the pros/cons of each of these options?

either way, is this something we can get into 3.0.0?

[elrayle]

@vantuyls I lean toward option 2. One less thing to manually have to do to get the out of the box Hyrax to work. I guess the question is how common is it for apps to use something other than noid-rails? The app is pretty heavily configured with the assumption that noid-rails is being used. So from my perspective, option 2 seems like the way to go.

@jcoyne @mjgiarlo Is there any history we should be taking into account for why Hyrax wasn't set up to check Fedora to begin with?

[vantuyls]

i'd suggest the following, unless @no-reply or the historians (@mjgiarlo @jcoyne ) chime in:
1) Modify the readme now as a stopgap
2) Implement Option2 for hyrax 3.0.0 and backport the change to the 2.x

[no-reply]

The FIX for current apps...

I'm not sure I would recommend this fix for current apps, unless they are actively experiencing issues, since it has potentially significant performance implementations for identifier minting (it can certainly be refactored to avoid sending two Fedora requests per id).

I still think it would be desirable to make this reproducible before recommending a particular implementation to anyone.

OPTION 1: Add the described fix to the README as a required step for hyrax to work out of the box with a link to other alternatives.

Can we take this option off the table? If we reach a place where this is the recommended Noid::Rails approach for Hyrax, we should include it in Hyrax and/or Noid::Rails by default and roll out a well tested solution to adopters using the normal release process.

OPTION 2: Add the described fix to config/intializers/hyrax.rb as a default implementation using noid. If a site wants to use something other than noid, they would want to comment out or remove the code from the hyrax.rb initializer.

If this approach is recommended, my preference would be to push it into Noid::Rails as a new default.

---

On the Hyrax side, the problem here still appears to be with ActorStack transactionality. If we can confirm this, a nicer fix (one that doesn't involve hitting Fedora on each NOID mint) might be to check and force update the minter state whenever the actor stack rolls back a database transaction.

[no-reply]

I would really like to confirm whether this problem occurs using the default Actor Stack, and whether it relates to ActiveJob queue adapter setup (https://github.com/samvera/hyrax/issues/3128#issuecomment-399562990).

Folks who are experiencing this issue could really help a lot by providing more details about their environments, and any Actor Stack customizations. It's challenging to diagnose this when we don't have a reliable way to recreate the conditions.

[fereira]

When I first encountered the issue, no changes had been made to the actor stack. At the time, the ActiveJob queue adapter was set to "inline" (the default out of the box implementation) and both @elrayle and I encountered the problem on our "dev" machines using the inline adapter. I have since migrated to use sidekiq for the job queue. I'm not sure if it will trigger the issue but I have encountered some files ingested that will fail in the CreateDerivatives job. These files are originally power point or word documents saved as a PDF. Apparently ImageMagick will fail on these documents, that I suspect that by the time that job runs that work will have been created correctly. I did have an occasion when redis stopped running, which caused sidekiq to fail so the CharacterizationJob or CreateDerivativesJobs didn't run.

To me, the bottom line is that the hyrax documentation indicates that the ID is tested to see if it has already been minted ( https://github.com/samvera/hyrax/wiki/Hyrax-Management-Guide#identifier-state ) and that's not true if the default implementation for identifier_in_use is always returning false.

[no-reply]

@fereira:

the bottom line is that the hyrax documentation indicates that the ID is tested to see if it has already been minted

Can you point me to where the documentation indicates this? As far as I understand, this is not at all how the minter is designed. The documentation reads in part:

The identifier minter is stateful, meaning that it keeps track of where it is in the sequence of minting identifiers so that the minter can be "replayed," for example in a disaster recovery scenario. (Read more about the technical details.) The state also means that the minter, once it has minted an identifier, will never mint it again so there's no risk of identifier collisions.

The point is that the minter state should prevent collisions, avoiding the cost of checking ID availability, (in rare cases, repeatedly).

In my understanding, the underlying issue isn't that we're failing to check availability (which is by-design), but that we're rolling back the minter state after a work has been created. Given the error state, that leaves us with two problems: an orphaned/unintended Work and an out of sync minter.

Checking the ID only addresses one of the problems, and in a way that seems counter to the minter design and serves to further hide the errored Work.

[jayanthdeejay]

We encountered this problem when were test ingesting in our development server. Our application is still the skeleton that hyrax engine produces with an added work type ETD. Application was running with fedora and solr wrappers with sidekiq and redis running, all default dev ports. I just test ingested few works backend and worked, but Ldp::Conflict in web browser. We created the default admin set and with default workflow loaded.

Currently on Hyrax 2.1.0. I am open to zoom in and share my screen if you'd like to see anything.

[bhavanaananda]

'hyrax' - '2.4.1'

I got the same error to begin with. Having read this thread, I did the following at the rails console: [::Noid::Rails::Service.new.minter.mint, ::Noid::Rails::Service.new.minter.mint]

I could then save the object, and when it returned to the short view record, the file wasn't listed under the Items section. It displayed the message 'This Work has no files associated with it. Click "edit" to add more files'

I then opened the record in edit mode and saw the "Add files" with a green tick indicating the record was 'valid'.

Why is the file just ingested not showing up? Any pointers please?

[no-reply]

@jayanthdeejay, @bhavanaananda:

Can either of you post the output of your ActiveJob::Base.queue_adapter?

Why is the file just ingested not showing up? Any pointers please?

File attachment happens in a background job. It's probable that some failure in the AttachFilesToWorkJob is causing both of your problems. Could you scan your log/development.log for errors related to Job classes and post anything you find here.

[no-reply]

The Hyrax WG is looking at this issue from two perspectives:

First, we want to determine what errors are causing the minter state rollback that gives rise to this issue? There may be a number of places in the Actor Stack where errors can occur after a work is created. Some of these seem to be due to incomplete development or production environment setup, and in these cases we need to get better warnings in place for developers (e.g., if files aren't attaching due to Image Magick not being installed, we should be able to provide runtime guidance in :development; likewise, startup warnings if :inline or :async is being used in production). Other bugs may be identified and prioritized for upcoming work. If you are experiencing this problem, please help us by posting configurations and logs that you think may be relevant.

Second, we need to patch the minter state bug described here. Since Actor Stack transaction rollback is partial, and it doesn't seem likely that getting real transactionality across the Relational DB, Solr, and Fedora is possible... this fix probably looks like some post-failure cleanup. Particularly: when the Actor Stack's ActiveRecord transaction rolls back, we need to fix the minter state and clean-up any orphaned works/FileSets. I'm working on this starting today, and expect that we'll release fixes on the 2.x and 3.x lines.

[jayanthdeejay]

@no-reply

We are running our sandbox with sidekiq

ActiveJob::Base.queue_adapter
=> #<ActiveJob::QueueAdapters::SidekiqAdapter:0x00007fdd403af348>

Some of the errors found in the log.

1. [ActiveJob] [CreateDerivativesJob] [31bec694-a7e4-4d92-a791-3894b5dca520] Error performing CreateDerivativesJob (Job ID: 31bec694-a7e4-4d92-a791-3894b5dca520) from Sidekiq(default) in 2892.81ms: MiniMagick::Invalid (`identify /tmp/mini_magick20181011-2833-1vivv3n.pdf` failed with error:
2. [ActiveJob] [CreateDerivativesJob] [95856dbf-971c-4a5e-992d-bfe33dcd10f0] Error performing CreateDerivativesJob (Job ID: 95856dbf-971c-4a5e-992d-bfe33dcd10f0) from Sidekiq(default) in 1509.98ms: MiniMagick::Invalid (`identify /tmp/mini_magick20181013-18604-1ddagm.png` failed with error:
3. [ActiveJob] [CreateDerivativesJob] [88bfe3a8-32b3-4426-b654-596dc7ca0165] Error performing CreateDerivativesJob (Job ID: 88bfe3a8-32b3-4426-b654-596dc7ca0165) from Sidekiq(default) in 27656.46ms: RuntimeError (Solr Extract service was unsuccessful. 'http://127.0.0.1:8983/solr/hydra-development/update/extract?extractOnly=true&wt=json&extractFormat=text' returned code 500 for [path_to_file]/wilson.pdf
4. [ActiveJob] [IngestJob] [20881da6-f5b5-46cc-a47b-d163bb911069] Error performing IngestJob (Job ID: 20881da6-f5b5-46cc-a47b-d163bb911069) from Sidekiq(default) in 4030.86ms: LoadError (Unable to autoload constant Hyrax::Actors::FileActor, expected [path]/.rvm/gems/ruby-2.5.1/bundler/gems/hyrax-cea87d99966c/app/actors/hyrax/actors/file_actor.rb to define it):
5. [ActiveJob] [CreateDerivativesJob] [1d3052d7-d365-48a8-986b-de3cfb706e1a] Error performing CreateDerivativesJob (Job ID: 1d3052d7-d365-48a8-986b-de3cfb706e1a) from Sidekiq(default) in 19415.33ms: Errno::EIO (Input/output error @ io_write - <STDERR>):
6. [ActiveJob] [CreateDerivativesJob] [316d85f7-a5b7-457b-a286-b5d4fcb8a7a9] Error performing CreateDerivativesJob (Job ID: 316d85f7-a5b7-457b-a286-b5d4fcb8a7a9) from Sidekiq(default) in 106907.46ms: MiniMagick::Error (`mogrify -resize 338x493 /tmp/mini_magick20181012-27851-168o9jw.pdf[0]` failed with error:
  Error: /undefinedfilename in (/tmp/magick-24288hAf4vASaMOiN)
  Operand stack:

  Execution stack:
     %interp_exit   .runexec2   --nostringval--   --nostringval--   --nostringval--   2   %stopped_push   --nostringval--   --nostringval--   --nostringval--   false   1   %stopped_push
  Dictionary stack:
     --dict:975/1684(ro)(G)--   --dict:0/20(G)--   --dict:78/200(L)--
  Current allocation mode is local
  Last OS error: No such file or directory
  GPL Ghostscript 9.25: Unrecoverable error, exit code 1
  ):
7. [ActiveJob] [InheritPermissionsJob] [3eeede0b-e768-4d10-8123-749806fa43ae] Error performing InheritPermissionsJob (Job ID: 3eeede0b-e768-4d10-8123-749806fa43ae) from Sidekiq(default) in 1959.71ms: Ldp::Gone (Discovered tombstone resource at /dev/a0/e2/7c/db/a0e27cdb-be6f-4e64-9dcc-3de9f07c0ed2/50/b4/d7/a6/50b4d7a6-1302-40aa-b6eb-73f1040b5d04, departed: 2018-10-15T16:01:37.964-04:00):
8. [ActiveJob] [AttachFilesToWorkJob] [902b6c87-d563-477f-86fc-12a5426b50d2] Error performing AttachFilesToWorkJob (Job ID: 902b6c87-d563-477f-86fc-12a5426b50d2) from Sidekiq(default) in 1607.61ms: Ldp::Conflict (Can't call create on an existing resource (http://127.0.0.1:8984/rest/dev/98/80/vq/96/9880vq965)):

All unique errors from the log. Let me know if you need complete log for any of the above errors.

[vantuyls]

I like the approach outlined by @no-reply - the WG is a good place to sort out any ??? about this issue. thanks to everyone here who has contributed to this discussion.

[fereira]

I can provide a full stack trace that I saved when encountering the LDP::Conflict about a week ago if that helps.

The errors that jayanthdeejay shows look similar to something I saw when trying to ingest certain documents. There is, apparently a problem with ImageMagick when it is used to identify the type of document if it was created using a MSOffice product using the "print as PDF" (or export as PDF) function. The "convert" command will generate an "invalid format" error. Note also that the IM convert command uses ghostscript for some file types. I had an issue with some pdf files when using version of ghostscript installed on AWS (Amazon Linux). After upgrading ghostscript to the latest version I was able to successfully characterize and created derivatives for some of the documents that had previously failed.

[jayanthdeejay]

Just encountered the following when I ran

$ rails ci

1) Create a GenericWork a logged in user should text "Please note, making something visible to the world (i.e. marking this as Public) may be viewed as publishing which could impact your ability to"
     Failure/Error: click_on('Save')

     Ldp::Conflict:
       Can't call create on an existing resource
.....
.....
# ./spec/features/create_generic_work_spec.rb:62:in `block (3 levels) in <top (required)>'

https://travis-ci.org/DigitalWPI/digitalwpi/builds/463014999

[orangewolf]

@no-reply we have encountered this problem in several scenarios - all of which are some version of "we started creating a work or collection, but the there was a failure". I think the big thing is that the minter can be rolled back and when it is that id is not cleared from Fedora. Here is a simple example of the problem:

5.times do
  ActiveRecord::Base.transaction do
    c = Collection.create(title: ["Some title #{rand}"], collection_type_gid: Hyrax::CollectionType.first.gid)
    raise ActiveRecord::Rollback
  end
end

This will fail on the second and all subsequent creates with the Ldp::Conflict (Can't call create on an existing resource) issue. So I don't think the bug is in the actor stack or in the queing mechanism, but in the minter itself. Minter should be a committing transaction and should not roll back if other parts do.

From reading here and looking around it seems like this is most common during imports. There will likely always be the occasional malformed record in an import, and so even though this is a error caused by another error, I think it is probably one worth fixing.

[jcoyne]

I like @orangewolf's suggestion of the minter having it's own transaction to wrap id generation.

[no-reply]

@orangewolf that all seems congruent with my understanding.

I've looked at adding a separate transaction for the minter; the thing is that it already has one: https://github.com/samvera/noid-rails/blob/master/lib/noid/rails/minter/db.rb#L58-L63

The Actor Stack's transaction wrapper creates a nested transaction. If we aim to keep the stack wrapped in a transaction, there are two possible behaviors:

• default: the minter's nested transaction is a part of the parent (actor stack) transaction. this is what is happening now.
• requires_new: true: the nested transaction is treated as a check point. if it the minter transaction fails, the actor stack transaction may still succeed without it. in this case, the nested transaction still does not succeed when the parent transaction fails. i have tested this with sqllite3.

As far as I can tell, there's just no way to make a sub-transaction succeed independent of its parent.

Increasingly, I'm convinced that making the stack transactional is just a bad idea. Unless we can neatly couple them with robust rollback of ActiveFedora changes, it seems better to let the database collect cruft along with Fedora/Solr. @orangewolf: if you have an in-development app you are currently working with, I might suggest you remove the TransactionalRequest actor and see if that introduces new problems for you.

The other issue is that we're still not making any progress on identifying the causes of the Stack failures. Within the stack, this issue should only happen when a Work has been successfully created, but some other failure has blown up the stack. This is really nasty behavior, however the transactionality is handled. This is why I'm eager to see reproducible errors. It would be best if we could identify the kinds of env states that give rise to this and reject them much earlier in the creation process.

[jayanthdeejay]

As mentioned before, we could create new works using the rake task that we have but not from the WebUI. We encountered this again in our sandbox. We also noticed that this issue pops up when a work creation fails halfway through.

[no-reply]

Another alternative would be to mint the ID outside the transaction, at the top of the stack, then pass it through the attributes. If the stack fails for any reason, the minter state will still have been updated. If a work was created, it will have that ID (oh, and we could log the ID for the error!), but otherwise that ID will just be dead.

This works to the extent that the stack doesn't create any other objects (collections, file sets, etc...) while it runs. I haven't looked to see how frequently this is true, but it might not be the best assumption for a customizable stack.

[bhavanaananda]

@no-reply sorry for the late reply.
I have got this to work somehow by reinstalling FITS. I simply can't replicate this again. I can now find all the files ingested in the short record view.

[fereira]

I ran into an issue with a later fits version that would characterize some files correctly and cause issues with the CreateDerivatives and Characterize jobs. Not that FITS uses several external applications such as ImageMagick and Ghostscript to read/convert files to create the thumbnails and getting a version of ImageMagick that handles all of the file types one might need is tricky (it generates no decode delegate for this image format). We're using FITS version 1.2.0, ImageMagick 6.7.8-9, and Ghostscript version 9.20. That said, I would assume that if FITS is failing it shouldn't cause any issues with the minter state or problems with a resource that has been created.

[orangewolf]

The other issue is that we're still not making any progress on identifying the causes of the Stack failures. Within the stack, this issue should only happen when a Work has been successfully created, but some other failure has blown up the stack. This is really nasty behavior, however the transactionality is handled. This is why I'm eager to see reproducible errors. It would be best if we could identify the kinds of env states that give rise to this and reject them much earlier in the creation process.

@no-reply wanted to speak to this part individually - It feels like everyone who is here describing this are indeed seeing "after fedora save but before the end of the stack" failure. The easiest way I have to reproduce it is the code above, but the best real world example is to break your fits install (which explains @bhavanaananda experience above). This causes the stack to throw fits! ;-)

Given that there are a lot of factors that go in to this stack that _could_ be wrong at any given time, I think that we have a couple possible paths forward (hoping for other suggestions though).

1) Remove the transaction entirely. We don't delete the object from Fedora, and it doesn't rollback, right? What about Solr? I know we don't kill any generated background jobs. That makes me feel like we're only really doing 1/4 transaction, and that seems pretty gnarly to me.

2) Remove transactions and implement manual transactions as a convention. We could change the stack to pass along an array of created objects and mandate that a) any actor that creates an object adds it to the array and b) that those objects handle a destroy method which does the right thing to clean it up.

3) A shorter term fix would be to just fix this particular problem and pass in the minter state as you suggested. Though I do think collections and file-sets are going to be issues here. Also compound objects. I like the idea of it as a quick fix, but I feel like it might be the beginning of a pretty slippery slope.

If it were just me all alone with nothing better to do, I'd probably test and release (1) and make an open ticket to implement (2). But I totally acknowledge that this world is better because I'm not usually left to my own devices.

[jayanthdeejay]

I'm assuming most people who encountered LDP::Conflict were running their application in development mode with SQLite as the database.

When creating new works, along with LDP::Conflict, I also encountered the following error.

ActiveRecord::StatementInvalid in Hyrax::EtdsController#create
SQLite3::BusyException: database is locked: UPDATE "minter_states" SET "counters" = ?, "seq" = ?, "rand" = ?, "updated_at" = ? WHERE "minter_states"."id" = ?

At least, in our case, we were ingesting a lot of works using rake tasks and may be during one of these work creation, the database lock may have started this issue and probably is the root cause.

[no-reply]

@orangewolf:

I'm absolutely in favor of (1).

My only hesitation is that I don't think our automated testing is robust enough to spot new problems that could be introduced by removing the transaction actor. As mentioned in https://github.com/samvera/hyrax/issues/3128#issuecomment-451687145, I'm hoping someone encountering this problem on an in development app can try removing the TransactionalRequest actor locally. If they can rerun the kinds of processes that trigger this problem and see improved behavior, that would help my confidence that this fix improves things.

The following in a config.to_prepare in application.rb should remove the actor.

Hyrax::CurationConcern.actor_factory.delete Hyrax::Actors::TransactionalRequest

For me, at this point this is the preferred fix for any app encountering this that doesn't yet have production data. I'd love to see what, if any, problems folks see with this change in place.

[no-reply]

@bhavanaananda, @fereira, @orangewolf:

It seems odd to me that FITS, CreateDerivatives or Characterize jobs would cause this problem unless the environment is running those jobs inline. Is there more info you can share about what you're seeing with a bad FITS install? It seems worth tracking this down.

[fereira]

I would not think so either, and on the production site which I was encountering the issue, it's now running Sidekiq and since we implemented something for the lambda which tests for existing identifiers I have not had a LDP conflict since.

I ran into some issues (I don't recall exactly what they were) using FITS version 1.3.0 and had to revert back to version 1.2. I have encountered issues with some file types that won't identify (e.g. some data files such as SPSS files) which results in the generic blue icon rather than a thumbnail but the documents still seem to deposit correctly.

[no-reply]

I have two proposed fixes.

I think #3482 is desirable for master, and #3481 may be suitable for 2.x

[orangewolf]

Fun fact - this also appears to clear up an issue with minters taking forever to get their lock - since in postgresql the lock stays open until the transaction is over. we've seen a big improvement with multiple workers with this fix.