Darktable: Broken multi-image copy/paste workflow with recent master (ignores wb, lens)

[johnny-bit]

Describe the bug

On recent master I tried to process a studio session of jewellery shot. The shot was done with constant light and same lens. Unfortunately I had WB set to auto, but I did shot color checker frame. I processed color checker frame and then wanted to apply settings to all 100+ shots using normal copy all, paste all workflow. It seems that WB settings as well as lens corrections were ignored.

To Reproduce

1. Have 100+ shots done with same lighting condition and same lens
2. Process first one
3. Go to lighttable, select processed file, press ctrl+c (copy all)
4. using "select" module invert selection so that all other shots are selected
5. paste all
6. see that neither WB nor lens corrections are applied

Expected behavior

WB and lens correction should be applied. All other corrections (exposure, filmic rgb...) did apply correcty :/

Platform (please complete the following information):

• Darktable Version: 3.3.0+1130~g96726f7cc
• OS: Gentoo Linux
• OpenCL activated: yes
• NVidia GeForce1060, driver 455.23.04

Additional context

• Not reproducible on 3.2.1
• Didn't try with JPEGs
• All edits were fresh
• darkble compiled using ./build.sh --enable-opencl --disable-game --install --build-type RelWithDebInfo
• The issue still present using an empty/new config-dir

[junkyardsparkle]

You missed the discussion in #6520. :-)

I tried to suggest there that this was going to surprise people, because I use that workflow all the time myself. Ultimately it comes down to who gets forced to use selective copy/paste more often, and I don't mind adapting, but... I have a feeling that changing something that's been in use this long is going to generate more issues than it prevents... and that feeling just got stronger.

[parafin]

Maybe lensfun and wb iops should be removed from the unsafe list. Default camera WB, I think, can be safely copied without changing anything, and changed WB probably should be copied. Not copying it by default won't save us from people trying to copy wb to pictures from another camera vendor. Rather unsafe list should prevent from accidental corrupting of the images like with rawprepare settings. On the other hand I still think colorin should be in the unsafe list.

Also, there seems to be another missing detail with that change - copy/paste selected should unselect unsafe modules by default, I think it wasn't implemented.

[johnny-bit]

I went over some of #6520 and I do think the ONLY modules that shouldn't get picked up by "copy all/paste all" are ones intristincly tied to image itself with no transferable settings, like rotatepixels.

And I do agree that copy parts dialog is now counter intuitive - it has ALL modules and order selected by default while by default it should have selected only the "safe" ones.

[elstoc]

If modules are truly unsafe to copy they should be removed entirely from the "copy parts" and "paste parts" dialogs and it should not be possible to copy/paste them at all. White balance and lens correction should not be included in this list. I agree with @johnny-bit that this should only include modules whose settings cannot be altered. If you want to include the other modules then some paramaterisation is necessary to allow people to override.

Currently the only way to copy and paste all modules is by using the "copy parts" and "paste parts" button which is entirely counter-intuitive.

[jenshannoschwalm]

Let me counter-advocate here :-)

The copy&paste all should only include modules with a safe bet they work on all images it gets pasted to.

There have been quite some issues exactly about white-balance and not giving expected results.
This can't be fixed atm as the module just knows parameters and knows how to apply that parameter matrix to any other image but has no idea whether that is good or total crap. The users probably know -- so if they know it's safe they should be encouraged to use copy&paste in selected mode.

The lens is not much better in my opinion, some of us might use a preset to switch on a lens. Right? Do you only have one lens? If you paste the lens parameters from such a preset it copies the lens data and your ff nikon images might get an olympus12-40mm lens applied as an example. Quiet strange and surely not expected.

So i would not call it counter-intuitive but for me a desired behaviour.

Just arguments for my personal opinion. BTW this resulted in dt before this to never use the copy&paste all exept in situations like multi-shots.

[johnny-bit]

Let me counter your counter :P

I only copy all/paste all on multi-shots. All other - i use copy parts. Now Copy parts by default has everything selected, while copy all doesn't copy all. Not only counter-intuitive but totally contrary to name ;)

There's additional problem with logic of it: If I press copy all, and then press paste parts - the paste dialog has selected all modules and indeed pastes all modules. but paste all does not. if i do copy parts (it opens dialog with all modules selected) and then paste all, it pastes all.

So copy all/paste all logic is bass-ackwards and modules are not being pasted only if i use copy all/paste all workflow, which i'd say is pretty weird logic.

[elstoc]

The way I see this working is that you have two sets of modules

• always unsafe
• sometimes unsafe

The first set should never be copied or pasted and should not be shown in the copy/paste dialogs. The second set should be enabled or disabled by default in the preferences.

If you select "copy all" with the "copy sometimes unsafe modules" disabled, this automatically deselects those modules (as if you'd selected "copy" and deselected them). If you select "copy" with "copy sometimes unsafe modules" disabled, the dialog appears with those modules deselected by default.

The "paste" and "paste all" buttons should operate as currently.

[jenshannoschwalm]

I only copy all/paste all on multi-shot

Yeah, i did that too :-) I guess groups won't help here. Config option - i think we already have much too many.

I don't know what is the "right" way.

[elstoc]

Two new buttons "auto-copy" and "auto-paste" which are auto-magic for noobs

[johnny-bit]

I think one more option didn't hurt anybody ;)

A way that would "fix" the problem for me without addind option described by @elstoc would be to simply:

• copy all - copies including "sometimes unsafe" modules
• copy parts - should open with default deselected "sometimes unsafe" modules.

that's all. This won't break workflows for those who know what they're doing :)

"safe copy"/"safe paste" buttons/actions which copy only "always safe" modules is 3rd option ;)

[parafin]

I find current copy-paste system unintuitive (and it's unrelated to unsafe iop list). I think there should be 3 buttons instead of current 4:

1. copy (without dialog)
2. filter (dialog where you select the iops to transfer)
3. paste (without dialog)

The current situation where iop selection is bundled with copy and paste operations results in complexity and non-transparent behaviour without adding anything.

Now, filter button in the proposed scheme is where all the magic happens. It should display its status (all selected, only safe, custom selection), and we can have presets for it. By default it should be only safe modules (whatever the list might be), and reset action should reset it to that state. Not sure if its state should be saved across dt restarts, probably not, presets should be used instead. I think it will fix all use-cases without the need for yet another preference option (really, you gotta stop adding them).

[elstoc]

Don't forget this needs to work with keyboard shortcuts in other views as well.

[junkyardsparkle]

Don't forget this needs to work with keyboard shortcuts in other views as well.

Yeah, let's think in terms of "actions" instead of "buttons"... that said, I kind of like the idea of having an additonal "copy/paste safe" action rather than a preference toggle. I think we've established here that both "safe" and "unsafe" versions are useful under different circumstances. Then it would just be a debate about which ones get the familiar key bindings by default. ;-)

[parafin]

I don't see any problem with shortcuts in my proposal. What is problematic is that indeed in darkroom it's possible to copy-paste on filmstrip, but "filter" button wouldn't be visible to indicate what is being pasted. Something to think about if people want to go this way.

[junkyardsparkle]

Maybe the "copy/paste selected" dialogs could have tabs for configuring the behavior of the "quick" modes?

[TurboGit]

I think there should be 3 buttons instead of current 4:

Not sure, one of my workflow is:

• copy all
• select some pictures
• paste parts
• select some other pictures
• paste parts (not the same iop than above)

The other is:

• copy parts
• select some pictures
• paste all

I never use copy-all / paste-all combination for sure. This may change with new way of copying only safe modules done recently.

[TurboGit]

The copy&paste all should only include modules with a safe bet they work on all images it gets pasted to.

I strongly agree with @jenshannoschwalm here.

[TurboGit]

I only copy all/paste all on multi-shots. All other - i use copy parts. Now Copy parts by default has everything selected, while copy all doesn't copy all. Not only counter-intuitive but totally contrary to name ;)

Agreed ! We should at least rename the buttons to remove the "all" as it is not true anymore.

[trougnouf]

I understand copy-all as copy anything that's been done manually (as well as anything auto-applied), that definitely includes the white balance iff it was modified from the camera defaults.

I expect the crop and perspective correction to be part of it too then, but I can see how that would be counter-productive (few use-cases) so I can appreciate those not being included (although even that may have thrown me off when applying copy-all to a duplicate).

However the modified white balance settings would be part of a set of images shot in the same conditions, which is the prime use-case of copy-all. A reasonable use-case where this behavior throws users off is applying the same settings to a set of images for export to a stitching program s.a. hugin, where the user needs to set a static white balance and the other images would then have subtle differences if that modified (but not really) white balance is not carried over.

[TurboGit]

Yes, thinking more about this and I see that white-balance module could be part of copy.

But on the other end one could say: do copy-parts, select white-balance and that's it. One more button...

Anyway, I'm ready to revert to copying white-balance if more people agree.

@jenshannoschwalm @elstoc @junkyardsparkle : would that be a possible compromise for you too?

@parafin , @johnny-bit : should be ok given your comments, right ?

@trougnouf : you'll be happy I suppose!

[elstoc]

I'm a bit confused about what the end result of all this conversation would be. @TurboGit could you do a quick summary including the final functionality from the original PR. Or is the only proposal here to remove white balance from the list of unsafe modules?

[TurboGit]

Or is the only proposal here to remove white balance from the list of unsafe modules?

Yes, only that for now as it is the issue that has raised this discussion.

[elstoc]

Then it probably addresses the issue in the OP. Should the other discussions be moved to a separate issue?

[johnny-bit]

I meant white balance and lens correction as both of those are "safe if you know what you're doing" :) plus for lens correction I rarelly ever fine-tune settings, just blank "enable" them, so it's imo safe even to add since no this way lens coirrection will simply be "automagic".

[TurboGit]

Then it probably addresses the issue in the OP. Should the other discussions be moved to a separate issue?

Yes.

[jenshannoschwalm]

I can live with all solutions, it's just about what is best for all. Newbies included

[junkyardsparkle]

I can live with all solutions, it's just about what is best for all. Newbies included

Same here. My concerns have more to do with the fact that removing certain modules from "copy/paste all" without any other visible changes might cause a lot of headaches to people who have come to expect things to work the way they do now. This would include any kind of stacking/stitching/sequencing workflow, not uncommon at all but possibly not something that everybody thinks about.

So, if there's interest in working on copy/paste to optimize for both "safe for newbies" and "efficient for productivity", I would propose a new discussion based on renaming "all" to something like "quick", and allowing the user to make some configuration to how it works (maybe in the copy/paste dialog, to avoid cluttering preferences). I do still like the idea of having two "quick" modes with their own shortcuts, but that might be more than is really needed. :-)

[parafin]

I agree that WB should be removed from unsafe list (and maybe lensfun?). I also like the idea to rename "all" to "quick". I will create a separate feature request with my proposal to redesign the c&p interface.

[TurboGit]

I'm not found of "quick", simply removing "all" to just have "copy" ; "paste" seems enough to me.

I agree that in "copy..." with dialog it would be nice to un-select unsafe modules too. I'll propose a PR.

[TurboGit]

See #6587

[TurboGit]

Closing as done & merged.