Latex-workshop: Add functionality to remove duplicated in a bib file

[drdebmath]

Is your feature request related to a problem? Please describe.
It is always a hassle to check for duplicates when merging two bib files.

Describe the solution you'd like
Option to delete the entries having the same bib keys.

Describe alternatives you've considered
JabRef has some visual guides and functionalities. But it is preferable to have it in the LatexWorkshop menu.

Additional context
Does this feature already come with biber? Please let me know if there are simpler solutions.

[trevorgunn]

I'm thinking we can tack this onto the current bibtex sorting function and just flag all the bibtex entries that compare as equal. Then create a new setting for whether to warn about duplicates, warn and remove, or ignore with warning being the default.

One possible issue is that the bibtex fields a user wants to sort by may not be the same as the fields they want to check for duplicates by.

I do want to keep the bibtex functionality in LW simple, so I think this is a reasonable solution.

[drdebmath]

Bibtex only warns for duplicates only if the key is the same, otherwise, it treats them as two different entries. The main use case I am troubled with is that when I copy references from one bib file to another bib file, there would be some references that are already present in both and create duplicates for the same entry. Since I am using a common source for bibliographies like dblp, all the duplicate entries are exactly the same unless I have specifically removed/modified some fields.

As per @trevorgunn's suggestion, one option is to just remove duplicates based on the key field, since that gives the error when running BibTex.
The other option should be to flag and allow the user to compare and decide which to keep.

For sorting the BibTex entries, both title and author name should be options apart from keys, because some people create keys manually which doesn't follow any standards.

Another suboption I am thinking of is to remove particular fields from all entries, such as URL or crossref. Shall I make a separate issue for this?

[trevorgunn]

For sorting the BibTex entries, both title and author name should be options apart from keys, because some people create keys manually which doesn't follow any standards.

We have this already.

Another suboption I am thinking of is to remove particular fields from all entries, such as URL or crossref. Shall I make a separate issue for this?

This is beyond what I'm interested in implementing. Which fields are relevant depends on the entry type and the bibliography style file. Stripping unnecessary fields is best done by a program that is aware of the style file such as biber.

Look into biber's tool mode if you haven't already.

[drdebmath]

We have this already.

Thanks, I will look into this. Perhaps add a menu option in the miscellaneous section of LatexWorkshop for better accessibility(?).

Look into biber's tool mode if you haven't already.

I looked into it briefly. Many answers in stackoverflow suggest writing specific scripts for removing fields. I just do regex matching to remove quickly.
Thanks for your suggestions.