[Feature] Duplicate Rules

[johnny2678]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Great job with the duplicate feature.

I have 75,000 dupes! So while I click away, I have had some time to think about how the "recommendations" might be more finely tuned.

ex. If photo resolution is the same and a HEIC and JPG show as dupes, immich usually recommends the JPG as the file size as bigger. However, I prefer the HEIC because of the compression and the live photo

another one - if an item exists in an external library and in the immich library and the resolutions are the same, I prefer the external library version.

I'm sure others have different opinions, but that's why it would be nice to be able to define our own rules.

Thanks for the project!

Platform

• Server
• Web
• Mobile

[marciopamplona]

Another very useful feature is enable/disable video dedupe. I have a lot of false positives due to the AI model.
It could be very useful to have a different configuration for detecting videos with a more strict rule (different maximum distance).

[radh21301]

Here are some more points to think about. It would be great if we could set a list of rules and prioritize them. And each user can have a different priority - basically, an option to reorder them as needed.

The following are some rules I could think of:

1. GPS Available (Yes/No)
2. Exif Tag Available (Yes/No).
   • DateTimeOriginal:
     • I have this criterion because there could be pictures with the metadata stripped, say a WhatsApp image. These images that don't have the metadata will most certainly show up in an odd location in the timeline. I always prefer images with the right metadata.
   • CreateDate - This is for videos
   • Starred
   • The user could choose any tag from a certain pool of most commonly used exif tags
3. Image/Video size (Small/Large)
4. Prefer external library OR immich internal library
5. Prefer Immich Favorited Asset

The list can be refined based on what people think.

[premiumcola]

This would be awesome :)! thanks for your detailled thoughts on this topic

[Rundervinkje]

I'd like to add-on to your list with:

1. Duplicates (same name). Like D001_img2.jpg and D001_img2.jpg in a different folder.
2. Duplicates (partly same name). Meaning, it could even account for partial matches, like recognizing D001_img2.jpg and attachment_D001_img2.jpg as the same image despite slight differences in the name.
3. Duplicates per Maximum Detection Distance (MDD). Meaning, run the duplicate search job with different MDD's and sift through them at a later date.

[tairasu]

this is fairly easy to implement since the suggestDuplicateByFileSize object is just sorting each iteration by file size and selects the last object (biggest file). instead you could first check if there is one or more HEIC file and then ignore JPGs.

obviously this should be optional. and it would need a UI for user preferences.

[Ccheese4]

Also been thinking about this as I go through my 13,000 duplicates. Would at least reduce my clicks so I don't have to switch the heic and the jpg auto selection.

[Ccheese4]

also the ability to skip a duplicate pairing would be nice and come back to it later.

[chrissi5120]

I effed up big time and uploaded from my wifes "recent" folder which seems to be populated by google photos?

We now got like 8k photos and video as duplicates with/without GPS.

being able to select by GPS would save me tears.

[githubsean]

What I would like is if some information from the deleted duplicate is copied over to the kept duplicate. A good example is folders.

Say there are two photos. One is low resolution (from Google Takeout), one is high resolution (original manually copied off phone). The low resolution version is in an album, the high resolution one is not in any album. The duplicate checker will correctly suggest to keep the high resolution version, but the effect is to remove this photo from the album the low resolution version was in.

Similar issues might happen with EXIF dates or GPS coordinates - the photo being discarded might have this detail, but the kept photo might not (though given the example above, the "original" will likely have all this). My main issue is with keeping album information.

[njspix]

Wanted to bump this - file size is a decent criteria but might end up selecting for poor compression algorithms (JPEG instead of HEIC, for example) rather than actually more data. For a preliminary go, how about sorting by (decompressed) pixel count rather than size? I believe it's possible to get the 'real' pixel dimensions of HEIF/HEIC and JPEG images without decoding them.

[JasonMAnderson]

A dilemma I run into is camera backup from my phone. I want my photos from my phone immediately available, so I turn-on camera backup. But I also manually copy my camera photos to my external library every month because my external library is my "single source of truth" as it's accessible outside of Immich. I need a rule to always delete the duplicate in the camera backup if a dupe exists in external library.

[premiumcola]

Yeaaah thanks for the comment on this aspect:
I would also like to set my "external libary source" as first priority - and keep all fotos once in there but delete the duplicates on all other sources.

[rwlove]

Prefer to keep photos already in an album.