Deduplication not working as expected. How to clean the library now?

[Vogeluff]

So, I have about 30k images/videos in my library. Due to bad backup management i have lots of duplicates (whatsapp compressed versions, originals, google photos versions, etc.). The deduplication found a couple thousand and I'm glad about this but even after I mostly removed those I still have very obvious duplicates (resized/compressed) in my library. And not just a couple, it's a regular thing.

Im using ViT-SO400M-16-SigLIP2-384__webli for the smart search,
and 0,01 as the threshhold in the duplicate detection.

as far as i understand those are the 2 things that can impact the duplicate detection.
I don't think the treshhold is bad as it also detected lots of not identical images in its scans which I manually unselected. so increasing its threshhold would just flood the results with false positives.

I tried Immich duplicator finder Version: v0.1.3 as an external tool thinking that it maybe uses a different approach but it doesnbt find any more duplicates aswell even after playing around with the thresholds. currently im back to stock with 0,00 min, 0,60 max and 10 pairs to display but nothing shows up.

i dont wanna us general image duplication scanners on the media folder itself as i dont want to fuck with the database or anything but would that be an option? does immich handle that well? or what else can i do other than manually going through everything?

thanks for any tips and ideas

[schuhbacca]

You can't directly modify anything in the upload location. So even if you pointed some third party duplicate tool at the folder, it shouldn't touch any of the media in there.

If you are very worried about accidental media deletion from the Duplicate Detector and have a lot this is probably what I would do.

1. Set a very small threshold, starting at .001. Run the detection job. See how many assets it finds. Go through them and maybe use the "All"
2. Bump up the threshold after resolving those.
3. Repeat until you are no longer using the "All" function and then you will need to review manually.

[schuhbacca]

I have never had an issue with false positives being detected and "close" items not. I have a setting of .004 and I find that works quite well. You may need to post some examples as I have just never seen that behavior.

[Vogeluff]

It got better once I deleted all .dng and other raw files. it seems those confused the algorith a lot? it often suggested me duplicates between 2 .dngs rather than between the jpg and its corresponding .dng for example aswell. i reran the duplicate finder and now there are other 2000 new results. didnt change other settings, only mass deleted .dng and .arw and some other file format i cant recall right now.

[Vogeluff]

@schuhbacca

ive got a new interesting find just now. it basically says "well these 1600 images all have a black cat so they are the same". wtf

[bo0tzz]

That really looks like your threshold is set too wide.

[Vogeluff]

That really looks like your threshold is set too wide.

i set it to 0.06 as in testing lower thresholds resulted in obvious duplicates (same image with different compression) not being detected. i think the ai based logic should be using something else to keep itself in check. for example it could downscale the images it found to maybe something like 50x50 px and xompare those images. if they differ too much skip. just an idea of how to detect images that are visually the same but technically very different.