Entity Resolution Using Sentence Embeddings and a Small LLM

When two shops or two libraries merge their catalogues, the same product or paper appears under different names ("Sony Turntable - PSLX350H" vs "Sony PS-LX350H Belt-Drive Turntable"). This project finds such duplicates in two steps - cheap blocking with TF-IDF and all-MiniLM-L6-v2 sentence embeddings, then a matcher - and compares classical record linkage (TF-IDF threshold, Fellegi-Sunter in Splink) with a learned matcher and a small local LLM judge (qwen2.5:3b through Ollama) on three public benchmarks, split by entity so that no record is in two splits. The learned matcher reaches test F1 0.955 on Abt-Buy, 0.722 on Amazon-Google and 0.987 on DBLP-ACM, far above TF-IDF (0.707 / 0.626) and Splink (0.711 / 0.442) on the products. MiniLM helps blocking, not the matcher, and the small LLM judge did not improve test F1. A web application matches new records live and gives a data steward a review queue.

Category: Information Science and Technology

Information Science & Technology Project IST08 from ProTechSC Academy, with graphical abstract, video demonstration and project guidance.