How to Deduplicate and Clean Large Datasets Using AI Decision Models

Load your list of records (e.g., company names, user accounts) that you suspect contain duplicates. This can be a large JSON file, a CSV, or a database query result.

Feed the list of records to Jev. Configure it to compare entries and identify potential duplicates, such as 'Cedar Grove Office Products' vs 'Cedar Grove Office'.

For each potential match identified, Jev should return a confidence score indicating the likelihood that the two records are duplicates. This score is key to automating the next step.

Define a confidence score threshold to determine what to do with the matches. For example, evaluate a 99% confidence threshold against labeled examples before enabling automatic merges; keep a backup and review uncertain matches.

For pairs with a lower confidence score (e.g., between 75% and 99%), flag them for manual review by a human operator. This creates a human-in-the-loop system that balances automation with accuracy.

For the pairs flagged for review, pass them to a cheap generative model (such as one accessed through OpenRouter). Prompt the model to explain why it thinks the pair is a match, providing a qualitative reason to assist the human reviewer.