One of the features that help maintain an accurate EMP database is the deduplication service. The deduplication service scans and compares every record in the database against every other record in the database in an attempt to find records that are duplicates. To access the deduplication service and listing of potential duplicates, click on Duplicate List under the Students tab on the navigation menu. The Duplicate List page will display all currently found duplicates. Duplicate pairs are only added to this listing when the duplicating service is run or a pair is manually added.

To run the deduplication service, click on the on blue ‘Run Deduplication’ button located in the top right corner of the Duplicate List page. Once initiated the deduplication service will run in the background and send notifications once done (to those users who have registered to receive them). The page will need to be reloaded to see the results of running the deduplication service.

To find potential pairs the deduplication service uses several proprietary algorithms. The following fields are used in those algorithms to match records: First Name, Middle Name, Last Name, Address1, City, State, Zip, Email, Entry Year, Gender, Date of Birth. Not all of these fields need to be present, but for the most accurate results all of these fields should be present for all records.

If a pair of records are found that are duplicates but not picked up through the deduplication service automatically, they may be manually submitted as a duplicate pair to be resolved. To do so enter the two records’ EMP ID into the two boxes under the ‘Manually Enter Duplicates’ label. Once both EMP IDs are entered, click ‘Submit Dupes’. This will add the two records as a matched pair below where the merge tool can be used. A records EMP ID can be found in the General Fields tab of the Information section on the student record page.

Underneath the ‘Manually Enter Duplicates’ area, a listing of all previously found or added duplicates will be shown. First Name, Last Name, Email, Stage, Rep and Rating will be shown for each pair. There are two actions that can be taken to resolve potential duplicates. If the records look like they could be the same, click the ‘Resolve’ button. Clicking this button will open the merging interface and show a side by side comparison of all of the data on these records. All fields that have different data will be highlighted red. After examination, if the records are determined not to be duplicates close the merge tool and click the Ignore button. Ignoring a duplicate pair will cause these two records to never be marked as possible dupes again. If the two records are duplicates, then the merge tool can be used to pick and choose which information to merge. When two records are merged, only one set of data can exist. Each field has a button next to it, and the selected data will be kept. Once merged, the record associated with the EMP ID not selected will be deleted. The remaining record will retain all of the interactions plus those of the deleted record. Please note that the deleted PURL will no longer work.