Fingerprint and Jaccard Comparison
MagPunk reduces each title to a set of phrases and compares these sets with the Jaccard ratio to find items telling the same event.
The title is first simplified: converted to lowercase, apostrophes and suffixes are deleted, punctuation is discarded (Turkish letters are preserved), standalone numbers and words shorter than three letters are eliminated. The root of the remaining words is taken; stopwords and roots shorter than four letters are weeded out. One, two, and three-word phrases are extracted from what remains. This set of phrases is the fingerprint of the title.
Example: If the Jaccard ratio is 50% and above, two items are considered reference, if it is between 30-49% they are considered similar, if it is below 30% they are considered unrelated.
Go deeper
The comparison is not made without satisfying four mandatory conditions: the language of the two items must be the same, their source different, their link different, and the age difference between them must be less than 24 hours. These conditions keep the repetition of the same source’s own content and random matches across days out of the pool.
Instead of comparing every title with every title, an inverted index is used: only pairs carrying at least one common phrase become candidates. The cost drops to around the O(N·k) level instead of O(N²).
Both thresholds can also be changed from Settings. The only constraint is this: the reference threshold cannot be lowered below the similar threshold — if it could be lowered, the “similar” range would close, and the two labels would be identical.
The reference label is not an accusation of plagiarism. What is measured is the word overlap of two titles; two publications using the same agency bulletin also receive a high ratio. The label allows you to see in how many sources and in what order an event appeared; it does not pass judgment on the source.