They call it the CSI effect — “the phenomenon of popular television shows such as the CSI franchise raising crime victims’ and jury members’ real-world expectations of forensic science, especially crime scene investigation and DNA testing.” But today, the Los Angeles Times reports that the FBI’s real-world rarity estimates of DNA matches may be unrealistic as well:
State crime lab analyst Kathryn Troyer was running tests on Arizona’s DNA database when she stumbled across two felons with remarkably similar genetic profiles.
The men matched at nine of the 13 locations on chromosomes, or loci, commonly used to distinguish people.
The FBI estimated the odds of unrelated people sharing those genetic markers to be as remote as 1 in 113 billion. But the mug shots of the two felons suggested that they were not related: One was black, the other white. […]
No one knows precisely how rare DNA profiles are. The odds presented in court are the FBI’s best estimates.
The article is basically about how Troyer’s discovery in 2001 has led to several other states — including Illinois — querying their DNA databases to see how unique their samples are. And that has led the FBI to try to stop those states from doing these database queries. The FBI says the states’ results are misleading; the states say that the FBI is trying to hide evidence that their rarity estimates are flawed. For example:
In July 2006, after Chicago-area defense attorneys sought a database search on behalf of a murder suspect, the FBI’s Callaghan held a telephone conference with Illinois crime lab officials.
The topic was “how to fight this,” according to lab officials’ summary of the conversation, which later became part of the court record. […]
A week later, the judge ordered the search. Lawyers for the lab then took the matter to the Illinois Supreme Court, arguing in part that Illinois could lose its access to the federal DNA database. The high court refused to block the search.
The result: 903 pairs of profiles matching at nine or more loci in a database of about 220,000. [Emphasis added]
State officials obtained a court order to prevent distribution of the results. The Times obtained them from a scientist who works closely with the FBI.
Recall from the earlier quote that the FBI’s odds of two people sharing the same nine out of 13 loci on their DNA is 1 in 113 billion. Yet, when the database of only 220,000 samples was queried, “903 pairs of profiles matching at nine or more loci” were found. That’s 1 out of 244.
Statisticians have many explanations for this phenomenon, and the FBI stands by their estimates. Bruce Weir from the University of Washington offers this explanation (PDF file). On the other hand, he’s also quoted in the article as saying “these assumptions should be tested empirically in the national database system. ‘Instead of saying we predict there will be a match, let’s open it up and look.'”
That sounds like a reasonable request, since so much weight is given to DNA evidence. We want to be sure that it’s as reliable as possible, and rarity estimates neither understated nor overstated.