An overwhelming 84 percent of the stops in the first three-quarters of 2009 were of black or Hispanic New Yorkers. It is incredible how few of the stops yielded any law enforcement benefit. Contraband, which usually means drugs, was found in only 1.6 percent of the stops of black New Yorkers. For Hispanics, it was just 1.5 percent. For whites, who are stopped far less frequently, contraband was found 2.2 percent of the time.In this analysis lies a fundamental mistake. We have not been provided with the correct statistic to determine whether the department is actually racist. Suppose, as in the figure below (click to enlarge), that minority members of the population and majority members of the population factually have a different distribution of "suspiciousness."
For example, the x-axis could represent the level of "suspicious dress" (let's suppose that this is not endogenous). The top chart represents the distribution of the majority population, while the bottom chart represents the distribution of the minority population. In this example, there is little signaling differentiation among the minority population. However, there is a great deal among the majority population. In this sense, although the population averages are the same, and although the "level of suspiciousness" trigger is the same, a much larger proportion of the minority population appears suspicious.
If suspiciousness is correlated with commission of crimes, the New York Police would not be discriminating by stopping more black men. Moreover, if suspiciousness in the majority population is so rare that it strongly signals criminal behavior, we would expect a larger proportion of police stops of majorities to lead to arrest.
Ultimately, what determines discrimination is not the average number of police stops that lead to arrest of minorities vs. majorities. The proportion of marginal stops that lead to arrest , which we do not observe, determine the level of discrimination on the police force. If the marginal (last) stop of a minority is less likely to lead to arrest than the marginal (last) stop of a majority, then the police department would be discriminating by stopping the minority. The only way the statistics we are given could actually lead us to the conclusion that the New York Police Department is racist is if they were the results of marginal, not average stops. This is not the case; the Times is mistaken.
I wonder if anyone has done the following meta-study.
ReplyDeleteSometimes an average or a percentage will (simplistically) make it look like we're discriminating; simple statistics almost always look like they say more than they do, in one direction or another. But is there a reason to suspect that simple statistics will systematically _tend_ to make it look like we're discriminating against a particular group? If you interpret a whole collection of simple statistics naively, do you expect to find a bias?
There are several possibilities. First, perhaps there is no systematic bias, but the media latches on to the simple stats that "point to" discrimination. (Note we don't even need to argue naivety to point out that this selection bias taints ALL media coverage of such stats). Second, if there is no inherent bias in simple stats, then we either will or won't find evidence of discrimination based on the actual stats we collect: yes, this stat you report isn't enough to say the police are discriminating, but if most similar stats "suggest" discrimination when they really shouldn't tend to be doing so, then perhaps we can conclude that discrimination is taking place. Third, if it happens that simple stats tend to bias us toward the appearance of discrimination, it would be interesting to ask why that may be.
I guess this would be answering a slightly different question than simply whether there's discrimination. To calibrate our expectations we could use real, sophisticated stats. There's some circularity and some question about whether it's possible to answer the question empirically, but maybe something interesting would come of it (my p=.1).
Corrections agrees. It is unclear whether or not simple statistics (average arrest rates) offer a posterior probability that differs from our a priori beliefs about marginal statistics. It is certainly possible that given the average arrests are skewed, marginal arrests may be skewed. In order not to be, they would have to have a negatively correlated relationship with simple statistics.
ReplyDeleteAdding to this problem is that the very fact the statistics are being reported may or may not give information (viz., newspapers on a crusade, or newspaper reporters on a crusade, may have a propensity to only report distortionary statistics and leave out statistics that hurt their point, giving the statistics less weight due to data mining).