MBA Executive Committee Issues Statement regarding the recent study alleging racial disparities in federal sentencing that published by the Institute for the Quantitative Study of Inclusion, Diversity and Equity
As members of the bar, we take seriously our obligations to promote equal justice, exhibit the highest ethical conduct, and work to eliminate bias and discrimination in the justice system and elsewhere in society. We are aware that peer-reviewed research has indicated, and peoples’ lived experiences confirmed, that structural racism affects the criminal justice system, and that it is the responsibility of all participants in that system to work to oppose and remove those effects. At the same time, as Officers of the Court, we are acutely aware of our concurrent obligation, to defend judges and courts unjustly criticized.
With surprise and dismay, we reviewed the original paper posted online by the Institute for the Quantitative Study of Inclusion, Diversity and Equity, and the uncritical attention it has received in legal publications. This paper initially purported to identify specific judges as “the most discriminatory,” including the Honorable C. Darnell Jones II and the Honorable Timothy J. Savage of the Eastern District of Pennsylvania. The conclusions regarding Judges Jones and Savage are incompatible with the publicly stated experiences of such organizations as the Federal Community Defenders Office, the United States Attorney’s Office, Philadelphia Bar Association, Pennsylvania Fellows of the American College of Trial Lawyers, and others - including members of this committee - who have expressed their views that Judges Jones and Savage are of the highest integrity, fundamentally fair, and non-discriminatory of their judicial roles. In direct opposition to this attack on these Judges, critical commentary from economists, legal scholars, and other organizations contend that the faulty data and methodologies upon which the paper is based are fundamentally flawed. Following initial commentary from other organizations and scholars, the authors issued a “revised” study that omitted “findings” regarding individual judges. However, the authors have continued to disseminate the same flawed “analysis,” just with the authors’ original incendiary conclusions edited out. The authors also continue to make their supporting materials available to the public, which allows readers to view their “judge specific” findings. Because the “study” in question maligns the particular judges it cites and, as a result, unfairly and without justification questions the integrity of the Judiciary, we urge the authors to retract it in its entirety.
First, the data relied upon by the authors is incomplete, and the authors have acknowledged that they have not taken any steps to account for or control for this incompleteness. This incompleteness is both a result of the data sources relied on, and of decisions made by the authors. Namely:
- The authors acknowledge that they do not have data for all cases handled by each judge for the timeframe of their analysis. This is in no small part because the primary data source on which they rely contains less than half of the sentencing cases in the federal system.1 The authors moreover used only data they were able to “match” between the two sources they used, and were only able to achieve a match rate of roughly 50%.2 The authors went so far in their original draft to “urge readers not to infer much about a judge who is estimated to have a very high proportion of missing cases,” but fail to define “high proportion.” They nonetheless went on to provide conclusions regarding judges in their original paper – including Judges Jones and Savage – for whom they acknowledge they are missing data for more than half the cases within those judges’ districts and years on the bench. Apparently, recognizing the deficits in their analysis, the authors now state they “no longer place enough credence in judge-specific estimates to make sufficiently confident statements on any individual judges.” Simply put, the authors now also concede that their paper is unjustified and indefensible.
- The authors assume that the data they are missing for each judge is random, but acknowledge that they have not done anything to confirm this and indeed have no way of determining that the missing cases are, indeed, missing at random, or whether they are not random and therefore affect their results. (In their opening paper, they wrote, “One unknown potential source of error is that we cannot determine what percentage of each judge’s cases were matched in the JUSTFAIR database. If this missingness is as-if random with respect to sentencing variables of interest, that should not bias our results, but we have little way of determining this.”)
The authors’ use of incomplete data – and failure to perform any analysis to determine whether that incompleteness biased their results – renders their conclusions unreliable in the view of at least one leading expert in law and economics.3 Now, only after their data and methodologies have been exposed as inherently unreliable, the authors concede to having analyzed only a portion of the cases handled by each of the judges whom they have wrongly singled out. When challenged on the merits of their work, the authors are now forced to concede that they, themselves, do not know whether their own sample of analyzed cases are representative of the universe of cases handled by those judges, or whether there are differences between their undefined sample cases and the entire universe of the judges’ cases that could bias their results. Indeed, the authors now explicitly acknowledge this shortcoming in their updated paper, noting that “if sample inclusion is related to our measure of interest, then lacking the full population leaves our estimates more susceptible both to systematic error and random sampling error than they would be otherwise. In light of these limitations, our results should be seen as imperfect approximations for the degree of aggregate racial disparities and the interjudge variation therein.”
Second, the authors describe performing a statistical case data analysis, aimed at quantifying and measuring what they allege to be inequitable (and unlawful) Sentencing disparities of Criminal Defendants, attributable to each respective Defendant’s race. Specifically, based upon what they now acknowledge to be faulty data and flawed methodologies, the authors level the accusation that the cited Judges are “the most discriminatory” because in their solitary opinion, the authors allege that cited Judges allegedly dispensed preferential treatment to White Defendants when compared to the respective treatment of individual Black and Hispanic Defendants. The authors assert they have imposed controls in order to isolate the effect of discrimination by the sentencing judges. However, they acknowledge that there are a number of “unobserved characteristics” – i.e., characteristics that they did not control for – that could be influencing the outcome of their analysis, and that could therefore be responsible for the differences in sentence length, rather than discrimination by the sentencing judges. Disconcertingly, the authors acknowledge that their data models do not account for these characteristics whatsoever. Here, only after they have publicly tarnished and attacked the cited Judges as “the most discriminatory among their peers,” the authors now acknowledge their irredeemably flawed model.
Setting aside these breathtaking errors, the authors have also failed to control or account for a number of patently observable characteristics, such as: the specific offenses charged; 18 U.S.C. § 3553(a) factors; and what effect (if any) is attributable to individual prosecutors assigned to particular cases. As practitioners, we readily recognize that any one of these factors, taken alone, can result in a dispositive outcome which would affect the length of a Defendant’s sentence, regardless of the sentencing judge. Moreover, it appears that the authors have made no effort to account for factors, which one could reasonably describe as structural racism within the system. These include factors like statutory sentencing disparities between different kinds of drug offenses; Defendants’ criminal histories discretionary charging decisions; and statutorily imposed mandatory minimum sentences.
We are aware that experts in the field have raised a number of other concerns regarding the econometrics underlying the author’s conclusions. See Footnotes 1 and 2, supra. These concerns, combined with the data and methodological flaws outlined above, lead us to conclude that the “study” does not accurately capture differences in sentencing patterns driven by discrimination and does not offer conclusions that are reliable or accurate. It seems the authors themselves have now recognized the limitations of their original paper. It is unfortunate that this recognition only occurred after their highly inflammatory remarks regarding sitting judges were widely disseminated in the legal press and elsewhere.
As an organization, we believe that racial injustice must be uncovered wherever it exists in the legal system. Any substantiated accusation of racial bias must be confronted. However, such accusations must also be fully substantiated; the levying of baseless and unfounded accusations undermines both the effort to create a more equitable legal system, and the public’s faith in the integrity of the legal system.
In targeting individual judges – on the basis of what are now retracted conclusions – one must question whether the attorneys who contributed to or publicized this “study” consulted the Rules of Professional Conduct.4 Even though the authors have amended their study in light of universal rebuke from the legal community, they continue to publicize their unreliable data online. As such, in defense of the Courts, we are obliged to call upon the authors to retract their study in its entirety.
We also call upon legal publications who have publicized the prior version of the report to acknowledge its portion of responsibility for publicizing such a reckless, inflammatory and universally discredited report. In the future, our legal community can only benefit from greater editorial scrutiny and discernment.
1. https://sentencing.typepad.com/sentencing_law_and_policy/2021/07/provocative-new-empirical-paper-claims-to-identify-the-most-discriminatory-federal-sentencing-judges.html
2. https://sentencing.typepad.com/sentencing_law_and_policy/2021/08/guest-post-another-critical-look-at-provocative-paper-claiming-to-identify-the-most-discriminatory-f.html
3. https://sentencing.typepad.com/sentencing_law_and_policy/2021/08/guest-post-another-critical-look-at-provocative-paper-claiming-to-identify-the-most-discriminatory-f.html
4. Pa. RPC 8.2(a); See also Pa. RPC 8.4
