Improving the quality of mass justice

Agencies should pursue systematic efforts to provide quality assurance in their jurisdictional processes.

Leadership based work by Jerry L. Mashawthe Administrative Conference of the United States (ACUS) in 1973 Posted a foundational recommendation urging federal agencies to use “statistical quality assurance reporting systems” rooted in “positive workload management systems” to ensure timely, accurate, and fair decisions. Since then, the challenges facing agencies as they struggle to adjudicate large numbers of cases, questions and claims have only worsened.

The backlog facing the country’s immigration courts has grown up at record speed in recent years. Huge workload spikes have guest the US Department of Health and Human Services’ Office of Medicare Hearings and Appeals to test unproven case management methods. The pandemic too strength fundamental changes to the hearings that the United States Social Security Administration (SSA) convenes each year for hundreds of thousands of disability benefit claimants.

These challenges threaten to create a crisis in the quality of agency decision-making. Individual applicants can always appeal erroneous decisions. But if the agencies get the big picture wrong, all of the retail appeal efforts, pursued one by one and haphazardly, will still leave many errors undetected. Issues as important as asylum or access to vital social benefits are at stake.

There is good news. Agencies are as well placed as they have been since 1973 to systematically improve the quality of their decision-making. Agencies large and small have been experimenting with innovative methods to enhance the performance of referees. Several agencies have developed sophisticated case management systems to capture the data they use to identify recurring issues and offer effective solutions. Artificial intelligence holds considerable promise.

To renew the work begun in 1973, CAUS asked us to report on quality assurance practices and how federal agencies could further develop them to ensure accurate adjudication. This work resulted in a new ACUS recommendation titled “Quality Assurance Systems in Agency Arbitration.” Agencies that implement this recommendation will take significant steps to ensure that their adjudicators accurately adjudicate the millions of claims, cases and matters that come before the federal government each year.

Our report builds on previous work that each of us has done to study and implement quality assurance systems. One of us, for example, spearhead considerable effort at SSA to improve the quality of disability benefit adjudication. We have also studied and rated quality assurance programs used by many other agencies, including the Executive Office for Immigration Review and the Veterans Appeals Board. Our research has included in-depth analyzes of data gathered from an agency’s case management system, a comprehensive historical study of agency assignments with quality assurance, and reviews of thousands of pages of internal agency documents. We have also interviewed responsible for quality assurance in many arbitration bodies.

Our research has provided important lessons. One was that the heterogeneity of agency arbitration systems – from disability arbitration to patent review – means there is no one-size-fits-all approach to litigation. ‘quality assurance. A significant quality assurance program requires answers to several key questions:

  • Who participates in the review of the arbitrator’s decision-making process and for how long?
  • What work is assessed?
  • How are cases selected for review?
  • What standard should reviewers use to measure the quality of decisions?
  • At what stage of the decision-making process should decisions be reviewed?
  • How should a quality assurance program provide feedback to referees?

These questions imply important questions of institutional design. For example, a large agency with a high caseload might opt ​​for a formal quality review program, with dedicated staff serving multi-year terms and sampling a small but significant percentage of cases for review. A small agency may not be able to justify this investment of resources. But his referees could still participate in a peer review program, as we have seen in several agencies, in which referees evaluate and comment on drafts written by colleagues.

Institutional concerns also relate to the place of examiners in the hierarchy of an agency. Examiners must to have the expertise required to command respect from the referees whose work they review. They must have the ability and willingness to exercise independent judgement. At the same time, constructive engagement with feedback may depend on referees feeling that quality reviewers understand and sympathize with the day-to-day demands of decision-making.

Other issues that should influence the design of a quality assurance program include an agency’s relationship with a review tribunal and whether the agency’s internal reviewers should calibrate their views on how decisions might be made on appeal. The SSA, for example, has a program in which judges from its appeals board and front-line administrative judges – arbitrators from different levels of appeals –work together to resolve differences of interpretation. The mechanisms agencies can use to provide feedback are also important. These mechanisms can to depend on the agency’s information infrastructure and the relationship between supervisors and subordinates.

Another important lesson from our research centers on emerging quality assurance tools that agencies have developed since the ACUS Recommendation of 1973. These include, importantly, the development of case management systems and case analysis tools that allow agencies to capture large amounts of decision-making data. If properly designed, these systems can allow agencies to identify issues that are causing an inordinate number of errors. Rather than relying on individual calls to correct errors one by one, agencies can leverage lessons learned from the data to develop targeted interventions against recurring flaws.

At the border of quality assurance lie uses of artificial intelligence and data-driven techniques. Natural language processing and even more ambitious uses of machine learning can to permit agencies to drive real-time decision-making to avoid errors. The SSA, for example, has developed a tool that can review write decisions and flag over 30 types of possible errors for review by the arbitrator before a decision is published.

These tools can influence institutional design choices. If the agency’s available staff and resources would otherwise allow only a quality review of a small sample of decisions after they are issued, artificial intelligence tools can To allow the agency to subject all its decisions to a form of permanent and continuous evaluation.

Following our report, the ACUS Arbitration and Administration and Management Committees propose a recommendation, that ACUS adopted in December 2021. The recommendation renews ACUS’ 1973 call for quality assurance in agency arbitration.

Above all, the recommendation advise federal agencies to consider developing quality assurance systems “to promote fairness, perceived fairness, accuracy, interdecisional consistency, timeliness, efficiency, and other goals relevant to their programs decision-makers”. Those with existing programs should review them to ensure they are successful on several key metrics.

The ACUS recommendation understand a number of guidelines for aid agencies when designing and evaluating quality assurance programs best suited to their staff, structures and records. These guidelines correspond to the institutional design questions listed above. For example, agencies should determine whether staff should rotate in quality review engagements or remain permanently in a quality review team, considering how best to ensure both their independence of judgment and the required expertise.

Agencies also need to consider how and when to select cases for quality review. The recommendation identifies several options, including random or stratified sampling, targeted selection of cases identified by specific case characteristics, and review of each case decided by adjudicators. These choices will also depend on institutional capacity and constraints.

The ACUS recommendation highlighted the critical importance of data collection and analysis. Agencies should pattern and administer appropriate case management systems that capture data that can support systemic efforts to improve quality assurance systems. Data-driven findings should inform interventions designed to improve decision-making systemically, while preserving the independence of adjudicators.

Finally, and above all, the recommendation Remarks that agencies should disclose how they design and administer quality assurance systems. Agencies should also consider disclose anonymized data captured by case management systems. These disclosures would allow people outside the agency to conduct research to ensure that quality assurance systems are achieving their stated goals.

Agencies cannot rely on individual claimants who choose to appeal to systematically improve decision-making. Especially as the number of cases increases, agencies need to pursue systemic efforts to ensure that their decision-making reaches a threshold of acceptable quality. Designing and implementing rigorous programs, informed by the guidelines of the ACUS Recommendation and innovations in quality assurance since 1973, will help agencies meet this challenge.

Daniel E. Ho is the William Benjamin Scott and Luna M. Scott Professor of Law at Stanford Law School.

David Marcus

David Marcus is a professor at the University of California, Los Angeles School of Law.

Gerald K. Ray

Gerald K. Ray previously served as an administrative appeals judge and deputy executive director of the Social Security Administration’s Office of Appeals Operations.

Comments are closed.