This post is the twelfth contribution to Notice & Comment’s symposium on AI and the APA. For other posts in the series, click here.

If you had an AI bingo card, “use case” and “human in the loop” would be right there, nestled among “hallucinate” and “algorithm.” As we try to understand the consequences of machine learning, large language models, generative AI, and whatever else we are calling it when you read this essay, a slew of terms have migrated into the law and policy domain. In one sense, this vocabulary helps us keep up with exhilarating changes. But sometimes it is used as a sleight of hand.

For example, various AI “use cases” have been offered for regulators, many of them in this symposium. Regulators could use algorithms to help find regulations that refer to repealed statutes. They could use these tools to observe patterns in large sets of data to help reveal fraud or other noncompliance with existing regulation. They could rely on topic modeling and other techniques to satisfy the APA’s requirement to consider public comments. They could allow generative AI to offer suggestions on human-written materials such as the preamble, regulatory text, or regulatory analysis for a notice of proposed rulemaking. They could direct such a system to write the first draft and then respond to prompts from humans to refine the draft. They could ask generative AI to write the whole thing, give it cursory human review, and publish it. Hmm, that escalated quickly.

These are all use cases, but that doesn’t mean they are all good ideas. Best understood, use cases are requirements, goals, proposals. Software developers, to ensure that their work will meet users’ needs, ground their work in use cases. Ivar Jacobson’s Object-Oriented Software Engineering: A Use Case Driven Approach (1992) explains: “We ask the users what they want to change (which use case)” and go from there (p. 129). As the process unfolds, developers “talk to the users and see if we are building the correct system according to their requirements.” Id. This happens in an iterative manner, with developers ensuring that “users are pleased with what we are about to design, before we start to build the actual system.” Id. 

Who are the “users” of regulatory or rulemaking systems? Jacobson tells us that a “user” is the “actual person who uses the system” (p. 127). I’m not sure this helps to answer our question. Perhaps it is the regulators, but who exactly does that mean? The political leadership, or the rank and file staff? Maybe it depends. And while that might be who prompts the system to produce regulatory output, that output is meant to produce regulatory outcomes that are in service to us all. Perhaps that makes us the users. Did any of us ask the Department of Transportation to use Google Gemini to write safety regulations, as reported by Jesse Coburn for ProPublica? I’m going to go out on a limb and say no. One can want a faster pace of regulatory change while not wanting the government to embark on a plan to produce “word salad” and pass it off as “good enough.” Id. Who, exactly, asked for this, and do they speak for us? In this phase of AI’s development, use cases fall like embers from a wildfire.  

When we identity a use case that pushes too far into a value-laden decisionmaking, we are often told not to worry because the system will have a “human in the loop” (HITL). A system with HITL contrasts with a system that is entirely autonomous. In the rulemaking context, an example of an autonomous system is one that agency tasks with both locating “problematic” regulations and then taking the legal steps needed to repeal them. Even DOGE, a proponent of very aggressive uses of generative AI in government, was quick to promise HITL. In an astonishing slide deck obtained by the Washington Post, DOGE proposed to delete 50% of all federal regulations with only 7% of the human involvement it would otherwise require. Don’t worry about unlawfulness, it implied, because lawyers will spend 30 minutes working on each proposed and final rule. Id. slide 8. (If you have even passing familiarity with rulemaking and the legal complexities it involves, you will appreciate the absurdity of asserting that this will suffice.) 

HITL is far from a silver bullet, and it especially underdelivers when it flattens the role of human judgment in public decisionmaking, as DOGE did in its slide deck. There are certainly instances in which people can help keep algorithm-supported decisionmaking on track, but we need to discard the notion that algorithms are capable of making “decisions” for us on policy matters. Too often, HITL is offered as a multipurpose fail safe, when it actually serves to demote people to being merely “in the loop” rather than being in the driver’s seat. For decisions that involve a cascade of values-driven choices, like rulemaking, algorithms are not capable of making needed judgment calls, and a putting someone “in the loop” is not enough.

In a forthcoming paper, Ghostwriting the Government, I argue that regulatory preambles and analysis should not be outsourced to anyone; not stakeholders, not contractors, not machines. Now, the impulse to turn to machines for rulemaking is understandable. Rulemaking documents like proposed and final rules follow a format that involves a lot of text and a lot of human effort. And generative AI can produce a lot of text, fast, and with almost no human effort. In my paper, I argue that where an official has a duty to engage in reason-giving, they may not outsource the writing of those reasons without breaching their duty. To get there, I draw upon the deep connection between writing and thinking and how it relates to human judgment. You might be familiar with the idea of a judge finding that their initial views about a case “don’t write.” This is because, sometimes, once a judge puts their fingers to a keyboard, the process of writing reveals problems. This is a regular feature of writing; it gives us a chance to reflect on it and decide if it’s right. If you skip the writing, it’s easy to skip the thinking.

I was surprised that, compared to judges and legislators, regulators have the strongest legal obligation to express their reasons in writing because of the APA and surrounding doctrine. (I thought judges might be in first place, but when they write out their reasons it is not necessarily because of a legal duty to do so.) This legal duty places limits on when regulators can outsource. I argue that this duty extends even to editing someone else’s work. The temptation to accept the framing of the draft in front of you can be just too great, especially when you have a lot of demands on your time. I do not conclude that this means that Cabinet secretaries need to be the ones drafting proposed rules. Instead, agency rules are the product of the agency writ large, not one individual. But signing off on work generated by outside interests—and I include AI systems in this category—gives in to the idea of “good enough for government work” while also turning it on its head.

In sum, there are plenty of good use cases for AI in government decisionmaking, but sometimes we need to say no. Right now it seems like it’s harder than it should be to say no. AI systems are truly remarkable but they are not capable of making values-laden policy decisions. We kid ourselves if we think that a “human in the loop” is more than an impoverished way to think about what agencies owe the public when it comes to significant regulatory actions. We can likely make great progress in regulatory policy by letting algorithms into our loop, not the other way around.

Bridget C.E. Dooling is Assistant Professor of Law at The Ohio State University.