This post is the fourth contribution to Notice & Comment’s symposium on AI and the APA. For other posts in the series, click here.

Artificial intelligence (AI) systems are poised to become embedded in the daily operations of government agencies. From processing masses of public comments to suggesting regulatory standards, AI promises to bring greater efficiency to the complex world of rulemaking. And the Trump administration is enthusiastically pushing for greater integration of AI within the federal government.

But the price of greater efficiency may be less transparency. And government use of AI raises a fundamental legal question: Can agencies that rely on AI to develop rules satisfy the Administrative Procedure Act’s bedrock requirement of “reasoned decisionmaking”?

The APA, enacted in 1946, envisions governance grounded in deliberation, transparency, and accountability. Agencies must engage with the public through notice-and-comment rulemaking, consider public input, and provide reasons for their final decisions. Courts reviewing agency rules must be able to trace a rational connection between the evidence, the agency’s reasoning, and its ultimate decision. The APA reflects a traditionally human idea of reason: that administrative decisions should be justified through discernible logic, not intuition, emotion, or inscrutable processes. 

AI, by contrast, operates through methods that are often inscrutable to human observers. Rather than applying algorithmic rules supplied by human designers (like a traditional computer program), generative AI systems such as ChatGPT operate by detecting and predicting patterns within a vast “artificial neural network” of training data. Where traditional computing programs will only execute tasks based on human-coded instructions, AI systems learn from their datasets and autonomously adjust their processes. As a consequence, even the designers who build AI models are often unable to understand the underlying steps that lead to a certain output. The tension between the APA’s expectation of traditionally legible reasoning and the often-inscrutable processes of AI models may well define the next era of administrative law. 

At first glance, AI and “reasoned decisionmaking” seem inherently at odds. The APA assumes a human decisionmaker who can be held accountable for explaining and defending a decision. But because of the difficulty in discerning an AI model’s decisonmaking process, even if an agency staffer were to ask an AI system to justify a proposed safety standard, the staffer might not be able to confirm that the AI system’s justification actually reflects how it reached its decision. And AI’s “sycophancy problem,” where it tends to agree with users’ assumptions even when they are incorrect, poses further risks. If agencies defer to an AI system’s reasoning without sufficiently examining its inputs and processes, they risk adopting rules that appear reasonable at a quick glance but fail to rationally consider all important factors: a textbook violation of the traditional “arbitrary and capricious” legal standard.

But this intuitive tension may overstate the legal barrier. Nothing in the APA prohibits agencies from using computational tools to gather, synthesize, or even recommend policy choices—and agencies already often rely on modeling tools to inform regulatory standards. What the APA requires is that the final rule itself be the product of reasoned judgment—supported by evidence, responsive to significant comments, and explained in a coherent manner. If those conditions are satisfied on the face of the rule, it is unlikely that a court reviewing a challenge to the rule will examine how AI was used in the decisionmaking process, perhaps absent some external reason to suspect overreliance on AI. If, on the other hand, the rule ignores key evidence, fails to address major concerns or alternatives, or offers inconsistent reasoning, it will be struck down as arbitrary regardless of whether AI was used in its development.

In truth, the APA’s ideal has already been significantly watered down by the realities of the rulemaking process. Industry, interest groups, and other stakeholders use the notice-and-comment process mainly to preserve arguments for litigation, not to inform the agency’s decisionmaking process—sometimes to great success. In response, agencies do significant front-end work to essentially finalize their rule before taking comments, and use the final rule to respond to comments in an effort to litigation-proof the rule. 

Further, agencies receive such an enormous volume of comments that they frequently hire contractors to read and summarize them. What AI changes is the scale at which agencies can delegate functions of the rulemaking process and the speed at which these functions can be performed. Compared to human contractors, an AI model could presumably review and summarize comments in a drastically shorter period of time. 

This is not to say that agency use of AI does not involve legal risks. While the APA may not impose a categorical bar to the use of AI, AI use may make agency actions more legally vulnerable. If a judge is already suspicious of an agency’s rationales for its rule, learning that the agency relied on AI to develop and justify its decision may tip the judge towards ruling that the agency did not satisfy its obligation to engage in reasoned decisionmaking, even if AI use alone is unlikely to lead a judge to this decision.

Agencies should therefore exercise caution in how they integrate AI into the rulemaking process, and be sure that agency staff fully understand how the AI tools they use function. For instance, there is a significant difference between agency staff using an AI product that has been trained specifically on data provided by the agency, and agency staff using a publicly available model such as ChatGPT that is trained on a vast, unfiltered public dataset outside the agency’s control. In the first case, the agency can control and review the inputs, and ensure that they are accurate and high-quality; in the second case, the agency has no control over the inputs, and little to no ability to even review what inputs the AI model selects. Relying on an analysis without examining or understanding its inputs would run the risk of “fail[ing] to consider an important aspect of the problem” and violating the APA.

Agencies should ensure that AI remains an analytical assistant with human oversight, not an autonomous decisionmaker. For instance, say an agency tasks an AI model with reviewing data and proposing standards. If the AI system proposes three regulatory standards and explains the advantages and disadvantages of each, and the agency then selects one of the standards, the agency may be on firmer legal ground than if the AI model proposes a single standard which the agency then adopts without further explanation. Otherwise, the agency may have failed to satisfy its legal obligation to consider alternatives to its final action (just as it would if it developed a standard without AI and failed to consider any alternatives). 

Finally, agencies should be transparent about their AI use; in fact, an executive order from the first Trump administration directs agencies to “disclos[e] relevant information regarding their use of AI” to the public. Ideally, agencies should not only disclose how AI was used in a rulemaking process, but should also solicit public comment on their design choices. For instance, agencies could disclose and allow public comment on the specific AI model used and the entity responsible for it (e.g., developer, provider, or vendor) and the agencies’ AI risk assessment and AI risk mitigation practices. Agencies could also explain in detail at what stages they used AI in the rulemaking process, what inputs they provided, and how they reviewed the AI model’s outputs.

Transparency, accountability, and meaningful public input will be essential to preserving (and rebuilding) trust in both technology and government. Agencies should remain vigilant about maintaining human oversight, documenting how AI informs their choices, and inviting public scrutiny of those practices. But so long as agencies ensure that their final rules satisfy the APA’s well-established requirements—by weighing evidence, considering alternatives, and coherently explaining the reasons for the final decision—the APA is unlikely to significantly constrain the use of AI in the rulemaking process.

Jack Jones is a Legal Fellow at the Institute for Policy Integrity at NYU School of Law, where Burçin Ünel is the Research and Policy Director. This post does not purport to represent the views, if any, of NYU School of Law.