Can chain-of-thought prompt engineering solve the black box problem?

There’s a better way to develop administrative common law for these changing times: Rather than accept that this law is something that courts make through an exercise of judicial creativity and policy balancing, we should instead embrace an administrative common law that courts find in prevailing practices.

This post is the fifth contribution to Notice & Comment’s symposium on AI and the APA. For other posts in the series, click here. Among the operative principles of administrative law is the requirement that agencies “examine the relevant data and articulate a satisfactory explanation” for their actions, commonly known as the reasoned decisionmaking requirement. What this […]

Nothing in the APA prohibits agencies from using computational tools to gather, synthesize, or even recommend policy choices—and agencies already often rely on modeling tools to inform regulatory standards. What the APA requires is that the final rule itself be the product of reasoned judgment—supported by evidence, responsive to significant comments, and explained in a coherent manner. If those conditions are satisfied on the face of the rule, it is unlikely that a court reviewing a challenge to the rule will examine how AI was used in the decisionmaking process, perhaps absent some external reason to suspect overreliance on AI. If, on the other hand, the rule ignores key evidence, fails to address major concerns or alternatives, or offers inconsistent reasoning, it will be struck down as arbitrary regardless of whether AI was used in its development.

This post is the third contribution to Notice & Comment’s symposium on AI and the APA. For other posts in the series, click here. Much of the emerging thinking about the relationship between administrative law and generative artificial intelligence is premised, expressly or implicitly, on the assumption that AI systems might come to play a leading role […]

Agencies will not be able to rely solely on today’s most ubiquitous forms of AI—namely, those based on ChatGPT and similar large language models—to avoid their obligation under the APA’s arbitrary and capricious standard to understand the problems they seek to solve, assess alternative solutions against legally relevant criteria, and make some kind of forecast about how these alternatives would change outcomes in the world. Administrators’ forecasts need to be about tangible outcomes, not about plausible-sounding words in sentences, however confidently they might be expressed. 

This symposium is for AI skeptics, AI believers, and everyone in between.