This post is part of Notice & Comment’s symposium on Joshua D. Blank and Leigh Osofsky’s Automated Agencies: The Transformation of Government Guidance. For other posts in the series, click here.

Automated Agencies: The Transformation of Government Guidance, by Joshua D. Blank and Leigh Osofsky, is part of the authors’ larger study of informal legal guidance, such as agency publications. The informal legal guidance described here is “automated,” meaning that the guidance comes from government-created computer programs. The user inputs information, and the programs uses these inputs to determine which prewritten text to return. Some programs use natural language inputs to select the prewritten materials. Other programs use decision trees, asking a series of questions to guide the user through a branching “tree”: if the user says yes, go here and ask this question; if the user says no, return this information. The government does not use large language models or generative AI for its current automated resources.

In other words, the technologies the government uses now essentially chop up agency guidance, such as publications, to serve only the relevant parts to the user, sometimes with language even more terse than that in the publications. This approach may be why the agency officials that the authors interviewed are so certain that the material is “correct” or “accurate”: if what’s provided is just chopped-up, condensed informal guidance, and the officials believe that the underlying informal guidance is accurate, the chopped-up version is also accurate. The information provided might not be relevant to the individual, but it isn’t wrong. (As the book explains, informal guidance is sometimes actually arguably wrong, but let’s set that aside for now.)

Chapter 6 of the book imagines a world in which computers can fully manage every part of compliance and enforcement, with no human intervention. As the book describes, this world is fictional. No technology today can accomplish anything close to this. Countries that “prepare” tax returns for their tax filers are simply prepopulating the returns based on information reported to them by third parties (“information reporting”)–something that does not require “artificial intelligence” in any sense.[1] Similarly, tax preparation software such as TurboTax and Direct File guides the user through entering information onto Internal Revenue Service forms; this software therefore also does not involve artificial intelligence. Perhaps large language models and generative AI will someday allow governments to fully automate compliance and enforcement; perhaps large language models and generative AI are the wrong technology for automating compliance. The world of fully automated compliance is, at any rate, not the world we live in now, and it is not the world that the book describes.

Thus the “automated” legal guidance described in this book is closer to the guidance provided in publications than it is to guidance provided by imaginary robots. What the book calls automated legal guidance is nevertheless different from other types of informal guidance, in at least two ways.

First, as the book explains, some of the transparency associated with other informal legal guidance is missing. For example, the IRS website provides years of old publications, which allows changes to be traced through the years, and the publications have dates of revision. But there is no way to find out when the prewritten text behind the automated tools is changed, or when it was last revised. I strongly endorse the book’s proposal that there should be more transparency around automated legal guidance. The exact technology underlying these tools should also be transparent. The book proposes, for example, releasing the full decision tree if a decision tree is used in the guidance, and releasing the code for any programs that use natural language search.

I’d suggest going even further: for all automated tools, the code underlying the tool, including all outputs, should be made public and usable, and should have transparent version control. The code for the United States e-filing program Direct File is now almost entirely available through a public repository. (The repository of course omits code and data that would raise privacy concerns, such as personally identifiable information.) Fully releasing all nonprivate code and data should be the model for all automated guidance. Releasing the full computer code and outputs, including all the prewritten text, will show the extent to which these outputs are basically chopped-up guidance that is available elsewhere and will also allow people to determine systematically and efficiently whether and when the automated tools provide substantively different information than other informal guidance. And these repositories could allow people to use the government computer code to make other, similar tools, and to improve on what the government has created.

Second, nobody thinks a publication is talking “just to them.” But automated guidance may trick the user into believing that the information or guidance they receive is genuinely personalized–or even from a person, in some meaningful sense. The tendency of people to quickly endow the source of human-like speech with human characteristics has been much in the news since the advent of programs such as ChatGPT, but it long predates these programs. ELIZA is a language processing program created by Joseph Weizenbaum in the 1960s. Weizenbaum created a script for ELIZA of “a nondirective psychiatrist in an initial psychiatric interview,”[2] because this structure allowed him to avoid including “real-world knowledge.” The script simply reflected the user’s remark back at him. If the user typed, “My mommy took my teddy bear away from me,” the program might respond, “Tell me more about your parents” (and did not have to know anything about teddy bears). Weizenbaum describes what happened when people used the program:

ELIZA created the most remarkable illusion of having understood in the minds of many people who conversed with it. People who knew very well that they were conversing with a machine soon forgot that fact….This illusion was especially strong and most tenaciously clung to among people who knew little or nothing about computers. They would often demand to be permitted to converse with the system in private, and would, after conversing with it for a time, insist, in spite of my explanations, that the machine really understood them.

This credulousness has become known as the “ELIZA effect,”[3] and it persists today. Users of the government’s automated tools may thus be particularly at risk of misunderstanding what they are receiving from the automated tools, even with abundant disclosures.

The book states that the government rates the success of its tools based on user experience: the happier the user is with the tool, the more of a success the government deems the tool. But there may be an irreconcilable tension between what makes users happy and an accurate, thorough tool that does not mislead the user. These tools are attempts to explain the law to users, because the tools cannot prescribe what the users should do–the technology simply does not exist to give truly individualized guidance. It is not a road to happiness for a non-lawyer to receive an accurate explanation of complex law. One suspects that the simpler the explanation, the more colloquial the language, the happier the user. (Indeed, one suspects that many people would prefer simply to know what to do, with no explanation at all.) This is the tension the book describes between “simplexity” and accuracy.

Moreover, the more chatty and colloquial the response, especially if the user can use natural language to elicit the response, the greater the risk that people will fall prey to the ELIZA effect and will not understand–perhaps not be able to understand–that they are not receiving personalized advice. Even users who rationally accept that there is no person providing advice may still anthropomorphize the computer program providing that advice. Automated guidance thus might exacerbate the access to justice gap. Unlike publications, automated guidance may fool people into thinking that they have received individualized guidance, when they actually have not. People without resources won’t receive personalized legal guidance, and will not know that they have not received personalized legal guidance. They will have neither the legal knowledge and guidance that they seek, nor the knowledge that they lack that knowledge and guidance. This double ignorance may make people happier, but it is hard to understand it as a success.

Sarah B. Lawsky is the L.B. Lall and Sumitra Devi Lall Professor of Law at the University of Illinois Urbana-Champaign College of Law.

---

[1] OECD, Tax Administration 2024, Part 4: Assessment: Pre-filled Returns, available at https://www.oecd.org/en/publications/tax-administration-2024_2d5fba9c-en/full-report.html, provides an overview of approaches to prefilling returns in various countries.

[2] This description of ELIZA is drawn from Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to Calculation 188-192 (1976).

[3] See, e.g., Douglas R. Hofstadter, Gödel, Escher, Bach: An Eternal Golden Braid 600 (1979) (describing this response and referring to “this weird ‘ELIZA-effect’”).