Last summer, the Department of Housing and Urban Development (HUD) received a PowerPoint presentation introducing an Artificial Intelligence (AI) tool: SweetREX, named after its creator, a third-year undergraduate in economics. Consistent with the Trump administration’s stated goal of eliminating 50 percent of all federal rules by the first anniversary of President Trump’s inauguration, Elon Musk’s so-called Department of Government Efficiency (DOGE) had used to SweetREX to review more than a thousand of HUD’s regulatory sections and reportedly used it to write 100% of deregulations at the Consumer Financial Protection Bureau (CFPB). These were not simply idle recommendations for human decisionmakers to consider; rather, the AI recommendations were treated as presumptively correct, with staffers required to justify in writing any disagreement with what the model suggested.

The growing use of AI for such complex governance functions as regulatory decisionmaking is attracting attention from policy experts and policymakers alike. The example of HUD’s and the CFPB’s use of SweetREX, in particular, illustrates how such situations risk putting two existing bodies of public law—those governing the rulemaking process and procurement, respectively—on a collision course: in particular, rulemaking’s emphasis on transparency and procurement’s emphasis on quick commercial transactions that often do not include robust transparency audits.

The impetus for this potential clash is that the architecture of many of the AI systems that the federal government employs are developed by private contractors. For instance, rather than being  developed entirely in-house, SweetREX is powered primarily by Google Gemini’s model. This means that the federal government is unlikely to have a full understanding of how the model was trained.

This gap in understanding is problematic in the context of agency rulemaking decisions because the law that governs these actions—the Administrative Procedure Act (APA)—requires the responsible agencies to offer an accounting of the deliberative process that undergirds their final decisions. Under that law’s “arbitrary-and-capricious” standard, which is used to evaluate the policy rationale for a regulation, the Supreme Court has held that agencies may not rely on factors Congress did not intend them to consider (i.e., “improper factors”). Of course, policing this standard is not easy even when agency decisionmakers are human. Hesitant to probe the minds of agency decisionmakers, courts have turned to the functional solution of evaluating the agency’s decisionmaking process.

However, AI complicates this analysis due to the “black box” problem. The inability to truly understand how an AI system reaches its decision, including what factors it weighs and how it weighs each factor. If the model relied on an improper factor, the agency that deployed it has effectively done so as well, with no way of knowing it has.

The black box problem is serious enough when the government builds the model itself. But when it outsources that function—such as with SweetREX—the problem multiplies. Modern large language models are trained on massive datasets drawn heavily from forums, comment sections, and other user-generated content, which often contain racist remarks, gender stereotypes, and other material that would be plainly impermissible if consulted directly by a human decisionmaker. By virtue of their prevalence in training data, these expressions are, to some degree, incorporated in a chatbot’s output.

This can result in severe problems for application. Amazon’s attempt to build an AI hiring tool ended with the team scrapping the project entirely because the model displayed blatant gender bias in evaluating candidates. A federal agency deploying a commercially trained model in rulemaking faces the same problem, not just for gender, but for any improper characteristic.

Whatever one’s personal feelings about AI might be, it is unrealistic to believe that the technology will not become more integrated into the rulemaking process in the future. The question then becomes how can this use be better reconciled with the APA’s demands for decisionmaking based on reasoned deliberation of legally relevant factors. What the spirit of the APA requires in this context is, at bare minimum, information on how the model was trained so that agencies and reviewing courts have a starting point to assess to what extent the model was exposed to improper factors.

As it turns out a major legal barrier to achieving this goal arises from another area of public law: procurement. Specifically, the regulatory regime governing procurement of most third-party AI systems, in particular Federal Acquisition Regulation (FAR) Part 12, is structurally unfit to navigate the APA compliance challenges that AI use in rulemaking poses.

FAR Part 12 is the primary vehicle by which the government procures commercial goods, including commercial software. FAR Part 12 was established under the Federal Acquisition Streamlining Act (FASA) signed by President Bill Clinton in 1994. FASA was a cornerstone of the Clinton/Gore Reinventing Government initiative designed to ensure that the government mimicked the operation of a business as much as possible, with the fundamental presumption that government was inherently inefficient. FAR Part 12 was meant to allow the government to circumvent the more lengthy procurement process of FAR Part 15 for commercial items, with FAR Part 12 favoring efficiency to such a degree that it requires the government to require that agencies conduct market research to determine whether suitable commercial items are available acquire them if so. Essentially, FAR Part 12 is not just an option for agencies to expedite the purchase of commercial items, but a mandate for them to do so wherever possible. Currently, FAR Part 12 is the most common pathway for federal AI procurement.

By its very structure, FAR Part 12 limits the government’s ability to impose requirements beyond what is customary in the marketplace. Contractors selling commercial AI products, or any other commercial product under FAR Part 12, are not required to grant the government broader usage rights than those given to any other customer, although the government can request expanded rights that the contractor can grant at their discretion. The theory that justifies FAR Part 12’s speed is fundamentally that the government is just another customer, albeit a large one, and will be treated as such.

As a consequence, Part 12 is structurally ill-suited to allow the government to contractually compel disclosure on how an AI model is trained, including information on model weights, training data, and audit logs. The practical upshot is that government agencies utilizing AI in rulemaking generally cannot both procure the foundational model through FAR Part 12 and follow the APA. One of them must give way.

The Executive branch has not been completely blind to issues surrounding AI procurement. Under presidents Donald Trump and Joe Biden, the Office of Management and Budget (OMB) –  one of the agencies charged with implementing FAR – sought to supplement the existing FAR structure with new AI-specific procurement guidance. The Biden OMB’s M-24-18 directed agencies to require transparency documentation for AI used under certain circumstances. This was superseded by the Trump administration’s M-25-22, which addressed a different set of problems, mainly privacy, use of government data, and maximizing the use of American-made AI. Both memos identified some of the right problems. Neither solved them.

The reason is fundamentally structural. As OMB memoranda, they do not have any immediate binding effect on government contractors. Instead, their scope is limited to governing how agencies handle procurement of AI. These govern contracts only if an agency successfully incorporates them into contract terms. However, that incorporation is complicated by these requirements’ tension with FAR Part 12: the government cannot both acquire these tools with the ease of a commercial purchase, yet also seek bespoke requirements that no other commercial buyer would be entitled to.

Procurement policy affecting AI has not been limited to OMB memoranda. In March 2026, the Government Services Administration (GSA)—another agency that shares implementation responsibilities for FAR—proposed GSAR 552.239-7001, an addition to the General Services Administration Regulation (GSAR), which is a GSA-specific supplement to FAR. Unlike OMB memoranda, these governance requirements, which apply across a host of areas including data control, portability, domestic sourcing requirements, and ideological output standards (to prevent “woke AI”), can bind contractors directly. Backlash quickly ensued, as the technology industry criticized the proposal for threatening intellectual property rights, dismantling established contracting norms, and introducing overly vague compliance requirements.

All three attempts fail due to the same problem: attempting to overlay requirements onto aspects of procurement that are structurally unfit to incorporate them. If the fundamental authority of the transaction still draws from FAR Part 12, or another similar pathway typically subject to commercial defaults, the attempts by the OMB and GSA to overlay these requirements with custom AI procurement requirements are akin to attempting to fix a crumbling house by applying a new coat of paint rather than replacing deteriorating bricks.

The solution does not lie with a more specific OMB memorandum or GSA rule, although those could certainly help, but rather through returning to FAR Part 15: the lengthier negotiation framework that Part 12 was designed to circumvent. The government cannot continue to buy AI like it buys office furniture. While the FASA default of using FAR Part 12 wherever possible may be appropriate for other goods, it presents significant accountability and statutory risks when applied to certain uses of AI. FAR Part 15, on the other hand, is a natural fit. FAR Part 15, or negotiated procurement, provides the structural framework that AI procurement requires, specifically when AI is being used to support decisionmaking or used in a context that would be subject to the APA, as it allows agencies the broadest latitude to negotiate tailored terms, which in the AI context, could include provisions regarding model training and auditing.

This approach will result in a slower process: an anathema to the Reinventing Government instinct that has governed federal procurement for 30 years. For some procurement cases, they may have a point. It is in no one’s best interest if it takes months to order basic commercial items for routine use. In such cases, OMB and the GSA can explore tailored reforms to FAR Part 15 that help streamline its procedures for AI-specific procurement. This approach would be preferable to the status quo of starting with FAR Part 12.

For a technology as novel as this, though, when used in some of the most consequential agency decisions, a slower, more deliberative process is likely to be salutary. The alternative, acquiring systems the government cannot meaningfully audit, cannot fully explain, and cannot substantively defend in court, risks undermining the critical values of transparency and accountability that undergird the modern administrative state.

James Goodwin is policy director at the Center for Progressive Reform. Arvind Salem is a legal research intern at the Center for Progressive Reform.

Arvind Salem is a student at Stanford University. He studies political science and government, and is a member of the Stanford Institute of Politics.