A federal judge this week vacated over $100 million in grant cancellations by the Department of Government Efficiency (DOGE), ruling that the agency’s automated screening process violated the Constitution. The 143-page decision from US District Judge Colleen McMahon zeroes in on a pipeline that relied on ChatGPT as a binary classifier for diversity, equity, and inclusion (DEI) content.

DOGE staffer Justin Fox testified that he fed each grant description from the National Endowment for the Humanities (NEH) into ChatGPT with a single prompt: “Does the following relate at all to DEI? Respond factually in less than 120 characters. Begin with ‘Yes.’ or ‘No.’” Fox admitted he never defined “DEI” for the model and had no idea how ChatGPT interpreted the term. He also compiled a list of “Detection Codes”—keywords like “BIPOC,” “LGBTQ,” “Tribal,” and “Gay”—and ran those against every grant description. Grants flagged by either method were summarily cut.

From a system design perspective, this is a textbook case of treating a language model as a reliable decision engine without any validation layer. There was no human-in-the-loop review, no confidence threshold, no fallback for ambiguous outputs. The judge noted that DOGE didn’t bother to verify whether ChatGPT’s “DEI” rationale actually made sense. In engineering terms, they deployed a black-box classifier with zero observability and no rollback mechanism.

McMahon rejected the government’s defense that ChatGPT was the one making biased judgments. “There is no distinction to be drawn here between the Government and ChatGPT,” she wrote. The ruling restores over 1,400 grants, including projects on Holocaust education, civil rights, and indigenous knowledge. For data engineers, the lesson is clear: if you pipe a spreadsheet into an LLM and use the output as a kill switch, you’re not automating—you’re outsourcing accountability.

Source: The Verge