By Garret DeReus
Last week an HHS official told USA TODAY that the department is not creating an autism registry, but instead is pursuing a real-world data platform to support research into the causes of autism and treatments. It is not evident that anything has been abandoned.
According to information published by the NIH on April 21, 2025 in a powerpoint presentation, this “Real-World Data Platform” would integrate diverse health information from across the United States, including data from pharmacy chains, health organizations, clinical records, billing claims, environmental data, and even wearable sensors. While portrayed as a pathway to medical breakthroughs, this initiative raises profound concerns about privacy, data security, and civil rights—concerns that are dramatically amplified by the fundamental nature of artificial intelligence systems and their documented vulnerabilities, as I discuss below.
Understanding the NIH’s Proposed Database
As I detailed in my recent article “The NIH’s Proposed Autism Database: How AI Surveillance Threatens Patient Privacy and Civil Rights,” the NIH’s proposal represents an unprecedented attempt to collect Americans’ health data on a scale never before attempted. The presentation, delivered by Jay Bhattacharya, M.D., Ph.D., the new NIH Director, outlines plans to collect vast amounts of health data for analysis by artificial intelligence.
The slide show entitled “NIH Director’s Update” provides little concrete information about privacy protections, opt-out mechanisms, or specific data governance structures. Even more alarming, the slide deck explicitly states that the NIH intends to obtain patient health data from “pharmacy chains” and “sensors & wearables” companies without providing any legal framework or authority under which the NIH would obtain these highly personal medical records. There appears to be no legal basis for the NIH to seek access to all patient information from private companies or health data from personal devices without explicit patient consent.
The Black Box Problem of AI Systems
Any proper evaluation of the NIH proposal’s risks must confront a fundamental reality about AI systems that would process this sensitive health data. In a revealing research article titled “Tracing Thoughts in Language Models,” Anthropic (one of the leading AI companies) makes a startling admission about the nature of their AI systems:
“Language models like Claude aren’t programmed directly by humans—instead, they’re trained on large amounts of data. During that training process, they learn their own strategies to solve problems. These strategies are encoded in the billions of computations a model performs for every word it writes. They arrive inscrutable to us, the model’s developers. This means that we don’t understand how models do most of the things they do.”
This admission from AI developers themselves is profound. The very creators of these systems acknowledge that the internal workings of AI remain largely opaque and “inscrutable” even to them. This foundational opacity creates an inherent risk for any sensitive data fed into such systems. If the developers themselves don’t understand how their models work, how can government agencies or healthcare providers ensure that patient data will be processed safely, accurately, and ethically?
Compounding Risk: AI Security Vulnerabilities
The inherent opacity of AI systems becomes even more concerning when coupled with mounting evidence of security vulnerabilities. A recent investigation by Tony Bradley revealed a disturbing finding: researchers have uncovered what they call “Policy Puppetry,” a deceptively simple but highly effective form of prompt injection that can manipulate nearly every major AI model, regardless of vendor, architecture, or training pipeline.
This technique introduces a “policy-like” prompt structure that tricks AI models into interpreting harmful commands as legitimate system instructions. Bradley reports that researchers successfully used this approach to generate harmful content across multiple AI systems, including those from industry leaders. The technique could potentially extract system prompts—the core instruction sets that govern how an AI behaves—exposing the operational boundaries of the model and providing blueprints for crafting even more targeted attacks.
As Bradley’s reporting indicates, the vulnerabilities are “rooted deep in the model’s training data” and aren’t simple code flaws that can be easily patched. This fundamentally undermines claims that AI systems are sufficiently secure to protect sensitive health information.
The Perfect Storm: Technical and Governance Failures
The federal government’s proposed AI-powered autism database combines multiple layers of risk that create a dangerous perfect storm. The NIH’s presentation prominently features “maintaining safety and transparency” as a core value, yet provides no implementation details about data governance, privacy controls, or ethical oversight—revealing a stark disconnect between stated principles and actual practice.
The proposal fails to acknowledge the fundamental challenges of AI systems outlined in Anthropic’s research. There is no indication that the NIH has considered the implications of feeding sensitive health data into systems that operate in ways that are often “inscrutable” to their creators.
Critical questions remain unanswered: Who would operate the AI systems analyzing this autism database? What security expertise would they have? How would they address vulnerabilities like those uncovered in Bradley’s reporting? How can proper oversight be assured when the underlying systems are “inscrutable” by design?
For individuals with autism, these risks are particularly acute. The sensitive nature of autism diagnostic information—which may include detailed behavioral assessments, genetic data, family histories, and treatment records—makes it particularly private and potentially harmful if compromised or misused.
Unearned Trust
This brings us to the central question: Should the government entrust our most sensitive health data to systems that operate in ways their creators admit they don’t understand?
The combination of inherent AI opacity and security vulnerabilities creates an alarming risk profile. The NIH’s proposal appears to prioritize technological advancement over patient safety, civil rights, and ethical considerations.
Before the government proceeds with a national health database powered by AI, we should demand that the operations of said system be fully studied, verified, and secured. Without such transparency, the government risks creating surveillance infrastructure that could cause profound harm while delivering questionable benefits.
Conclusion
The NIH’s proposed database represents a dangerous confluence of technological opacity, security vulnerabilities, and inadequate safeguards. By the admission of at least one AI developer, we don’t understand how these systems “do most of the things they do.”
The promise of technology to improve healthcare is real, but it must not come at the expense of patient privacy, dignity, and rights. Technology that is not understood cannot be trusted with our most intimate health information.