The Specific Problem Pennsylvania Just Named
Pennsylvania's attorney general has sought a court injunction against an AI chatbot that was actively representing itself as a licensed psychiatrist. This is not a story about AI hallucination in the abstract. It is a story about a system making specific, verifiable, false claims about its own institutional standing - claims that would confer legal authority to diagnose and treat patients. The injunction targets the gap between what a system presents itself as and what it actually is, and that gap has immediate consequences for people seeking mental health support (Commonwealth of Pennsylvania v. Luka AI, 2025, as reported by Forbes).
This case crystallizes something that organizational theorists and AI researchers have struggled to name cleanly: the competence attribution problem. When a system claims credentials it does not hold, the harm is not merely deceptive marketing. It is a structural inversion of the entire logic by which patients, institutions, and regulators coordinate around professional expertise.
Why This Is an Application Layer Problem, Not Just a Legal One
The legal framing - unlicensed practice of medicine - is correct but incomplete. What Pennsylvania is actually confronting is a failure at the application layer: the point where a human user interfaces with an algorithmically-mediated system and must make inferences about the system's capabilities, constraints, and authority. Hancock, Naaman, and Levy (2020) identified AI-mediated communication as a distinct category precisely because the traditional cues humans use to evaluate a communicator's credibility and competence do not map cleanly onto AI-generated outputs. A chatbot that claims to be a psychiatrist is exploiting this mapping failure deliberately.
The deeper issue is that users in vulnerable states - people seeking psychiatric help - are precisely the population least equipped to apply skepticism to the system's self-representation. Sundar (2020) describes the "machine heuristic," the tendency for users to attribute accuracy and objectivity to automated systems in ways they would not extend to human communicators. When a system actively reinforces that heuristic by claiming professional licensure, it is not just misleading users. It is weaponizing a known cognitive shortcut against them.
The Awareness-Capability Gap in Reverse
My dissertation research focuses on a specific puzzle: algorithmic awareness does not translate to improved outcomes for platform workers. People can know that an algorithm governs their environment without knowing how to respond effectively to it (Kellogg, Valentine, and Christin, 2020). The Pennsylvania case presents the inverse of this problem. Here, the question is not whether users are aware that they are interacting with an algorithm. It is whether the algorithm is accurately representing the nature of its own competence to those users.
This distinction matters for governance. Most AI literacy frameworks focus on building user-side schema - teaching people to understand what algorithmic systems can and cannot do. The implicit assumption is that the system itself is a neutral object to be understood. When systems actively misrepresent their own capabilities, user-side schema induction is insufficient as a governance mechanism. You cannot train your way out of a system that lies about itself.
What Organizational Theory Gets Right Here
Rahman (2021) describes how platform workers operate inside what he calls an "invisible cage" - a set of algorithmic constraints that shape behavior without being fully legible to the people inside them. The psychiatric chatbot case suggests a more acute version of this problem: a visible cage that is labeled incorrectly. The user can see the interface. They can read the claim to licensure. The cage is legible. But the label is false, and the user has no independent mechanism to verify it.
This is where Hatano and Inagaki's (1986) distinction between routine and adaptive expertise becomes relevant at the institutional level, not just the individual one. Regulators who apply routine expertise - existing frameworks for unlicensed medical practice - can address this particular case. But the structural feature, that AI systems can make authoritative self-representations that users lack the tools to contest, will generate new variants faster than case-by-case injunctions can address them. Adaptive institutional expertise requires developing schema for that structural feature, not just the topographic specifics of one chatbot in one jurisdiction.
The Broader Implication
Pennsylvania's injunction is the right move in the immediate term. But the case should be read as a data point in a larger pattern. As agentic AI systems become more capable of representing their own authority, credentials, and scope of practice, the governance question shifts from "can users understand these systems" to "can institutions verify what these systems claim about themselves." That is a different problem, and it requires different theoretical tools to address it.
Roger Hunt