The Incident and What It Actually Signals
Health New Zealand recently instructed staff to stop using free AI tools, specifically ChatGPT, to write clinical notes. The directive followed reports that workers had been caught using consumer-grade generative AI to assist with documentation that enters the formal medical record. The policy response was swift and largely procedural: stop doing this. What the response did not address is the organizational condition that produced the behavior in the first place. Staff were not using ChatGPT because they lacked ethics training. They were using it because it worked well enough, it was free, and no sanctioned alternative was immediately available. That gap between institutional prohibition and practical need is where most AI governance failures actually live.
The Competence Assumption Problem in AI Policy
Most institutional AI policies are written as if the target population already possesses a stable, accurate mental model of what AI tools do, how outputs are generated, and where failure modes cluster. Health NZ's directive presupposes that staff understand why free consumer tools are inappropriate for clinical documentation, specifically the risks around hallucination, data residency, and audit trail integrity. But research on algorithmic literacy consistently shows that awareness of a technology does not translate into accurate structural understanding of that technology (Kellogg, Valentine, & Christin, 2020). A clinician who knows that "AI can make things up" holds a folk theory, not a schema. Folk theories are anecdotal and impressionistic; they do not generalize reliably to novel risk scenarios. When the next tool appears, the folk theory does not transfer.
This is the awareness-capability gap applied to compliance rather than performance. Telling staff to stop using ChatGPT addresses the topography of the problem - this specific tool, in this specific context. It does nothing to address the topology, meaning the structural features of AI-mediated documentation risk that would allow a clinician to evaluate the next tool, or the one after that, without waiting for another institutional prohibition.
Why Prohibition Is a Procedural Response to a Schema Problem
Hatano and Inagaki (1986) drew a distinction between routine expertise and adaptive expertise. Routine expertise produces correct behavior under familiar conditions. Adaptive expertise produces correct behavior under novel conditions by applying underlying principles rather than recalled procedures. Health NZ's response trains routine expertise: do not use ChatGPT for clinical notes. It does not build adaptive expertise: here is how to evaluate whether any AI tool is appropriate for clinical documentation, based on the structural features that make such tools risky or reliable.
The distinction matters enormously in healthcare contexts because the tool landscape is not static. Vendors are actively positioning AI-assisted documentation as a productivity solution for clinical teams. Epic, for example, has integrated ambient AI note generation directly into its workflow. Staff who received only a prohibition on ChatGPT have no principled basis for evaluating whether the Epic integration carries the same risks, different risks, or no comparable risks. The procedural training does not transfer to the structurally similar but superficially different case (Gentner, 1983).
Organizational Governance and the Schema Induction Alternative
The more productive policy intervention would target schema induction rather than behavioral prohibition. This means training staff on the structural features that determine AI tool appropriateness in clinical contexts: data routing and storage, output provenance and auditability, model hallucination rates in low-frequency clinical domains, and liability implications of AI-generated text in the formal record. These structural features are stable across specific tools. A clinician who understands them can evaluate ChatGPT, an EHR-integrated assistant, and whatever comes next, using the same principled framework.
This is not a novel claim in the abstract, but institutional AI governance consistently ignores it in practice. Hancock, Naaman, and Levy (2020) argued that AI-mediated communication changes the conditions under which human judgment operates, often without making those changes legible to the humans involved. Clinical documentation is precisely this kind of AI-mediated communicative act: the output enters a record that shapes downstream clinical decisions, legal liability, and billing, yet the process by which it was generated may be invisible to subsequent readers. Governing this well requires that staff understand the communicative and evidentiary structure of the record, not merely which tools are prohibited.
What the Policy Failure Actually Costs
Health NZ's prohibition will reduce ChatGPT use among staff who follow it. It will not reduce the underlying pressure to complete documentation efficiently, and it will not equip staff to navigate the AI tools that healthcare organizations will continue to introduce through official channels. The cost of the competence assumption in this case is not just a compliance gap. It is the organizational condition in which workers lack the structural literacy to distinguish sanctioned AI risk from unsanctioned AI risk, and in which institutions respond to each instance reactively rather than building the evaluative capacity that would make reactive governance unnecessary.
References
Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7(2), 155-170.
Hancock, J. T., Naaman, M., & Levy, K. (2020). AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication, 25(1), 89-100.
Hatano, G., & Inagaki, K. (1986). Two courses of expertise. In H. Stevenson, H. Azuma, & K. Hakuta (Eds.), Child development and education in Japan (pp. 262-272). Freeman.
Kellogg, K. C., Valentine, M. A., & Christin, A. (2020). Algorithms at work: The new contested terrain of control. Academy of Management Annals, 14(1), 366-410.
Roger Hunt