Patient-facing AI agent in a hospital: no-incident launch

The question that gets asked in the steering committee is “how accurate is it”. That is the wrong question, and the first time I answered it honestly I lost the room for a month. Accuracy is a distribution. What the committee actually needs to know is the shape of the tail — what happens on the one conversation in two thousand where a patient describes something urgent and the system responds pleasantly instead of usefully.

A patient-facing AI agent in a hospital is not a content problem or a model problem. It is a clinical-risk surface that happens to be built out of software. Once you accept that, the design work becomes obvious and unglamorous: narrow the scope, write down what must never be answered, make escalation the default rather than the exception, log everything, and put a clinician’s name on the sign-off.

I have launched these and I have switched one off. Here is what I would insist on before any model speaks to a patient under your group’s name.

Decide what an incident is before you launch

Most teams cannot define failure, which is why they cannot design for it. Write the definition down in the first week, in plain language, and get your medical director to agree to it. Mine has four tiers.

Clinical incident — the agent gave advice, reassurance, a dosage, an interpretation, or a delay that could affect a patient’s health outcome. This is reportable internally on the same pathway as any other clinical event.
Safety near-miss — the agent should have escalated and did not, but no harm resulted because the patient did something sensible anyway. These are the ones you will learn from, and they only surface if you read transcripts.
Privacy incident — the agent exposed one patient’s information to another person, or persisted data it was not supposed to hold. Treat this with the same seriousness as the clinical tier, because the regulator will.
Service failure — wrong timing, wrong location, failed handover, dead end. Annoying, commercially costly, not a clinical matter.

That taxonomy changes behaviour. Teams that only track “accuracy” optimise the mean. Teams that report near-misses weekly start redesigning the escalation path, which is where the actual safety lives.

Scope by allow-list, never by deny-list

The instinct is to build a capable agent and then block the dangerous topics. That inverts the risk. Language is generative; your block list is finite. A patient will phrase something in transliterated Hindi with a spelling you did not anticipate and your filter will wave it through.

Build the opposite way. The agent answers a defined, enumerated set of intents and nothing else. In a hospital group that list is short and genuinely useful: doctor availability and timings by unit, speciality and sub-speciality mapping to a named list of doctors, directions and access, visiting hours, which insurers and TPAs are empanelled at which unit, what documents to bring, preparation for named diagnostic tests, and status of an appointment the patient already holds.

Anything outside that list gets one behaviour: a short acknowledgement and a handover. Not a guess, not a hedged answer, not a “generally speaking”. The hardest part of the build is resisting the pressure to widen the list, and that pressure will come from commercial teams who see the handover rate and read it as a failure.

The never-answer list, and whose signature goes on it

One page. Printed. Reviewed every quarter. On mine, the agent must never:

Interpret a symptom, a symptom cluster, or a duration. Including reassurance — “that sounds normal” is the single most dangerous sentence the system can produce.
Interpret a report, a value, a scan, or a discharge summary, even when it has the document in front of it.
Discuss medication: names, doses, timings, substitutions, interactions, whether to continue, whether to stop.
Recommend a specific doctor as better or more suitable than another.
Quote a price as a commitment rather than an indicative range, or state what insurance will cover for this patient.
Answer anything about a patient under a defined paediatric age threshold where a symptom is mentioned. Straight to a human, every time.
Answer when the conversation contains an acuity signal — chest pain, breathlessness, bleeding, unconsciousness, seizure, a fall in an elderly patient, anything post-operative — in any language or spelling the group serves.
Claim a cure, a guarantee, or a success rate for any procedure.

Three signatures: the medical director for the clinical boundaries, the data protection officer or whoever holds that accountability for the privacy boundaries, and you for the commercial consequences — because the handover volume and the lost conversions are your problem, and the sign-off should make that explicit rather than letting the clinical side carry it alone.

Escalation is the product

I have come to think of the agent as a triage and routing layer with a conversational front, not as an answering machine. Designed that way, the engineering effort goes where it matters.

Acuity detection runs first, before intent classification, on every incoming turn. It is deliberately over-sensitive; a false escalation costs you a contact-centre minute, a missed one costs you a patient. It runs in every language and script your group receives — and in India that means Hinglish, transliterated regional languages, and code-switched sentences where the verb is in one language and the symptom in another. Test it in those forms, not in clean English, because clean English is not what arrives.

Then the handover itself. These are the details that decide whether the whole thing is safe:

Handover is warm. The agent passes the transcript, so the patient does not repeat themselves. An escalation that drops the patient into a new queue with no context is worse than no bot.
Handover has a time guarantee and a fallback. If no human picks up within a defined window, the agent gives the emergency number and the nearest unit’s address, and says plainly that it cannot help with this.
Out-of-hours behaviour is designed separately. Most clinical near-misses I have seen happened after eleven at night, when the agent was the only thing awake. If you cannot staff a clinical escalation at that hour, the agent’s after-hours scope should be narrower than its daytime scope, and on the acuity path it should do nothing but give the emergency number.
The agent never apologises its way out. “I’m sorry, I can’t help with that” with no route attached is how a patient with a real problem gets abandoned politely.

Guardrails that hold, and guardrails that only look like they do

Prompt instructions are not a guardrail. They are a preference. Anything you genuinely cannot allow has to be enforced outside the model.

What holds: a deterministic classifier gating the clinical path; retrieval restricted to a reviewed corpus with no free generation on clinical content; a response template library for the high-risk intents so the wording is fixed and was approved by a clinician; a refusal path that cannot be talked out of by the patient, however the request is rephrased; and an output check that blocks any response containing a drug name, a numeric dose, or a diagnostic term from a maintained vocabulary.

What does not hold: confidence thresholds on the model’s own output, disclaimers appended to clinical answers, and the belief that because the corpus is yours the answer must be safe. A correctly retrieved sentence from your own discharge protocol can still be the wrong sentence for this patient, and a disclaimer does not transfer that risk to them. It only documents that you knew.

The medico-legal picture in India, soberly

I am not a lawyer and you should get your own view, but the contours matter for design. Telemedicine practice in India is framed around a registered medical practitioner being accountable for the consultation. An autonomous system giving what a reasonable patient would read as medical advice does not sit comfortably inside that frame — the accountability has nowhere to land, which in practice means it lands on the institution.

Consumer protection law treats a hospital’s service as a service, and a deficiency in it as actionable. A transcript in which your branded agent reassured a patient who later deteriorated is an unusually clean piece of evidence. There is no model explanation that reads well in that context.

On data, the DPDP regime pushes you towards purpose limitation, stated retention, and a consent record you can actually produce. That has concrete build consequences: conversation logs contain health data; they cannot sit in a vendor’s environment indefinitely with informal access; and if transcripts are used to improve the system, that is a separate purpose and needs to be handled as one.

Two practical consequences. First, disclose clearly and early that the patient is talking to an automated assistant, in the language of the conversation — not in a footer. Second, make sure every clinical-path conversation is either handled by a human or demonstrably refused. The defensible position is that the system routed, and a clinician decided.

Logging, because you will be asked to reconstruct a conversation

Assume that within the first year someone — a family, a unit head, a regulator, your own quality team — will ask exactly what the agent said to one patient on one night. If you cannot answer in an hour, you have a second problem on top of the first.

Log the full turn-by-turn transcript with timestamps, the channel, the language detected, the intent classified, the acuity decision and why, every retrieved document with its version, the template used, whether escalation fired, who picked up and when, and the outcome. Version everything — a prompt change, a corpus change, a template change — so that you can say what the system was when it said what it said.

Then control who can read it. These transcripts are health records in all but name. Role-based access, an access log on the access log, a stated retention period, and no analytics pipeline quietly copying them into a general marketing warehouse. That last one is the mistake I see most often, and it is made by good teams who simply did not think of a chat log as clinical data.

The review loop, and why it must be boring

Weekly, for the first six months. One clinician, one contact-centre supervisor, one person from your team. Ninety minutes. They read a stratified sample: every escalated conversation involving an acuity signal, every refusal, every conversation that ended without resolution, and a random sample of the ordinary ones. The clinician’s only job is to mark near-misses.

Two things make this work. The sample is stratified, not random — random sampling of a mostly-fine population finds nothing. And the output is a change, logged, with an owner and a date. A review meeting that produces observations rather than changes to the allow-list, the acuity rules or the templates will be cancelled by month three, and it should be.

After six months you can move to fortnightly, keep the acuity-escalation review at full coverage, and publish a one-page monthly note to the quality committee. That note is what buys you the licence to widen scope later.

If you’re starting this next quarter

Write the incident definition and the never-answer list. Get three signatures. Before any build.
Name the clinical owner — a practising clinician, not a committee, with time formally allocated. If nobody will take this, you do not have a programme, you have a pilot that will be switched off.
Build the escalation path first, with a warm handover and a staffed window, and run it with humans only. If your handover is not working without the agent, adding the agent makes it worse.
Build acuity detection second, and test it in Hinglish and transliterated regional language before English. Tune it towards over-escalation.
Launch on one channel, one unit, daytime hours, with the allow-list at its narrowest. Two weeks.
Run the weekly review from week one, even when there is almost nothing to read. The habit is the control.
Widen scope only on the evidence of that review, one intent at a time, and never on the evidence of containment rate.

The agent’s job is to know what it must not touch. Everything else is decoration.

Questions people ask

What is a patient-facing AI agent in a hospital, and how is it different from a chatbot?

It is any automated system that talks to patients under the hospital’s name — on the website, a messaging app or the contact centre — and answers rather than just routing. The difference from an old chatbot is that a language model will generate a plausible answer to almost anything, including questions it must never answer. That makes it a clinical-risk surface that happens to be built from software, not a content problem or a model problem. Design it as triage and routing with a conversational front.

What counts as an incident when a hospital AI agent goes wrong?

Define it before launch, in plain language, with the medical director’s agreement. I use four tiers: a clinical incident, where the agent gave advice, reassurance, a dosage or an interpretation that could affect an outcome; a safety near-miss, where it should have escalated and did not but no harm resulted; a privacy incident, where one patient’s data reached another person; and a service failure — wrong timing, dead end. Teams that only track accuracy optimise the mean. Teams that report near-misses redesign escalation.

Why should a hospital AI agent be scoped by allow-list rather than by blocking dangerous topics?

Because language is generative and a block list is finite. A patient will phrase something in transliterated Hindi with a spelling nobody anticipated and the filter will wave it through. Build the opposite way: the agent answers an enumerated set of intents — doctor timings by unit, empanelled insurers and TPAs, directions, visiting hours, test preparation, appointment status — and hands over everything else with a short acknowledgement. The hardest part is resisting pressure from commercial teams who read the handover rate as failure.

What should a hospital AI assistant never answer?

Symptoms, symptom clusters or durations, including reassurance — “that sounds normal” is the most dangerous sentence it can produce. Reports, values, scans or discharge summaries, even with the document in front of it. Medication of any kind. Which doctor is better. Prices as commitments, or what insurance will cover. Anything involving a young child where a symptom is mentioned. Any conversation carrying an acuity signal such as chest pain or breathlessness, in any language. Cures, guarantees or success rates. Put it on one printed page.

Who signs off on a patient-facing AI agent before launch in a hospital?

Three signatures on the never-answer list. The medical director for the clinical boundaries. The data protection officer, or whoever holds that accountability, for the privacy boundaries. And the growth or digital head for the commercial consequences — because handover volume and lost conversions are that person’s problem, and the sign-off should make it explicit rather than letting the clinical side carry the risk alone. Beyond signatures, name a practising clinician as owner with time formally allocated. Without one you have a pilot that will be switched off.

How long does it take to launch an AI agent safely in an Indian hospital?

Longer than the vendor demo suggests, and most of the time goes into things other than the model. Writing the incident definition and never-answer list comes before any build. The escalation path is built and run with humans only first. Acuity detection is tested in Hinglish and transliterated regional languages before English. Then a two-week launch on one channel, one unit, daytime hours, narrowest scope. Widening happens one intent at a time on evidence from weekly review, over months.

What does a hospital AI agent cost, and where does the money actually go?

The model licence is the smallest line. The real costs are a deterministic classifier gating the clinical path, a reviewed corpus with no free generation on clinical content, a template library approved by clinicians, an output check against a maintained vocabulary of drug names and diagnostic terms, and full versioned logging with role-based access. Then the ongoing cost: ninety minutes a week of a clinician, a contact-centre supervisor and a digital lead reading transcripts for six months. Budget for the review, not just the build.

How should escalation from an AI agent to a human work in a hospital contact centre?

Escalation is the product. Acuity detection runs on every turn before intent classification and is tuned to over-escalate, because a false escalation costs a contact-centre minute and a missed one costs a patient. Handover is warm — the transcript travels with the patient. There is a time guarantee and a fallback: if no human picks up in the window, the agent gives the emergency number and nearest unit. Out-of-hours scope is narrower, because most near-misses happen after eleven at night.

Are prompt instructions and disclaimers enough of a guardrail for clinical AI?

No. Prompt instructions are a preference, not a control; anything you genuinely cannot allow has to be enforced outside the model. What holds is a deterministic classifier, restricted retrieval, fixed templates for high-risk intents, a refusal path the patient cannot talk their way around, and an output block on drug names and doses. What does not hold is confidence thresholds on the model’s own output and disclaimers appended to clinical answers. A disclaimer does not transfer risk to the patient. It only documents that you knew.

What is the medico-legal risk of a hospital AI agent in India?

I am not a lawyer, but the contours shape the design. Telemedicine practice is framed around a registered practitioner being accountable; an autonomous system giving what reads as medical advice leaves that accountability with the institution. Consumer protection law treats a deficiency in hospital service as actionable, and a transcript in which your branded agent reassured a patient who later deteriorated is unusually clean evidence. Disclose early that it is automated, in the conversation’s language, and make every clinical-path conversation either human-handled or demonstrably refused.

How does DPDP affect the data an AI agent collects from patients?

Conversation logs contain health data, so purpose limitation, stated retention and a producible consent record all apply. Practically: transcripts cannot sit in a vendor’s environment indefinitely with informal access; using them to improve the system is a separate purpose and must be handled as one; and access needs role-based control with its own audit log. The mistake I see most often is an analytics pipeline quietly copying chat logs into a general marketing warehouse, made by good teams who did not think of a chat log as clinical data.

How do you know a patient-facing AI agent is working, beyond accuracy?

Accuracy is a distribution; what matters is the tail. Run a weekly ninety-minute review for the first six months with one clinician, one contact-centre supervisor and one digital person reading a stratified sample — every acuity escalation, every refusal, every unresolved conversation, plus a random slice. The clinician marks near-misses. The output must be a logged change to the allow-list, acuity rules or templates, with an owner and date. A monthly one-page note to the quality committee is what earns the licence to widen scope.

When should a hospital not put an AI agent in front of patients at all?

When the human escalation path does not work without the agent — adding one makes it worse. When no practising clinician will take ownership with allocated time. When the contact centre cannot staff a clinical escalation window, especially after hours. When commercial teams will judge the launch on containment rate and push to widen scope on that evidence. And when the group cannot reconstruct what the agent said to one patient on one night within an hour. I have switched one off, and I would again.