Hospital AI data preparation: the work nobody budgets for

Hospital AI pilots do not fail in month six. They stall in week three, and they stall in the same place every time: the moment someone tries to join two systems on the patient and discovers there is no patient — there are four records, three spellings, two phone numbers and a unit that has been using a different registration format since it was acquired.

Nobody budgets for this. The business case has a line for the platform, a line for integration and a line for change management. It does not have a line for eighteen weeks of entity resolution, a data steward in each unit, and a clinical informatics person whose entire job for a quarter is arguing about what counts as a speciality. So that work gets done badly, by whoever is available, under time pressure, and the model is then asked to be intelligent about a dataset that cannot tell you how many patients you have.

I have paid for this lesson more than once. What follows is the work I would now put in the business case before the platform line, because it is the work that actually determines whether anything reaches production.

What week three actually looks like

The symptom is always a meeting that runs long. Someone has pulled a sample to train or evaluate something, and the sample does not reconcile. Appointment counts do not match the unit’s own MIS. The same doctor appears twice with slightly different names. A patient who clearly had a follow-up shows as two first-time visits. The analytics team says the data is fine and the unit says the data is wrong, and both are telling the truth about their own system.

At that point you have two options. Bolt a reconciliation layer onto the pilot so it can proceed, and accept that you have built something that will never generalise beyond the sample. Or stop, and do the entity work properly. The second is correct and nobody wants to hear it in week three, which is why you have to say it in week zero.

The master patient index you think you have

Most Indian hospital groups have a registration number that is unique within a unit and meaningless across units. The de facto cross-unit key is the mobile number, and it is a genuinely useful key — it is almost always captured, it is almost always correct, and patients remember it. It is also structurally unreliable in ways that matter:

One mobile number covers a family. A father, a mother, two children and a grandparent on one number is normal, and joining on it silently merges five people into one.
Elderly and paediatric patients are registered on an attendant’s number, which changes when the attendant changes.
Corporate and camp registrations carry an HR coordinator’s number across dozens of records.
Front desks under queue pressure enter a placeholder when a patient does not have a phone. Every unit has its own favourite placeholder, and you will find it in the top ten most frequent numbers in your database.

So the key has to be composite — number plus name similarity plus date of birth plus gender, with a deterministic pass for exact matches and a probabilistic pass with a human adjudication queue for the rest. Duplicate rates in large patient indexes commonly run from the high single digits to well over a tenth of records before anyone touches them; whatever yours is, you need to measure it before you promise anything that depends on patient history.

Duplicates, and why you cannot simply merge them

The naive plan is to find duplicates and merge. In a hospital you cannot, for three reasons that only become obvious once you try.

Clinical records cannot be merged casually. If you combine two records and one of them was the wrong patient, you have just put someone else’s allergy history, or their diagnosis, into a chart a clinician will act on. That is a patient safety event created by a data project. Merges on the clinical side need a defined adjudication process with clinical authority behind it, and they are slow by design.

Billing and insurance records have their own integrity. A claim is filed against a specific registration, a TPA has its own reference, and retrospectively restructuring the identity underneath a settled claim creates reconciliation problems in finance that you will hear about for a year.

And the units have reasons. A unit that maintains what looks like a duplicate is often tracking something real — a patient who is registered separately for a corporate health check and for their private treatment, deliberately, because the billing and the reporting are different.

The workable answer is a link, not a merge. Build a cross-reference layer that asserts “these registrations are the same person” with a confidence and a provenance, leave the source records untouched, and let analytical and AI workloads read through the link. Clinical merges proceed separately, slowly, with clinicians deciding. This single architectural decision is worth more than any model you will buy.

Doctor master data is worse than patient data

This is the one that surprises people. There are a few hundred doctors, not a few million records, so it feels like a small problem. It is not, because every downstream system has its own version of each doctor and none of them agree.

In one group I found the same consultant represented as four different names across the HIS, the website, the appointment system and an external aggregator listing — an initial in one, an expanded middle name in another, a qualification suffix in a third. Add to that: a registration number held in HR but not in the HIS, a speciality label chosen by the marketing team that does not match the department code, visiting arrangements at three units with different timings, and a profile photograph of the wrong vintage.

Anything you build on top inherits that mess. An enquiry agent cannot reliably tell a patient whether Dr A is available on Saturday if there are four Dr As. Your search and answer-engine visibility suffers for the same reason — entity disambiguation is exactly what those systems are doing, and you have given them four candidates.

Fix it with a doctor master that is the single source: one canonical name, one registration number, one primary and up to two secondary specialities from a controlled list, a unit-and-schedule table, and an owner who is accountable for it. The owner should sit in medical administration, not marketing, or the clinicians will not accept corrections.

Speciality taxonomies, and the three vocabularies you already run

Every group I have seen runs at least three parallel vocabularies and believes it runs one.

The HIS department codes, which exist for billing and rostering and are shaped by how the hospital is organised, not by medicine.
The clinical speciality and sub-speciality structure, which is what doctors recognise and what NABH documentation tends to reflect.
The patient-facing vocabulary on the website and in campaigns, which is shaped by what people actually search for — “knee replacement”, “child’s fever”, “heart blockage” — and maps to clinical specialities only loosely.

AI projects break on this because every one of them needs to map between the three. A booking flow needs patient language to resolve to a department code. An analytics model needs department codes to roll up to clinical specialities consistently across units that organised themselves differently. A content programme needs patient language to resolve to a reviewed clinical owner.

The artefact you need is unglamorous: a mapping table, maintained, versioned, with a named owner, and a rule that nothing downstream invents its own labels. Build it once, centrally, and enforce it. The alternative is that every project builds its own mapping in a spreadsheet, and two years later you cannot compare anything to anything.

HIS, LIS and RIS fragmentation across units

Multi-unit groups grow by building and by acquiring, and acquisition is where the fragmentation comes from. You will typically find several HIS instances of the same product on different versions with locally modified masters, at least one unit on something else entirely, a laboratory system with its own test codes, and a radiology system whose identifiers have no relationship to either.

The practical consequences for an AI programme:

Lab test codes are local. The same test has different codes, different names and different unit-of-measure conventions across units. Without a normalisation layer mapped to a standard vocabulary, nothing involving results is portable.
Reference ranges differ legitimately between laboratories. Any logic that reads a value as normal or abnormal has to carry the range with the value, not assume one.
Free-text fields carry the real information. Indication, clinical notes, and the reason for referral are where the clinically meaningful content sits, and their conventions are local to the unit and sometimes to the individual.
Integration surfaces are uneven. One unit has a usable interface engine, another has a nightly file drop, a third has a vendor who will quote for an API and take a quarter to deliver it.

Do not try to harmonise the source systems. That is a five-year programme and it is not your programme. Build a canonical layer — patients, doctors, encounters, orders, results, appointments — and onboard units onto it one at a time, accepting that the first unit will take three months and the fourth will take three weeks.

Consent and DPDP are build requirements, not a legal review

The most common sequencing error I see is treating consent as something legal signs off at the end. Under the DPDP framework you need a notice, a stated purpose, a consent record you can produce, a retention position and a withdrawal mechanism that actually works. Each of those is a feature somebody has to build, and retrofitting them is considerably more expensive than building them in.

What that means concretely:

Consent has to be captured per purpose, stored against the patient identity with a timestamp and the version of the notice shown. A tick box on a registration form from four years ago does not cover a new automated outreach programme.
Withdrawal must propagate. If a patient opts out, it has to reach the campaign tool, the contact centre’s dialler list, the recall engine and any vendor environment holding a copy. Most groups cannot do this on day one and discover it when someone complains twice.
Children’s data carries a higher bar, including verifiable parental consent. For a paediatric-heavy group this is not an edge case, it is the main case, and it reshapes how you design outreach.
Processor arrangements matter. If transcripts, notes or images sit in a vendor’s environment, the contract, the location, the retention and the access controls are your accountability, not theirs.
Using patient data to improve a model is a distinct purpose. Decide early, because deciding late means re-consenting or discarding.

What to actually budget

Mostly people and time, which is why it gets cut. The roles I would insist on:

A data steward in each major unit — usually an existing medical records or quality person with formally allocated time, not a new hire. They adjudicate identity conflicts and own the local masters.
One clinical informatics person centrally, with real clinical credibility, to own the taxonomies and the mapping table. Without clinical credibility they will not get decisions out of doctors.
An adjudication queue and the staffing to work it, permanently. Identity resolution is not a project, it is an operational function.
A data-quality dashboard with a handful of metrics — duplicate rate, records missing a usable contact, unmapped department codes, doctor-master mismatches — reported in the same pack as commercial performance.

That last point is the one that makes it stick. Data quality that is only visible to the data team gets deprioritised every quarter. Data quality on the same page as revenue gets fixed.

The order of operations

Measure before you promise. Duplicate rate, contactability, unmapped codes, doctor-master conflicts. Two weeks of work, and it reframes the business case honestly.
Build the doctor master first. It is small, it is high-leverage, it unblocks booking, enquiry and discovery, and it earns you clinical goodwill because doctors care about how they are represented.
Build the cross-reference identity layer next — link, do not merge — with a human adjudication queue from day one.
Publish the mapping table between the three vocabularies, with an owner, and make it mandatory.
Get consent architecture into the registration and digital flows before you launch any outreach that depends on it. Including withdrawal propagation.
Onboard units to the canonical layer one at a time, in order of data readiness, not in order of which unit head asked loudest.
Only then start the AI pilot — and start it on the unit whose data you just cleaned.

The model is the cheapest part of the programme. The patient index is the programme.

Questions people ask

Why do hospital AI pilots stall in week three?

Because that is when someone tries to join two systems on the patient and discovers there is no patient — there are four records, three spellings, two phone numbers and a unit using a different registration format since it was acquired. The sample does not reconcile with the unit’s MIS, the same doctor appears twice, and a follow-up shows as two first visits. The analytics team says the data is fine and the unit says it is wrong, and both are right about their own system.

What is a master patient index, and why doesn’t a hospital group have one?

A single identity for each patient that every system recognises. Most Indian hospital groups have a registration number unique within a unit and meaningless across units, so the de facto cross-unit key is the mobile number. It is genuinely useful and structurally unreliable: one number covers a family, elderly and paediatric patients sit on an attendant’s number, corporate camps carry an HR coordinator’s number across dozens of records, and every front desk has a favourite placeholder. The key has to be composite, with a human adjudication queue.

Why can’t a hospital simply merge duplicate patient records?

Three reasons that only become obvious once you try. Clinical records merged wrongly put someone else’s allergy history into a chart a clinician will act on — a patient safety event created by a data project. Billing and insurance records are filed against specific registrations, and restructuring identity under a settled claim creates finance problems for a year. And units keep some apparent duplicates deliberately, for corporate versus private billing. The workable answer is a link, not a merge: a cross-reference layer with confidence and provenance, source records untouched.

Why is doctor master data worse than patient data in a hospital?

It feels small — a few hundred doctors, not millions of records — but every downstream system holds its own version and none agree. I have found one consultant represented as four different names across the HIS, the website, the appointment system and an aggregator listing, with a speciality label chosen by marketing that does not match the department code. An enquiry agent cannot tell a patient whether Dr A is available on Saturday if there are four Dr As. Search and answer engines fail on the same ambiguity.

What are the three speciality vocabularies every hospital runs?

The HIS department codes, shaped by billing and rostering rather than medicine. The clinical speciality and sub-speciality structure that doctors recognise and NABH documentation reflects. And the patient-facing vocabulary — knee replacement, child’s fever, heart blockage — shaped by what people search for. Every AI project needs to map between all three: a booking flow resolves patient language to a department code, analytics rolls department codes up to clinical specialities across units organised differently. The artefact is a maintained, versioned mapping table with a named owner.

Should a hospital group harmonise its HIS systems before starting an AI project?

No. That is a five-year programme and it is not yours. Groups grow by acquisition, so you will find several HIS instances on different versions with locally modified masters, one unit on something else, a lab system with its own test codes and a radiology system related to neither. Build a canonical layer instead — patients, doctors, encounters, orders, results, appointments — and onboard units one at a time. The first unit takes three months; the fourth takes three weeks. Carry reference ranges with lab values rather than assuming one.

What does DPDP consent mean for a hospital AI project?

That consent is a build requirement, not a legal review at the end. You need a notice, a stated purpose, a consent record you can produce, a retention position and a withdrawal mechanism that works — each is a feature someone has to build. Consent is captured per purpose with a timestamp and notice version; a four-year-old tick box does not cover new outreach. Withdrawal must propagate to the campaign tool, the dialler and every vendor copy. Children’s data needs verifiable parental consent. Model improvement is a distinct purpose.

What should a hospital budget for data work before an AI pilot?

Mostly people and time, which is why it gets cut. A data steward in each major unit — usually an existing medical records or quality person with formally allocated time. One clinical informatics person centrally, with real clinical credibility, to own the taxonomies. An adjudication queue and the staffing to work it permanently, because identity resolution is an operational function, not a project. And a data-quality dashboard reported in the same pack as commercial performance. The model is the cheapest part of the programme.

Who should own the doctor master in a hospital group?

Medical administration, not marketing, or the clinicians will not accept corrections. The doctor master needs one canonical name, one registration number, one primary and up to two secondary specialities from a controlled list, a unit-and-schedule table, and a named owner accountable for it. Build it first: it is small, high-leverage, unblocks booking, enquiry and discovery, and earns clinical goodwill because doctors care how they are represented. It is the one piece of data work that clinicians will thank you for.

How long does hospital data preparation take before an AI pilot can start?

Measuring takes two weeks — duplicate rate, contactability, unmapped codes, doctor-master conflicts — and it reframes the business case honestly. Entity resolution done properly is more like eighteen weeks, and it is the line the business case never has. The doctor master comes first, then the cross-reference identity layer with its adjudication queue, then the published mapping table, then consent architecture in the registration and digital flows, then units onboarded one at a time in order of data readiness. Only then the pilot, on the unit you just cleaned.

How do you know the hospital’s data is ready for an AI project?

A handful of metrics on a dashboard that sits in the same pack as revenue: duplicate rate in the patient index, records missing a usable contact, unmapped department codes, and doctor-master mismatches. Data quality visible only to the data team gets deprioritised every quarter; data quality on the page next to commercial performance gets fixed. You are ready when those numbers are known and moving, a consent record can be produced for any patient, and the first unit reconciles to its own MIS.

Does a single hospital need this data work, or only multi-unit groups?

A single hospital escapes the cross-unit identity problem but not the rest. It still has a mobile number covering a family, a doctor represented differently on its website and its HIS, three speciality vocabularies and a consent form that predates any automated outreach. The work is smaller — one HIS, one set of masters — and the doctor master and mapping table can be done in weeks. The trap for a standalone unit is assuming that because the data is in one system, it is one dataset.

What should a hospital tell an AI vendor about its data before signing a contract?

The measured duplicate rate, the contactability, the number of unmapped codes and the state of the doctor master — before the vendor prices anything that depends on patient history. Also which unit will be onboarded first and how its integration surface actually works: interface engine, nightly file drop, or an API quoted for and delivered in a quarter. And the processor terms: if transcripts, notes or images sit in the vendor’s environment, the location, retention and access controls are the hospital’s accountability, not the vendor’s.