The situation
The client operates a UK digital health platform connecting users to healthcare and wellbeing support: practitioner directories, structured assessments, educational resources, and workflows on the professional side. The product was already built and in active development with real users; what did not exist was any AI capability, and the client had a long list of places they believed it would help: personalisation, navigation, practitioner matching, content recommendation, summarisation, search, administrative triage, and risk detection.
They were clear from the first call about the constraint. AI was not to replace healthcare professionals and was not to make diagnoses. Everything had to sit inside defined safety, governance and human-review boundaries. This is the rare case where the client stated the hard part correctly before we did.
What they had not yet worked out was the difference between the two questions that look identical on a feature list. "Can AI rank practitioners by fit" and "may AI decide which practitioner a person sees" describe the same screen and are completely different systems underneath.
Why off-the-shelf didn't fit
The client had scoped two alternatives before us. One was a general-purpose assistant bolted to the platform to handle navigation and questions. They abandoned it themselves, correctly, on a single test: asked a question that sat somewhere between "how do I book" and "should I be worried about this symptom," it answered the second question. There is no prompt that reliably fixes this in a product where users will not respect the boundary between platform help and health advice, because the user does not know where that boundary is and has no reason to.
The other was an off-the-shelf recommendation engine of the kind that works well for retail. The problem is structural rather than technical: a collaborative-filtering recommender infers things about a person from the behaviour of similar people, and in a healthcare context that inference is a clinical inference the system is not permitted to make. "Users like you also viewed" is an unremarkable sentence on a shopping site and an unsupported assertion about someone's health on this platform.
Neither vendor would carry any of the safety design. Both positioned the guardrails as the client's problem. In a regulated healthcare environment the guardrails are not a wrapper around the product, they are most of the engineering, and a supplier who treats them as out of scope is selling the easy 20%.
Scoping the real workflow
We spent the first session going through the client's feature list and sorting every item into three categories: things the model may decide, things the model may rank or draft with a human deciding, and things the model may not touch. Almost nothing landed in the first category, which was the right outcome and took some getting to.
Practitioner matching was the case that established the pattern. The client's own instinct, combining deterministic matching rules with AI-assisted ranking, was already correct, and the scoping work was making it precise. Eligibility is a rules question: does this practitioner hold the relevant expertise, are they available, do they deliver in the way this user needs, do they cover the location, do they hold the specialisms the user selected. That is a database query and it must stay one. Ordering the already-eligible set by fit is a judgement question, and that is where a model adds something. The invariant we wrote into the scope document is that the model can never introduce a practitioner the rules excluded, and can never remove one the rules included. It reorders a set it did not choose.
Risk detection was scoped against a different principle. Here the asymmetry is total: a false positive costs someone on the client's team a few minutes reviewing a conversation that turned out fine, and a false negative can mean a person in crisis is not seen. We scoped the deterministic layer as primary and independent, with the model additive only, never as a gate the model could close.
The scope came out as eight deliverables: a modular AI service layer, the matching architecture, hybrid semantic search, controlled summarisation, a strictly bounded support assistant, the risk-detection layer, the evaluation framework, and the audit and observability layer underneath all of it.
What we built
The AI layer sits behind a single internal service rather than being called from application code in a dozen places. That was a client requirement: models and providers should be swappable without redeveloping the platform. It also makes the evaluation framework enforceable, because every model call passes one boundary where it can be logged, validated and version-tagged.
Matching runs in two stages that cannot be collapsed. The deterministic filter produces the eligible set from the platform's own data. The ranker then orders that set, and is constrained to justify each placement using only fields the deterministic layer already holds: this practitioner ranks here because of this specialism, this delivery method, this availability. The user sees those reasons. A ranker that cannot say why is unusable in this context, and constraining the explanation to real fields also stopped it inventing rationales.
Search is hybrid, keyword and embedding retrieval combined, because pure vector search on healthcare vocabulary fails in a specific way: it happily returns semantically adjacent results for a term that has a precise clinical meaning, and adjacent is wrong when a user typed the exact name of a condition or a therapy.
Summarisation is rendered as a clearly labelled derived artefact, always displayed alongside the source rather than in place of it, and never written back over original healthcare data. The source record is the record; the summary is a view of it, and the interface says so.
The support assistant is scoped to the platform itself: how things work, what services exist, where to find something, how to book. It is not permitted to engage with clinical content, and when a question crosses that line it says so plainly and signposts, rather than deflecting in a way that leaves someone stuck.
Risk detection runs deterministic pattern matching and model classification as two independent paths joined by OR, never AND. A model failure cannot suppress a deterministic trigger, because the model is not in the path that would have to fail open. Everything either path raises goes to a human queue with the triggering content and the reason, and nothing auto-resolves.
Underneath: audit logging of every model call with prompt version, model version and output; structured output with schema validation on anything the application consumes; and a regression suite that runs on every prompt or model change.
Where it got hard
We tuned the risk layer the wrong way round to begin with. Our initial threshold was set to keep false positives manageable, which is the instinct any engineer brings from every other domain, and during clinician review of a held-out sample one of the client's practitioners identified a phrasing the system had not flagged. It was indirect, matched no pattern, and was the sort of thing a trained human recognises immediately. Nobody was harmed; it was a review set. But it made the tuning question concrete in a way the abstract discussion had not: we had been optimising the metric that was cheap to improve rather than the one that mattered. We re-tuned for recall, accepted a considerably higher false-positive rate as the correct trade, and stopped treating the resulting review volume as an engineering problem to be optimised away.
Prompt injection arrived through a route the client had not considered, and neither had we at scope time. Users enter free text into assessments, and some of that text is summarised. Content a user typed was therefore reaching a model in a position where instructions are honoured, meaning a user could, in principle, write text that steered the summary a professional would later read. The fix was structural: user content is never placed where instructions are read, retrieved material is passed as clearly delimited data, outputs are schema-validated before the application uses them, and we red-team the summarisation path specifically as part of the regression suite.
A smaller problem shaped the architecture. The requirement to swap providers without redeveloping ran straight into the reality that prompts tuned against one model regress against another, sometimes badly. Modularity at the code level does not give you modularity in behaviour. What actually delivers the client's requirement is the evaluation suite: a provider change is now a test run with a pass mark, not a judgement call, and the abstraction layer is only useful because that suite exists behind it.
Rollout & results
The layer shipped in week seven and went live feature by feature rather than at once, with search and the support assistant first, matching second, and risk detection last and in shadow for a further two weeks, running and generating a review queue that was worked in parallel with existing processes, so the threshold could be set against real volume rather than an estimate.
The invariant has held: no practitioner has appeared in a user's results who was not in the deterministic eligible set. The client's clinical governance lead asked for that number first and checks it monthly. Every risk escalation from either path has reached a human reviewer, with none auto-resolved and none dropped. Search abandonment fell meaningfully with the hybrid retrieval, which was the least interesting change to build and one of the most visible to users.
The output the client says they use most is not a feature. It is the evaluation framework: accuracy, retrieval quality, hallucination rate, false-negative rate, latency and cost per interaction, run as a suite. It converts "is the AI working" from a question people argue about in a meeting into a number with a threshold.
"We asked three suppliers how they'd stop the model giving clinical advice. Two answered with a prompt. The answer we wanted was an architecture where it never gets the chance." Clinical Governance Lead, client engagement
What we'd do differently
We would model the operational cost of the risk threshold with the client before choosing it, not after. Re-tuning for recall was unambiguously the right engineering decision and we would make it again, but it roughly tripled the human review volume, and that landed on a team whose staffing plan had been built against our earlier, quieter numbers. The threshold was presented as a safety decision, which it is, but it is also a staffing decision, and we let the client discover the second half of that after the fact. On any engagement where a safety layer generates human work, the alert volume at each candidate threshold now goes in the scope document next to the accuracy figures.
We would also have brought the clinician review forward. It was scheduled as validation near the end and it functioned as design input. The missed phrasing changed the architecture, not just a parameter. Expert review of a held-out sample is cheap and it belongs in week two, where it can still change what gets built.