Context
The client runs a network of multi-specialty hospitals and clinics in southern India. Each site has a front desk that handles walk-ins, phone calls and the hospital information system at the same time. The phones win: between eight and eleven in the morning, calls to book, confirm or move appointments outnumber everything else, and a patient standing at the counter waits while the staff member answers a call about a slot next Tuesday. Missed calls went to voicemail nobody had time to return, and a meaningful share of appointments were no-shows that a reminder call would have caught.
The network had tried an IVR. Patients pressed zero. The languages spoken across the sites, Kannada, Hindi, English and a good deal of switching between them mid-sentence, made off-the-shelf voice bots unreliable, and the operations team had understandably low expectations.
The problem to solve
The brief was narrow and measurable: take the routine appointment calls, booking, rescheduling, cancellation, confirmation and reminders, in the languages patients actually use, and take the action in the hospital information system during the call so that nothing is written down and re-keyed. Anything outside that, clinical questions, billing disputes, emergencies, had to reach a human quickly and cleanly. Patient data had to stay within the network's environment.
Approach
We began with the languages, because that was where previous attempts had failed. In the first two weeks we collected a consented sample of real call recordings across the sites and benchmarked speech-to-text engines per language, including on the Kannada-English switching that dominated the sample. No single engine won across the board, so the system routes each call to the engine that performs best for its detected language, and the text-to-speech side was chosen the same way, by listening tests with front-desk staff rather than by vendor claims.
- Real-time voice pipeline on LiveKit with sub-second turn-taking, connected to the network's existing SIP trunk so no phone numbers changed.
- Language detection in the first seconds of a call, with the ability to switch mid-conversation when the caller does.
- A wrapper API over the hospital information system's scheduling module, since the HIS had no clean external API, with strict scopes: read availability, create, move and cancel appointments, nothing else.
- Identity confirmation by phone number and date of birth before any appointment detail is read out, and no clinical information ever spoken by the agent.
- Warm hand-off to the site's front desk with a summary of the call so far, and an emergency phrase list that transfers immediately.
- Outbound reminder calls the day before, with one-touch confirmation or rescheduling.
What we built
The agent answers the main line at each site. It greets in the site's default language, switches on detection, confirms the caller's identity and handles the appointment request against live availability, reading back the confirmed slot and sending an SMS. Reminder calls go out each evening for the next day's appointments; a caller who cannot make it is offered the next available slots in the same call. Every call is transcribed and stored inside the network's environment with the actions taken, and a dashboard shows volumes, resolution, hand-off reasons and per-language speech accuracy.
Because the HIS integration was the riskiest part, we built it first and tested it against a staging copy of the scheduling module with the hospital's IT team before the voice layer was attached. The wrapper enforces the scopes, logs every call and has a kill switch the operations team can flip from the dashboard.
Delivery
One site went live first, taking a fixed share of calls with the rest still ringing at the desk, for three weeks. Front-desk staff listened to recordings daily and flagged anything odd; most of what they flagged in week one was pronunciation of doctor names, which we fixed with a pronunciation dictionary the sites now maintain themselves. The share rose to all routine calls at that site by week six, and the remaining sites were added one per week, each with its own doctor-name dictionary and default language.
Outcomes
Across the network the agent now handles the large majority of routine appointment calls without a human, and the front desks report that the morning queue is visibly shorter. Missed calls during peak hours dropped to near zero because the agent answers on the first ring. Reminder calls with in-call rescheduling reduced the no-show rate the operations team tracks, and the slots freed by cancellations are re-offered the same day. Per-language speech accuracy is reported weekly and has improved as the dictionaries and the sample have grown.
No patient data left the network's environment, and the hospital's information security lead signed off the audit log design before the first site went live.
What we measured
- Routine calls resolved without a human, by site and language
- Missed calls during peak hours
- No-show rate before and after reminder calls
- Average handling time
- Hand-off reasons
- Speech accuracy per language, weekly
Architecture in brief
Calls arrive over the network's existing SIP trunk into a LiveKit media server running inside the hospital's cloud environment. The voice pipeline detects language, streams audio to the chosen speech-to-text engine, passes the transcript through a small dialogue model with a tightly scoped tool set, and streams synthesised speech back with a turn-taking budget of well under a second. The scheduling tools call the HIS wrapper, which enforces scopes and logs every request. Transcripts, actions and audio are stored in the network's environment with retention rules set by their information security lead. A dashboard, built in React on the same data, shows the operations team volumes, resolution and hand-off reasons per site.
Everything the front desks can change themselves, doctor-name pronunciations, site-level default language, opening hours and the emergency phrase list, lives in a settings screen rather than in code, which is why the rollout to further sites needed no engineering time.
What we learned
Benchmarking speech engines per language on the network's own recordings, rather than trusting any single vendor, was the difference between this attempt and the previous ones. The second lesson was to build the boring integration first: the HIS wrapper took longer than the voice pipeline and was the part the hospital's IT team cared most about. The third was to give the front-desk staff a way to fix the small things themselves, which turned them from sceptics into owners.
The network is on an Enterprise Care Plan with twenty-four-hour cover, since the phones do not stop.
"Our front desks used to lose the phone battle every morning. The agent takes the routine calls in the patient's language, and our staff talk to the people who are actually standing in front of them."
Client details are anonymised. Figures are described qualitatively and are available under NDA on a call.