Total cost of ownership for AI systems

AI total cost of ownership is the build price plus everything it takes to keep the system correct, safe and running for its life: inference, retrieval infrastructure, monitoring, evaluation upkeep, the cost of changing models when providers change them, and the people who own it. Over three years the build is usually the smallest line. The budgets that go wrong are the ones that price the build carefully and treat the rest as a vague monthly figure. This article lists each line, explains what drives it, and shows how to build a TCO model that finance will accept, drawing on how we cost systems in AI product strategy engagements.

Why AI total cost of ownership is different from software TCO

Conventional software costs are dominated by build and licences, with maintenance as a predictable percentage. AI systems add three things. Usage cost scales with volume and with how much context each request carries, so it can rise faster than the business grows. Quality decays without attention: documents change, user behaviour shifts, and models are deprecated or updated by their providers, so the evaluation suite and the prompts need regular work. And correctness has to be checked continuously, which means monitoring and human review are operating costs, not project tasks. A TCO model that ignores those three will be wrong in the second quarter.

The lines in an AI TCO model

Inference and AI running costs

Inference cost is requests multiplied by tokens multiplied by price per token, and the middle term is where forecasts go wrong. A retrieval-backed request carries the retrieved passages and a system prompt on every call, so a chat message of twenty words can be a request of several thousand tokens. Multi-step agents make several model calls per user action. The forecast must be built from measured tokens per action in the pilot, at the model tier that passed the evals, with the caching rate observed. Model-agnostic routing, sending simple requests to cheaper models and reserving expensive ones for hard cases, is the main lever once the system is live; our guide to forecasting the monthly inference bill covers the method. For private deployments, inference becomes a fixed GPU line instead, sized as described in GPU sizing for private AI.

Retrieval, monitoring and the hidden costs of AI

Retrieval infrastructure includes the vector store, the pipeline that ingests and chunks documents, the embedding calls each time content changes, and storage. The refresh cost is the one that surprises: a knowledge base that changes daily re-embeds daily. Monitoring includes tracing every request with its prompt, response and cost, dashboards for quality and latency, alerting when quality drops, and the retention of those traces, which in regulated sectors can be years. The pilot probably used a free tier of a hosted tool; production volume, and the requirement to keep traces private, usually means a paid or self-hosted stack. Both lines are modest individually and both are omitted from most first budgets.

Evaluation upkeep

The evaluation suite is a living asset. New failure cases from production are added to the golden set, human reviewers score samples each week, and every prompt or model change is run against the suite before release. That is engineering and reviewer time every month, and it is the cost that keeps the system from decaying. Companies that cut it save a little and then pay for it in an incident. The practice is described in evals: the practice that separates AI demos from AI products.

Model changes: the cost nobody budgets

Providers deprecate models, change prices and update behaviour. Each event forces a re-evaluation and sometimes a prompt rewrite; a change of provider forces both plus integration work. Provider deprecation schedules are published, for example in OpenAI's model documentation, and a TCO model should assume at least one significant model change per year. Model-agnostic routing turns each change from a project into a configuration task with an eval run, which is the strongest cost argument for building the routing layer in from the start.

People: the largest recurring line

Every production AI system needs an owner who reads the dashboard, runs the weekly review of escalations and failures, prioritises changes and answers for the system's behaviour. It needs engineering time for those changes, whether in-house or through a support plan, and it needs reviewers for the cases the system escalates. These people usually already work for the company, which is why the line is treated as free; it is not. A fraction of a product manager, a fraction of an engineer and some reviewer hours, costed honestly, is often the largest recurring line and the one that most determines whether the system improves or decays.

Building the model finance will accept

Lay the lines out monthly over three years, with the build in month zero and a one-off re-platforming allowance in year two for a model change. Put the value line beneath it in the same units the process owner committed to: hours returned at a known cost, contacts resolved at a known cost per contact, revenue protected. Show the break-even month. Then run two sensitivities: volume double the forecast, and inference price double the forecast. If the system still pays under both, the case is robust. If it only pays under the base case, say so, and consider a cheaper model tier or a narrower scope. This is the format our AI development cost guide uses for the build side.

A worked example

A hospital network costing a multilingual voice agent for appointment handling had a first budget with two lines: the build and a per-minute usage estimate. The revised TCO model added telephony and speech costs per minute, the retrieval layer that held department and doctor schedules with its daily refresh, a self-hosted tracing stack because call transcripts could not leave the network, weekly reviewer time for escalated calls, an allowance for a speech or language model change each year, and a fraction of a patient-services manager as the system's owner. The recurring lines were several times what the first budget assumed, but so was the value once measured against the call-centre hours returned, and the case remained clearly positive. The build is described in our multilingual voice agent case study; the TCO model is what got it approved.

Team and timeline

A TCO model is a standard deliverable of a Sprint Zero, ten working days at $3,250 or ₹2,00,000, credited to the next build; it is prepared by a strategist and an architect with your finance contact and process owner over two working sessions. For an existing system, a TCO review takes about a week and often finds savings in model routing and retrieval refresh. The people line is typically covered by a Care Plan: Essential at $1,000 a month with ten engineering hours, Standard at $2,500 with twenty-five hours and 24×5 cover, or Enterprise at $5,250 with sixty hours, a named engineer and 24×7 cover, all listed on the pricing page so the recurring line can be written down before the build starts.

Before you start: a checklist

• Measured tokens per user action from the pilot, at the model tier that passed evals
• A volume forecast for peak, not just average, over three years
• The refresh frequency of every knowledge source the system retrieves from
• Trace retention requirements from compliance, and where traces may be stored
• A named owner and an honest estimate of their time
• Reviewer hours per week for escalations and evaluation samples
• An allowance for at least one model change per year
• The value line in the process owner's own units, with a break-even month

Glossary

• Inference cost: what the model calls cost, driven by requests, tokens per request and model tier
• Retrieval refresh: re-embedding and re-indexing content when sources change
• Evaluation upkeep: maintaining golden sets and running evals on every change
• Model change allowance: budget for re-evaluation and rework when a provider deprecates or reprices a model
• Owner: the person accountable for the system's behaviour and its weekly review
• Break-even month: the month in which cumulative value exceeds cumulative cost

Related reading

How much does an AI agent cost in 2026, what a care plan should cost and include, and multi-model routing to cut LLM costs.

Price the recurring lines as carefully as the build, put the value beneath them, and the decision to proceed becomes a calculation rather than a leap.