Questions to ask an AI proof of concept development vendor before you sign

Ask an AI proof of concept development vendor five things before anything else: what the written pass thresholds are, who builds the evaluation set, whose accounts pay for inference, who owns the code and prompts afterwards, and exactly what happens if the result fails. Vendors who have shipped answer all five in specifics. Vendors who demo answer in adjectives.

Below are the questions we would ask if we were buying rather than selling, grouped by what each answer actually reveals, with the strong answer and the warning sign side by side. Take the checklist at the end into the call.

What a good answer sounds like

A proof of concept is a measurement exercise, so the quality of a vendor is visible in how precisely they talk about measurement. Someone who has run these before will reach for numbers unprompted: sample sizes, threshold percentages, latency budgets, cost per document. Someone who has not will reach for capability claims and a list of logos.

The second tell is willingness to describe failure. A vendor who says a proof of concept can fail, and can tell you what their last failed one taught the client, is describing a real practice. A vendor who treats failure as an unlikely edge case is selling a demo with a longer timeline. Our stance on this is set out in evals over demos.

The third tell is boundaries. Good vendors are specific about what is not included, because they have been burned by ambiguity. If everything you suggest is waved through as in scope, the scope is not real and the argument is merely deferred to week three.

What should you ask an AI proof of concept development vendor?

Start with the eight questions below. Each one has a distinguishing answer, which is what makes it worth asking rather than a question every vendor answers the same way.

Questions about evidence and evaluation

Push hardest here, because evaluation is the part that is easiest to skip and most expensive to add later. Ask how the vendor decides an answer is correct when correctness is a judgement rather than a string match, and listen for whether a human adjudicates or a model grades itself. Ask what the baseline is, because an improvement claim with no baseline is a sentence, not a result. An evaluation suite that runs on every change is the difference between a measured system and an anecdote.

Then ask for a walk-through of a previous evaluation report with the client details removed. You are not checking the numbers. You are checking that the artefact exists and that it contains failure analysis rather than a chart of wins. A vendor who cannot produce one has probably never had to defend a result to a sceptical CTO.

Questions about ownership, security and exit

Ask who holds the repository during the engagement, not just after it. Ask whether prompts, retrieval configuration and the eval harness transfer with the code, because a prompt library left behind is the most valuable artefact in a small AI project. At Eazyware you own the code, infrastructure, prompts, model choices and documentation, which is the position argued in who owns the code, prompts and models.

On security, ask where your data will sit during the three weeks, whether any of it leaves your region, whether a data processing agreement is signed before the first working session, and which sub-processors are involved. Under India's DPDP Act 2023 you remain the data fiduciary regardless of who does the engineering, so a vague answer here is your regulatory problem, not the vendor's. The security questionnaire for AI vendors has the full list worth sending in advance.

Questions about money

Ask for the price as a fixed figure with a named scope, not a day rate. Eazyware's three-week AI POC Sprint starts at $6,250 or ₹4,00,000 and runs to $10,500 or ₹6,80,000 depending on data complexity and how many models are benchmarked. A ten-day AI Discovery Sprint at $3,250 or ₹2,00,000 is credited to the next build if you need the scope decided first. Published starting figures sit on the pricing page, and what a proper fixed quote should contain is covered in what a fixed-price AI quote should contain.

Then ask about the money after the proof of concept. What does the build cost if this passes, what does a monthly Care Plan cost, and what is the re-evaluation cost when a model version is deprecated. Care Plans start at $1,000 or ₹68,000 per month, with an AI system add-on at $750 or ₹40,000 covering evals, cost monitoring and prompt regression. A vendor who has never been asked this has never supported a system past launch.

The checklist to take into the call

• Name the hypothesis. Make the vendor state, in one sentence, what this proof of concept will prove or disprove.
• Fix the thresholds. Accuracy, latency and cost per task, with numbers, before tuning starts.
• Confirm the sample. How many records, which edge cases, who exports them and by when.
• Settle the accounts. Whose API keys, what budget ceiling, which dashboard shows spend.
• Check the artefacts. Repository, evaluation report, production readiness assessment, updated proposal.
• Test the no. Ask when they have told a client that AI was the wrong answer, and what happened.
• Meet the engineers. The people in the room during the sale should be the people doing the work.
• Agree the exit. What you keep, what you owe and what happens next if the answer is no.

Questions that waste your time

Some due diligence questions feel rigorous and tell you nothing. Asking which model a vendor uses is one of them, because the honest answer is whichever benchmarks best for your task across OpenAI, Anthropic, Google, Meta, Mistral and open-weight options, and most production systems route between two or three anyway. A vendor committed to one model before seeing your data has a partnership, not an opinion.

Asking for headcount or years in business is similarly weak. A three-week proof of concept is done by two or three people, and what matters is whether those specific people have measured accuracy on messy data before. Ask about the team assigned, not the company. If you are weighing an external partner against hiring, Eazyware versus building an in-house AI team sets out the trade in detail.

When this level of due diligence is the wrong choice

If the engagement is under roughly $5,000 or ₹3,00,000 and the data is non-sensitive, running a full vendor interrogation costs more in your own time than the risk it removes. Buy the small thing, judge the work, and apply this list to the build that follows.

It is also the wrong instinct when you have not decided what you want proved. A vendor cannot give you specific answers to questions about an unspecified problem, and the conversation degrades into capability claims because nothing else is available. Decide the use case first, then interrogate the vendor. If the use case itself is unclear, buy discovery rather than a proof of concept.

What a straight answer looked like in practice

A B2B field-service SaaS team came to us wanting an in-app copilot and asked exactly these questions, including the uncomfortable one about failure. The honest answer was that the retrieval part was low risk and did not need proving, while the action-taking part did, so the measured slice was scoped around actions and permissions rather than around answers. That is a smaller proof of concept than they expected to buy. The finished product is described in the in-app copilot case study.

Related reading

Fixed price versus time and materials for AI projects explains why the commercial model changes what you should ask, and why AI pilots never reach production covers the failure the whole exercise exists to avoid. The US National Institute of Standards and Technology publishes an AI Risk Management Framework organised around four functions, govern, map, measure and manage, which is a useful neutral structure for a vendor's answers on governance.

Buy from the vendor whose answers contain numbers and a description of how this could fail, and treat confidence without evidence as the risk it is.