Buying an AI customer-service agent requires more than comparing channels, integrations and demonstration results. Organisations also need to know how they can verify the system’s work, control its actions and challenge its bills once it is operating at scale.
A procurement framework published by Kaizo sets out 38 questions across eight areas, from billing definitions and conversation records to testing, human handoff, incident response, data protection and commercial dispute...
Continue Reading This Article
Enjoy this article as well as all of our content, including reports, news, tips and more.
By registering or signing into your SRM Today account, you agree to SRM Today's Terms of Use and consent to the processing of your personal information as described in our Privacy Policy.
The distinction matters because “resolved” can mean different things to different providers. Intercom’s documentation says Fin may count a conversation as resolved when a customer confirms that its answer helped, or leaves without asking for further assistance. The rules also describe other outcomes, including escalations and technical failures, that are classified differently. Zendesk, meanwhile, distinguishes contained resolutions from verified ones: the latter are assessed by a large language model reviewing the conversation text. Its documented time windows for deciding when a conversation has ended vary by channel.
These definitions can determine both reported performance and the invoice. Yet neither a customer’s silence nor an assessment of conversation text necessarily proves that an answer complied with company policy or that a promised action was completed. Buyers should therefore ask for the trigger, verification method and time window behind each charge, as well as what happens if the customer returns with the same problem.
Kaizo illustrates the financial impact with a hypothetical example, not data from a vendor. If an agent reports 24,000 billable resolutions at $1 each, a random audit of 400 conversations that finds an 18 per cent failure rate would imply about 19,680 correct resolutions. The effective cost would rise from $1 to roughly $1.22 per correct resolution, before any contractual credits. The example underlines why an agreement should specify how buyers can dispute charges, what evidence counts and whether failed resolutions qualify for a refund or price adjustment.
That evidence should be available conversation by conversation. A useful record includes the full exchange, the material used to generate the answer and a trace of actions taken through connected systems, including their inputs and results. Kaizo recommends continuous export in a documented format so a buyer can conduct its own quality assurance rather than relying solely on a vendor’s dashboard. Export rights are also important for investigating incidents, measuring repeat contacts and comparing the supplier’s figures with the organisation’s own assessment.
Testing should continue after launch. Buyers can ask vendors to run the system against historical conversations before signing, allow customers to maintain their own test cases and rerun them whenever prompts, policies, knowledge or models change. A release gate can prevent a change reaching customers if it fails agreed checks. The same scrutiny should apply to permissions: organisations need to know which actions the agent can take independently, where approval is required, how limits such as refund ceilings are enforced and how quickly the system can be paused.
Human escalation is another operational safeguard. Buyers should establish when the agent must hand a conversation to a person, whether customers can always request one, and what information the employee receives. A useful handoff includes the conversation history and actions already attempted, rather than a brief summary. Procurement teams should also ask how the supplier identifies late or failed handoffs.
Model changes and incidents need written processes, not assurances that the service is continually improving. The framework recommends asking how much advance notice a vendor will provide before changing a model, whether customers can test the change first and whether they can remain on an earlier version if results deteriorate. Incident questions should cover severity levels, notification deadlines, named contacts and written follow-up, alongside monitoring for quality drift.
Data and regulatory responsibilities require similar precision. Article 28 of the General Data Protection Regulation sets requirements for processor contracts, including written authorisation for sub-processors. Where a controller has given general authorisation, the processor must notify it of intended additions or replacements and allow it to object. The article also provides for information demonstrating compliance and for audits. These obligations make it important to identify model providers and other sub-processors, understand how changes are communicated and establish what happens to data when a contract ends.
The European Union’s Artificial Intelligence Act adds a further reason to clarify responsibilities. The European Commission says the Act’s transparency obligations under Article 50 are due to apply from 2 August 2026, while provisions on AI literacy have applied since 2 February 2025. The regulation’s logging and monitoring duties for deployers apply specifically to high-risk systems; they should not be treated as a universal requirement for every customer-service agent. Whether a particular use is high-risk depends on its purpose and deployment, and should be assessed by the organisation with appropriate legal advice.
Kaizo proposes evaluating vendors in two stages: first remove any that fail the essential safeguards, then score the remaining written answers against a consistent scale and evidence standard. A “yes” unsupported by a document, demonstration or contractual commitment should not be treated as proof. Support operations, quality assurance, IT, legal, privacy and finance teams each have a role, while the scoring weights should be agreed before responses arrive.
The final comparison should be practical as well as written. Each shortlisted provider can be asked to work through the same anonymised customer conversations in the same sequence: answer selected cases, explain why a particular exchange was billed, show its transcript and action log, export records, change a policy and rerun tests, handle difficult scenarios, demonstrate a stop control and describe its process for a model update. This tests the product’s controls as well as the quality of the presentation.
The degree of formality can be scaled to the deployment. A narrow agent answering a small volume of routine questions without taking action may need a shorter review focused on billing, evidence and escalation. Systems that issue refunds, change orders or handle sensitive topics warrant broader checks. In either case, the procurement decision should establish how performance will be measured and challenged after the demonstration ends.
Source: Noah Wire Services



