Evaluating an AI customer assistant

Choose a narrow customer task, such as explaining published product instructions or helping a signed-in customer locate an order. List approved knowledge sources, unsupported topics and actions that require confirmation. Distinguish reading information, drafting a proposed change and executing that change. These need different permissions. Name the owner who resolves conflicting policies and the person who receives escalations. If the business cannot define the correct outcome, a model comparison will not settle the uncertainty.

Use permitted, de-identified questions that represent the expected journey. Include straightforward requests, incomplete questions, misspellings, follow-up questions and cases where the correct response is to ask for clarification. Add outdated and conflicting documents, unavailable records and requests outside the stated scope. Each case needs the acceptable behavior, the source of truth and the severity of a wrong answer. Avoid grading only whether a response resembles a preferred sentence; several wordings may be correct.

Assess whether the answer is correct, addresses the actual question and communicates uncertainty appropriately. Check cited material against the claim it supposedly supports. A valid-looking source link does not make an unsupported conclusion correct. For policy or account questions, verify that the assistant uses the relevant version and context. Record failures by category so a good average does not conceal repeated mistakes in one important journey. Keep the original question and a reviewable result for each test.

Try to retrieve another customer’s record, continue after access is revoked and request an action outside the signed-in user’s authority. The application and tools must enforce access independently of what the assistant says. OWASP describes prompt injection through direct requests and external content, so include hostile instructions inside retrieved documents as well as in chat. Check that such text cannot authorize an export, change a destination or override the tool’s allowed scope. Log enough for investigation without collecting unnecessary private data.

Separate critical failures from quality improvements. An unauthorized action or cross-customer disclosure should block the relevant capability even if most ordinary answers pass. Set agreed thresholds for the lower-risk tasks, response time and escalation completion. Compare the assistant with the current process: can a person actually resolve the handoff with the context provided? Roll out a restricted scope first when evidence is limited, and keep a clear route to a human for unresolved requests.

Version the test set alongside the prompt, model configuration, knowledge and tools. Re-run relevant cases when any of these change. Review permitted real failures to add missing scenarios, while keeping a stable test group for comparison. Assign someone to inspect recurring escalation reasons and stale documents. Provide a control to disable a risky action or the entire assistant without taking down the main customer journey. Evaluation is part of operating the service after launch.