Five questions to ask before you build an AI agent
AI agents can answer customers, process documents and complete multi-step tasks across your systems. Answer these five questions first, and the agent you build is far more likely to succeed.
1. What exact job will it do?
“Help customers” is not a job description. “Answer delivery and returns questions using our policy documents” is. A narrow, clearly defined job is easier to build, test and trust.
2. What can it access and change?
List every system the agent can read from and every action it can take. Give it the minimum access it needs. An agent that can only draft a reply is far lower risk than one that can issue refunds.
3. When does it hand over to a person?
Design the hand-off before launch: complex cases, unhappy customers, low-confidence answers and anything involving money or personal data. Klarna’s AI assistant handled most customer service chats within a month of launch, and the company later rebalanced towards a mix of human and AI support for harder cases. See our case studies.
4. How will you test it?
Build a set of real example questions with known good answers and run the agent against them before every change. Morgan Stanley tested its advisor assistant with a full evaluation framework and daily regression checks before and after rollout.
5. How will you know it’s working?
Decide on your measures in advance: resolution time, accuracy on reviewed samples, hand-off rate and customer satisfaction. Log every conversation so problems can be traced and fixed.
A good AI agent has a narrow job, minimal access, a clear route to a human, a test suite, and agreed success measures.