Define a complete task
“Help the operations team” is too broad to evaluate. “Collect the evidence for this SIM verification case and propose the next step” has inputs, outputs, and a boundary. A useful agent starts with a task that a person can recognize as finished. The completion criteria should account for the quality of the evidence, not simply whether the agent produced a response.
Make permissions part of the design
Reading a log, drafting a change, and activating a service are different actions. Give tools narrow interfaces and explicit permissions. Decide which steps may run automatically, which require review, and which must remain outside the agent’s scope. A conversational request should not silently expand the authority of an underlying service account.
Expect retries and partial failure
Operational workflows contain slow services, missing records, duplicate events, and changing state. Agents need the same protections as conventional software: timeouts, bounded retries, idempotency, validation, and a way to recover from a partially completed action. Natural-language reasoning does not replace these mechanics. It depends on them.
Evaluate the path as well as the answer
Record the tool calls, retrieved evidence, errors, and state changes needed to explain an outcome. Test a representative set of normal and difficult cases. Include a wrong tool response, incomplete documentation, conflicting instructions inside retrieved material, and an unavailable dependency. The system should fail in a way an operator can understand and recover from.
Keep people effective
Human review is useful when the reviewer sees the relevant evidence and understands the consequence of approving an action. A generic confirmation button is not enough. Design review surfaces around the actual decision, and make it possible to pause, correct, and resume work. The goal is a more capable team with a clear line of control.