Brainblog for tricentis by eric newcomer
By now everyone is aware of the limitations inherent in generative AI and the AI agents that use it to complete their tasks, and the challenges involved in getting them enterprise quality. If you are planning to incorporate AI agents into your enterprise IT architecture, how are you planning to validate their quality and accuracy?
Perhaps you are thinking about using testing agents to ensure your enterprise agents produce correct results? A good way to think about it is as an automation system in which testing validates agentic operations but humans maintain oversight, as they would with any set of external consultants. In his recent blog post on leveling up quality engineering for agentic development, my colleague Jason English identifies three “bosses” to satisfy: Quality, Business Outcomes, and Organizational Risk. In this follow up, I’m going to discuss how to test AI agents to validate enterprise quality and satisfy the “bosses.”
By starting with the test management, test automation, and test data tools that your team already uses, you can establish the right foundation for validating your AI agents. Use your testing tools to establish a baseline of QA metrics, including test coverage, defects, and bugs fixed. Generative AI coding systems and agents rely on non-deterministic LLMs and the code will therefore contain errors, hallucinations, and incorrect results, so it’s an important first step to collect this data.
