An Intellyx Brain Candy Brief
Distributional designed an unconventional testing platform specifically tuned for measuring confidence levels and predicting the potential non-deterministic risks of embedding AI-based features into applications.
While we’ll always need traditional software test coverage for structural and functional bugs, and observability tools to find failures and performance issues in production, conventional tooling fails to recognize the unpredictable results and emergent behaviors introduced by LLMs and AI inference models, which are often stacked to provide specialized insights.
DevOps teams may conduct AI testing in a simplistic manner, looking to validate acceptable responses from LLM prompts, but that’s not good enough to confidently release software with AI dependencies.
Distributional can also operate in an unsupervised mode to look for emergent behaviors and model drift across a complex graph of training data pipelines, retrieval-augmented generation, intra-model interactions, and inconsistently introduced model changes to help mitigate and control ongoing AI development risks—while providing some auditability and confidence for nervous executives.
Copyright ©2024 Intellyx B.V. Intellyx is an industry analysis and advisory firm focused on enterprise digital transformation. Covering every angle of enterprise IT from mainframes to artificial intelligence, our broad focus across technologies allows business executives and IT professionals to connect the dots among disruptive trends. At the time of writing, Distributional is not an Intellyx customer. No AI was used to write this article. To be considered for a Brain Candy article or event visit, email us at pr@intellyx.com.
