Simply put, the innovator’s dilemma is whether to continue improving a current product, or develop a new product based on an innovation that disrupts the current product.
Clayton Christensen describes the phenomenon in detail in his classic book, “The Innovator’s Dilemma.”
“What often causes this lagging behind [innovation] are two principles of good management…: that you should always listen to and respond to the needs of your best customers, and that you should focus on innovations that promise the highest return,” he says in the introduction.
“That’s why we call it the innovator’s dilemma: doing the right thing is doing the wrong thing.”
In other words, operating your business successfully based on monetizing a prior innovation may be the reason you miss out on monetizing a new innovation that ends up disrupting your business long term.
Are we witnessing the “innovator’s dilemma” in the AI agent market?
What’s an AI Agent?
First of all, let’s define an AI agent. Everyone is talking about them, and it seems clear that they are the next big step in the evolution of the generative AI market.
But do we all agree on what an AI agent is? Let’s see what some gen AI industry leaders say.
OpenAI: Agents are systems that independently accomplish tasks on your behalf. … Applications that integrate LLMs but don’t use them to control workflow execution—think simple chatbots, single-turn LLMs, or sentiment classifiers—are not agents.
Microsoft: An agent can tackle certain tasks with you or for you, from acting as a virtual project manager to handling more complex assignments such as reconciling financial statements to close the books. … An agent takes the power of generative AI a step further, because instead of just assisting you, agents can work alongside you or even on your behalf.
Google: AI agents are software systems that use AI to pursue goals and complete tasks on behalf of users. They show reasoning, planning, and memory and have a level of autonomy to make decisions, learn, and adapt.
Anthropic: “Agent” can be defined in several ways. Some customers define agents as fully autonomous systems that operate independently over extended periods, using various tools to accomplish complex tasks. Others use the term to describe more prescriptive implementations that follow predefined workflows. At Anthropic, we … draw an important architectural distinction between workflows and agents:
- Workflows are systems where LLMs and tools are orchestrated through predefined code paths.
- Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.
So basically some AI vendors describe agents as specific to a task, while others describe them as workflows, and Anthropic says they can be both.
Anthropic also says “Over the past year, we’ve worked with dozens of teams building large language model (LLM) agents across industries. Consistently, the most successful implementations weren’t using complex frameworks or specialized libraries. Instead, they were building with simple, composable patterns.”
The phenomenon known as the “innovator’s dilemma” typically starts when a lower cost, simpler solution arises to a given set of requirements because of a new innovation. Gen AI represents a new innovation, and a stunning one at that, but does it really undercut existing solutions?
This is the crux of the biscuit, as they say. What requirements do successful AI agent implementations solve?
A Possible Example
An interesting case in point are the AutoGen and AG2 open source projects (as mentioned in my recent article about the AI Agent conference in New York).
AutoGen and AG2 are both open source AI agent frameworks, designed to help develop and deploy multi-agent orchestrations and workflows. They both build applications composed entirely of AI agents.
They are based on the same original open source project called AutoGen developed by Penn State academics.
Essentially, AG2 is the continuation of the original AutoGen project under a new name.
Microsoft is releasing a new version of AutoGen that maps AutoGen to its Semantic Kernel for “enterprise-grade” solutions, which will require a code migration.
Meanwhile, the original team behind AG2 has taken a fork of AutoGen to continue in the original direction, using a generic Python library that runs on any infrastructure.
(Here are links to the AG2 GitHub and the AutoGen GitHub for further details.)
In other words, Microsoft is positioning AutoGen as an extension of existing software, while AG2 is continuing on the independent path with a “simple, composable pattern” as Anthropic puts it.
Is this a mistake on Microsoft’s part? Will AG2 disrupt AutoGen? This remains to be seen, of course. It’s too early to be sure. But we see a similar approach by other enterprise software vendors.
We recently spoke with Vantiq, Akka, and SnapLogic and attended Boomi World. All of them told us they are adding AI agents to their existing platforms, and emphasize the benefits of their platforms for enterprise class AI agents.
At Boomi World, I spoke with several customers who confirmed their interest in developing and deploying AI agents with Boomi.
So this is all completely understandable and business as usual. Customers are asking for AI agents, and the agents leverage the capabilities of these existing platforms. Application and data integrations, event based communication protocols, and high reliability and performance levels are typical requirements of enterprise applications.
And it’s clear that AI agent capabilities augment and extend the capabilities of these platforms to meet additional customer requirements.
What’s not clear is whether customers of existing products will define or lead the AI agent market, or whether the core innovation of generative AI offered in simple, open source frameworks such as AG2, Arklex, and CrewAI will eventually disrupt these products.
Innovator’s Dilemma Revisited
One of the key factors in Christensen’s innovator’s dilemma is cost and the reasons people buy things.
For example, people buy laptops for very different reasons than they buy mainframes.
Christensen describes how large incumbent companies lose market share by listening to their customers and providing what appears to be the highest-value products, but new companies that serve low-value customers with poorly developed technology can improve that technology incrementally until it is good enough to quickly take market share from established business.
The industry is full of examples. When digital cameras first came out, the quality was really poor and nowhere near the quality of a film camera. Yet they were lightweight, relatively inexpensive, and consumers could take any number of photos without worrying about spoiling the film or the time and cost of developing the photographs. They could see the resulting image immediately and take a new one if they made a mistake.
Similarly the sound quality of MP3 based music is nowhere near the quality of a sophisticated stereo system, and yet is so much more convenient people prefer the convenience rather than deal with the complexities of a sophisticated, better sounding system.
The computer industry has a well known history of endless disruption. First was the mainframe, which was disrupted by the minicomputer, which was disrupted by the PC, which was distributed by the smartphone.
When the world wide web came out it was ridiculed as overly simplistic compared to existing enterprise software platforms. No one believed it would be used for enterprise applications, only for publishing. And yet, simplicity drove adoption, which drove ecommerce, and the web eventually took over the enterprise (RESTful APIs for example).
What is the Innovation?
At this point it might be helpful to take a step back and try to characterize the innovation and see if we can get a sense of how disruptive it may be. This should help us understand whether we’re seeing a true “dilemma” or whether it’s just a case of incremental functionality.
Generative AI enables people to communicate with computers using natural language rather than computer language. This appears to be a huge disruption. Of course how big it is depends on how well it works, and that is perhaps critical to determining which way the market will go.
Boomi’s CEO Steve Lucas has written a book called Digital Impact, which is a national bestseller. In it he calls generative AI the most significant technological disruption since the introduction of the world wide web.
“I will go so far as to say that everything we know about digital technology and the systems powering the world are going to change thanks to AI,” he writes in the introduction.
And when he first encountered Chat GPT, “I realized that all I had ever known about software would change drastically in the coming years.”
At Boomi World, Lucas advised customers to immediately invest in gen AI, or get left behind competitively.
From a technical point of view, the innovation boils down to creating a large language model (LLM) by transforming a large amount of human language into vectors and using statistical analysis across the vectors to produce a human language response to a human language prompt.
In other words, it’s not really intelligent in the sense of thinking or reasoning to produce an answer (and thinking or reasoning could sometimes produce new answers, unlike gen AI). It’s based on probabilistic statistical matching algorithms. But it’s on the other hand amazing at what it gets right, even when it gets many things wrong.
The essential question in my mind – hallucinations and incorrect answers aside – is what can we do today that we couldn’t do yesterday? In other words, are there business (or personal) requirements that deterministic technology could not accomplish that the non-deterministic gen AI can accomplish?
Common AI Agent Use Cases
Maybe we can figure this out by looking at what people are using AI agents for. Searching the web for common use cases results in the following list (in no particular order):
- Customer support and service
- Sales and marketing
- Business and finance
- Fraud detection and cybersecurity
- Education and training
- IT development and operations
- Staff augmentation
- Multi-agent conversions for complex workflows
I’m sure there are more. Most of these use cases sound like things people and computers already do today, only faster and more productively with the use of gen AI based automation and agents.
If that’s the case, it makes sense to add AI agents to existing products.
However I can share one more example from the TNS article on the AI Agent conference. The founder of Arklex.ai, an AI Agent framework, Jo Yu, told me their agents continually learn by fine tuning the models. Walmart uses their framework, she said, to allow customers to chat about product features and capabilities, such as size, battery life, colors, and so on. Whenever the agent responds to a chat prompt, the agent uses the information in the prompt to fine tune the model, learning about the user’s preferences and interests.
Building up a chat context like this to improve future human language interactions is not a feature of existing products, since they are in the deterministic software category.
Is this capability enough to disrupt the established players?
The Intellyx Take
We are at a very interesting moment in time. Gen AI is becoming widely adopted and used, and everyone is developing and supporting AI agents as the next step in that evolution.
Agents can help address many of the challenges inherent to gen AI, namely the non-deterministic nature of prompt replies that result in hallucinations and inaccurate results.
Agents break the problem down into specific domains – very much following the principles of domain driven design to reduce complexity. It’s easier to evaluate the agent’s results and iterate on it to produce better results when the problem domain is focused on a particular task or skill than it is to improve a monolithic gen AI process that “answers everything.”
It seems to me that the capabilities people want from AI agents are easier to obtain from the independent agentic framework startups coming out of universities than it is to obtain them from existing enterprise products.
It’s true that using an AI agent for business requires enterprise levels of predictability, reliability, and performance. But I’m thinking it will be easier to improve the agentic frameworks than adapt existing enterprise products to deliver a similar range of functions.
I think we are going to see incumbents disrupted – gen AI just seems like a significant enough innovation. The incumbents are still focusing on their existing products, and finding good ways to incorporate AI agents. The startups are building out features and functions solely for AI agents.
Copyright © Intellyx BV. Intellyx is an industry analysis and advisory firm focused on enterprise digital transformation. Covering every angle of enterprise IT from mainframes to artificial intelligence, our broad focus across technologies allows business executives and IT professionals to connect the dots among disruptive trends. None of the organizations mentioned in this article is an Intellyx customer. No AI was used to write this article. Image credit: Derek Xing from Pexels.com
