Finding a needle in a haystack is a tough challenge right? Not if you have the right tools – simply run all the hay in the stack by a magnet and you’ll have your needle soon enough.
What about finding the particular needle you need in a stack of needles instead? A much more difficult task, one best handled by keeping track of every needle in the stack.
Now, let’s assume instead of a stack of needles, you have a never-ending barrage of needles. Now finding individual needles requires a jump in the level of sophistication.
This ‘stream of needles’ problem is one that Rocana faces every day, metaphorically speaking.
Among IT operations management (ITOM) vendors, Rocana’s primary differentiation is its focus on the massive scale end of the problem: dealing with management, anomaly detection, and root cause analysis in web-scale IT environments by applying big data technologies and practices.
Over the last few weeks, Rocana has rolled out several improvements to its platform that augment these big data-centric capabilities. Let’s take a closer look and see what Rocana can do for your never-ending barrage of needles.
Better Indexing and Search: Keeping Track of the Needles
The needles in our metaphor correspond to operational event data – data from logs, APIs, or any other operational data source. As the number of nodes under management increases and the quantity of data from each node explodes, managing all these event data quickly becomes a big data challenge.
The open source tools for managing such information are Lucene and Solr, both of which Rocana utilizes. However, Rocana determined that Solr was not going to scale at the rate their customers required.
As a result, Rocana rebuilt its search from the ground up. The new search capability requires only about half the system resources of previous versions for the same data volumes. In addition, the new search capabilities allow for the growth of indexed data over time, with minimal impact on the underlying system.
Rocana also provides time-based search functionality – especially useful in the context of streaming event data. For example, operators can run queries like stats for the ‘last five Black Fridays’ or other custom date ranges.
Centralized Agent Management: Stacks of Needles
Agent-based ITOM solutions are relatively common: place a small piece of code on each system, runtime, or piece of infrastructure you want to monitor, and this agent will serve as your eyes and ears, returning a stream of telemetry to the ITOM application for analysis.
As we scale up this scenario, however, we eventually run into the problem of too many agents to keep track of. While typical early-phase enterprise deployments may include perhaps a few dozen agents, full-scale, mature ITOM installations could easily rely upon thousands or tens of thousands of such agents.
What if one of them is misbehaving? Now you have the classic needle in a stack of needles problem – a problem that only gets worse as the number of agents explodes.
To address this challenge, Rocana is now offering centralized agent management. Admins now have a central location for sharing agent configuration profiles and templates with full version control for change management.
It’s straightforward to deploy agent profiles to thousands of agents through a single operation in Rocana Ops. Rocana Ops monitors each agent continuously to ensure it is collecting data properly and will alert admins promptly when there’s a problem.
In addition, centralized agent management helps IT manage and enforce policies across the infrastructure, as well as provide confirmation that such compliance is active – an important governance requirement.
Avoiding Death by Needle Pricks
Faced with the proverbial needle-wielding haystack, one might dig through by hand, as the prick from a single needle is manageable. But a barrage of needles? Not on your life.
Such is the situation many microservices developers find themselves in today. Each individual microservice is ideally quite parsimonious – doing only one thing and that thing quite well. As teams assemble full-blown digital apps out of these microservices, however, managing the complexity soon turns into a ‘not my problem, must be your problem’ debate.
Today it seems that every management tool vendor on the market is touting their microservices management capabilities – and after all, managing a small handful of needles is no big deal. Where Rocana shines, however, is when the number of microservices explodes.
Not only does the sheer number of such bits of running code present a straightforward management problem, but the event traffic flows that the management tools must monitor soon become a network-clogging pandemonium of data, if it weren’t for Rocana’s big data-savvy approach.
In fact, Rocana is doubling down on microservices management with their free Rocana One offering: monitoring of up to one terabyte of data per month, no charge, with all the functionality of its flagship Rocana Ops.
Given that most microservices deployments today are pilots or proofs of concept, a free tool is just the ticket for many organizations. Furthermore, with the terabyte limit, Rocana announces to the world that a terabyte of monitoring data per month really isn’t all that much in the grand scheme of things.
The Intellyx Take
Given their deep big data expertise, the Rocana team is intentionally targeting the high volume end of the ITOM marketplace. In situations where the sheer quantity of data promises to swamp normal tools – including both the business data themselves as well as monitoring data – Rocana’s scalable, high-volume approach becomes a must-have.
However, while today, only a modest segment of the enterprise space has such high-volume big data requirements, that reality is fast changing. Rocana is essentially staking out its territory ahead of the time when most large organizations will have the sorts of problems Rocana is especially well-suited to solve.
For many of today’s enterprises, ITOM challenges of anomaly detection and root cause analysis are as simple as finding needles in a haystack – but that won’t be true for long. We’re already well down the road to stacks of needles and never-ending barrages of needles. And when those needles come your way, you don’t want to be caught without the right tools.
Rocana is an Intellyx client. Intellyx retains full editorial control over the content of this article. Image credit: Fod Tzellos.
I dont see anything that substantiates this claim “The open source tools for managing such information are Lucene and Solr, both of which Rocana utilizes. However, Rocana determined that Solr was not going to scale at the rate their customers required.” So Rocana built its own search engine. Why would that be any better? Anything to support this claim? How does Rocana decide what needs to be collected in the first place? All data? How does one define what all is? Inevitably part of the reasons searching for needles is tough is because often you dont know what needles need to be collected in the first place. Nobody has yet been able to collect “ALL” since the set of ALL data is uncountable in mathematical terms. So I am not sure I buy this story.