An Intellyx BrainBlog by Jason English | Part 1 of the Auguria Data Experience Series
Despite complaints of a softening job market in certain tech sectors, we are finding that security, platform and observability roles are still massively understaffed. There are more than half a million unfilled US jobs in cybersecurity alone in 2025, and 3.5 million globally, according to current reports.
It seems we’ll never be able to train and hire enough great talent to effectively monitor our application estates for security and reliability, given the rate of growth of enterprise data that our systems generate.
A medium-sized enterprise might take in hundreds of millions of event logs in a single day, emanating from every gateway, cluster and node in a hybrid cloud infrastructure. Add in AI-driven application functionality, AI-generated code assets, and ‘autonomous’ agent traffic, and the whole estate can become ten times as chatty.
This results in a brutal data experience for analysts and operators, which we’ll never be able to hire our way out of as an industry. Perhaps it’s time to get ahead of the storm and change the way we experience data.
Caught in the storm of event data
Security pros and SREs are getting deluged with huge volumes of event data. These event data storms result in too many alerts and issues to process and prioritize. Teams wind up pulling together custom reports and participating in incessant incident calls and sev1 war rooms, which inevitably leads to employee burnout and turnover.
New high-volume data pipeline approaches are on the rise, accompanied by highly scalable back ends to receive them. But the data experience problem won’t be solved by pumping more logs and traces into our SIEM platforms and data lakes, so we can then sort through them to find out what mattered after the fact.
Even the best researchers want to give up and quit the data wrangling business entirely. Why try harder, if you are still unable to get to root causes before issues appear in production?
We need to abstract the data to make it usable in the here and now, so it’s easier for time-constrained experts to understand how that data will impact real-world outcomes…
Read Part 1 of the series here: https://auguria.io/blog/to-improve-the-data-experience-change-the-way-you-experience-data
