Those who can’t develop, test. Those who can’t test, manage. Those who can’t manage, become analysts.
I’ve been doing a lot of thinking lately on a current meta-trend of ‘hidden software testing.’ Perhaps writing my recent series of pieces on Shift-Right development practices — in which much of software testing gets pushed to post-production — led me here.
I’ve spent most of my life working with technology firms where testing was an unassailable virtue, and therefore, the concept of testing was constantly encouraged. In many cases, doing the testing itself became too much hassle, or pressure.
So why has testing gotten such short shrift lately?
Test teams have changed
Remember when there used to be a QA team, separate from the development team in any decent-sized IT shop with a software delivery function?
This QA team might have staffed as many as one tester for every 3-to-10 Dev and Ops people, and sometimes the test org would be more closely aligned with Ops than Dev. A dedicated team that understands business requirements certifying each build before deployment against agreed-upon standards provides assurance for risk-conscious enterprises against costly production failures.
Unfortunately, the balance of power didn’t play itself out very well for the QA team. The advent of Agile development practices arose as a response to the survival imperative of delivering new innovative features to customers faster than the competition. Test-driven development was a key part of this shift, causing earlier unit and regression testing burdens to be shared between teams.
However, when being first to market trumped everything else, the QA team faced ever-shortening test windows, and retained little authority to ‘stop the presses’ for quality issues that act as an inhibitor of change. Many companies abandoned QA teams in favor of mingling testing and development. Who wants to be the buzz-kill that stopped the killer software release customers expected?
When DevOps started to appear on the scene more than a decade ago, there was a brief glimmer of hope it’d get rebranded “DevTestOps.” This is where security put its foot down as another too-often-overlooked obstacle of the SDLC, so that if a middle name for DevOps is used, now it’s DevSecOps.
Test roles have changed
The sunset of the QA team also means we don’t have a ‘Head of QA’ any more (I am always here to volunteer as Head of QA at a barbecue or brewery, though!).
We also saw titles containing “QA Specialist” or “Tester” becoming scarce about a decade ago. In an agile or DevOps world, the testing function is increasingly integrated into pair programming or small Agile team exercises.
Testers are becoming co-located with developers (or working in sync on each release if remote). Some companies encouraged more coding on the part of testers, and more testing on the part of engineers, eventually leading to the rise of the ‘SDET’ (or software development engineer in test) cross-functional role.
In general, I would say having developers test and care about quality is a good thing, however testing is still a unique and separate discipline from development. If developers must test their own code, and testers must become developers, who will independently validate that the software meets business requirements?
Perhaps a magic testing box?
Test tools have changed
The grand age of automated testing tools has subsided. Richer freemium tools, and then, open source software took over for the huge enterprise licenses sold by the likes of Mercury and Rational and Segue, back when Y2K testing was still a real problem to solve.
Innovative testing vendors started testing behind the UI, and beyond simply unit testing code, to test integration layers, service calls and APIs, and the performance of databases and network connections. Testing tools became sophisticated enough to eliminate dependency constraints with service virtualization, and become automatically invoked within the flow of a complete software delivery pipeline.
At this point, we started to approach the concept of software observability as a hot new space, and testing lost its shine. Gartner even discontinued Continuous Testing as a product category altogether. How can a testing industry survive without a magic quadrant to rank the vendors?
Punting the responsibility for testing elsewhere
The only companies who seemed to be doing what I’d call 50% test coverage, were often rolling their own home-grown automation skunkworks rather than waiting for tool vendors to catch up with their quality issues. For instance, writing Junit or shell scripts to do things like sending a request, sniffing an ESB or message queue, then even going back and reversing the test transactions they created.
The reality remains that testing must still find a home within your process, or total failure will certainly catch up with your organization. That leaves several options:
- Offload it to a new QA team. Why not start over with a tiger team of expert testers?
- Offload it back to development. You’d need to seriously consider the incentives for success before attempting this.
- Offload it to a tool. Test automation tools keep getting better, though how unattended can they really be?
- Offload it to a services firm. This is still very commonly used, and you may benefit from alternate vendors who can develop and test on different projects.
- Offload it to a testing specialist. New types of test-specific services firms have arisen with their own expert methodologies and tool stacks. They may work as closely with your teams as you like.
- Offload it to customer support. Just tell them a new release is coming out, so they can report back if they are confronted with any issues. Everything’s gonna be alright.
- Offload testing to your customers. You want to Shift-Right as far as you can? Establish a beta tester program or try A/B, feature flagging and canary testing with willing (or unwitting) customers. It may not work for highly sensitive workflows, but it may give you the best real-world test feedback you’ve ever had.
The Intellyx Take
Like water overflowing a dam and washing out the banks of a river, errors always have a way of flowing downstream. Quality issues that aren’t addressed with testing will eventually build up and break the best software delivery automation efforts.
Denial is not a river in Egypt, but many companies use denial as a last-ditch effort to escape testing responsibilities. Basically, don’t admit there is even a problem, and hope someone else will pick up the slack. In situations like this, you might find the CFO of an org starting up their own hidden test factories of manual auditors, or a hybrid sales/legal team conducting compliance tests, if only to stave off a massive loss or regulatory penalty.
This trend of hidden testing — and thinking we can hide testing — will pass. In the meantime, it’s kind of fascinating to watch how organizations can adapt and survive the change.
Look out for me to write more on this topic soon, and if you have any perspectives on hidden testing you think I would benefit from, or if you just want to be contrarian and tell me I’m way off base, my door’s open. Just contact me at pr@intellyx.com.
© 2021, Intellyx, LLC. Intellyx retains editorial control over the content of this article. Get our Cloud-Native Computing poster. If you are a vendor seeking coverage from Intellyx, please contact us at PR@intellyx.com. Image source: Klaus Stiefel, flickr creative commons.
