Just returning from an excellent trip to Salt Lake City for KubeCon/CloudNativeCon North America 2024—you can read my coverage of the event in SiliconANGLE here.
At the conference, it really struck me just how far software quality has come as I looked at so many high performance enterprise applications, scaling with the help of Kubernetes and containerized workloads and running atop commodity public or private infrastructure that could live anywhere.
When it comes to open source software, we truly stand on the shoulders of giants. Any startup can come out of the gate with applications that leverage cloud native reference architectures used in the real world, which have just about everything a serious product needs: composition, service mesh, networking, workflows, data management, security, data stores, observability, autoscaling … on and on. It seems like there is a project for everything.
These sophisticated CNCF projects may have been designed and set in motion by very smart developers, but they didn’t become robust enough to use for business by themselves. They were continuously hardened by a massive community of end users who would truly push the limits of each new release, providing feedback and contributions as they use the software out in the world, in ways the original developers never intended.
In essence, these users form an army of really clever tech-savvy testers—more than any commercial software product could afford—to make open source software development scale new heights. But this story isn’t about them.
Sleepless in Salt Lake City
After a full day of analyst track sessions, briefings and booth visits, I was ready for bed. Took a melatonin, watched some hockey highlights, and was out cold.
Then I had this high-anxiety dream. One of those dreams like, when you are running late to class for your final exam, and you are struggling to run and get there, but then when you do, it’s the wrong room and you don’t have any pants.
However, this nightmare wasn’t about failing a test, it was failing to test.
It’s been a while, but I recently chatted with my friend and former drummer Joe who reminded me about an adventure we embarked on back in 1994, and it resurfaced long-forgotten memories.
At that time, I worked at Museworthy with the founder, Jim, in a shotgun-style office full of antiques in Addison, Texas, operating one of the first desktop video agencies.
Encouraged by the huge success of the CD-ROM game Myst, we looked around and realized we had already done plenty of print advertising, commercials and corporate videos with our own copywriting, design, and music. So, if those art geeks could string together a bunch of nice looking illustrations in Hypercard to make a game, why couldn’t we do it even better with our video gear and this new Macromedia Director app we just got in our design bundle, so players could just click on the videos?
Jim secured us a small seed funding, and we started turning down our advertising gigs. As I was finishing up the script for Paparazzi! Tales of Tinseltown, we brought in Joe as an additional designer/3D artist for the game and John to direct and edit video, but like any tiny company, all of us did a little bit of everything, especially casting our many celebrity lookalikes, finding locations and props, and testing.
We didn’t know it was agile
In my rare downtime, I had a 386sx PC at home, which was good enough to play my favorite games: those great Sid Meier simulators like Civilization and Railroad Tycoon. I was determined to put some kind of engine behind the game like that, rather than just letting player paparazzis click on videos and make binary plot choices.
I came up with an Integrity status variable, which would determine what messages or tips you get from others, and which locations you could get into as you chased down 24 different celebrities over the course of a game week.
To reconcile this, we made a 3×5 card for every scene from the script, and we punched holes in either side of the card, so we could tie different colors of yarn (red for required, yellow for optional) for upstream and downstream event dependencies. Since a paparazzi could only visit 2 locations each day or night, it had to be possible to get all the necessary clues before catching each celebrity in the act, and to see any after effects of the exposed celebrities in the news afterward.
So with all the cards and strings laid out left to right on the floor like a timeline, and vertically based on the player’s current Integrity score, we were able to game out the different streams, deciding when clues like TV shows and video messages would play, before the celebrity encounter game events, making sure no day or night had more than 2 possible encounters, and that no celebrity’s story in the game lacked an ending based on the player’s decisions.
It was a lot like an agile scrum, actually, though we had no such concept of vetting and requirements-based testing.
Getting down to speeds and feeds
With our casting complete and locations scouted, we were joined by some friends to help crew a month of constant little video shoots all over town to capture a few hundred segments ready to edit. Then we brought in Doyle, the best Director development expert we knew of in Dallas, to start media integration.
At first, everything was going swimmingly, as Doyle imported all the objects (cast members) into Director, and the gameplay atop video was already pretty solid, but that was happening on his Mac’s hard drive. Then, we burned our first test disc. Turns out, getting a normal, single-speed PC CD-ROM drive to play video within the game’s camera interface without hiccups and load delays was way out of reach.
To overcome the performance hitch, we ran a matrix test—burning more than 50 versions of one high-activity video onto a disc, with different video compression codecs, frame rates and keyframe settings, plotting each on a performance curve.
We settled on Cinepak, at 115K/second, with a keyframe every 2 seconds. This produced the best compromise between start time and smooth play, with acceptable image quality and compression artifacts in the game’s little 320X240 video monitor.
Further efficiencies were realized when Doyle figured out that if we could arrange some videos and sounds closer to the center of the disc, when the player flipped through TV channels, they loaded quicker. Longer, larger encounter video assets were farther out, because those preload times could be covered by a travel cut sequence.
We came up with a naming convention that alphabetized quicker clips with 8-character filenames starting with the letter A, so they got written closer to the center of the disc.
Acceptance testing of unacceptable behavior
We had a continuous stream of friends and acquaintances coming in to test our game software for functional bugs, especially ones related to game continuity (i.e. playing the right video for the player’s current status) and controls (i.e. the buttons will click responsively and execute the right action).
Unfortunately we couldn’t keep using the same testers. We needed fresh ones who would actually walk through the game for a few hours, and explore their options. People who have already finished the game could speedrun through it in under an hour without following up on clues, and miss most bugs.
Anyway, as Doyle and Jim were closing in on the final stages of game development, and Joe and John were helping arrange tests and demos, and reporting the last of the bugs, I was done with everything including the package and disc design.
To reach a broader market, we definitely wanted a game that ran on PCs just as well as Macs, and the game did, with just one hitch: where Macs allowed you to put an “autostart” file right on the disc, PCs on Windows 3.1 didn’t allow them at the time. We needed a Windows installer executable and since I was the only one who ever used a PC at home, I took over the role of Windows Release Manager.
I called TigerDirect and ordered the latest copy of InstallShield. The box of floppies came with a little guide, describing commands for a C: type language library with specific commands for presenting the user with any selections they might need to make as it moved files over.
So, I had programmed my first INSTALL.EXE file and it moved a little player.exe game file and a couple support .dlls and QuickTime 1.1 onto the Windows C: drive that could launch and use assets on the two discs.
After printing a short run of about 1000 copies of PC and Mac versions of the game, leveraging just about our last dollar of funding, we were ready to get the game out. This is going to be better than Myst for sure!
Install once, run almost anywhere
We sent out copies of Paparazzi! to Multimedia World and Wired, and a few dozen other pubs, and prepared ads to run in magazines that reviewed us kindly. We had a launch party, and rented a limo. We set up a 1-800 number where you could call us and we’d hand-fulfill each game order for now.
Soon after, I was alone in the office wrapping up a set of game shipments. I got the call from a reviewer at PC Gamer (Remember how many magazines there used to be like that?).
“The installer says it’s looking for Drive C, and it keeps slowly looking for C, then D, then E… This is crazy—I’m going to have to put this in my review, you know.”
At this point, I could have tried to hustle the guy, make some kind of brush off statements that indicated it was his lack of understanding that caused the problem, but I knew we were cooked. So I pleaded.
“Please, please, don’t review it yet. This would kill our little company. We are going to test it out, and send you a new one.”
“What’s the point kid, if you were already selling this one?” he said. [click]
Murder on the high C:
Despondent, I fired up the Compaq 486, and opened up the Win3.1 user’s guide. Sure enough, you could just change the drive letters in System settings, or in the config.sys file, so I changed the hard drive to Z:.
Looking for drive C:… Looking for drive D:… it would time out for about 30 seconds for each drive letter, before it moved onto the next. All the while, the status bar was showing me all the dumb jokes I put in there while it stood still.
“Draining the silver nitrate.” “Readjusting flux capacitor.” “Why are you still reading these?”
At the time, I had seen a few PCs with extra drives, but in almost all cases the first 4 addresses would be the same, as thus: A: would be floppy, B: would be nothing or a second floppy in case you are a floppy copier, C: would be the hard drive, and D: would be the CD-ROM drive.
The installer marched on. Trying Y: “Filling up on Phillips” … Trying Z: … and it loaded! I called the reviewer back.
“I changed my hard drive to Z, and it kept going, which is bad, but it eventually got there and loaded the game. Plus, you can change the drive letter real easy in System.”
“Too bad, my hard drive is on B. And, I’m not changing my drive letter just to play your dumb game. [click]”
It never occurred to me that a user would want to change their own drive letters, but that’s exactly what he did – he put his hard drive on B: and the installer on the disk started on C: — I didn’t bother to ask for the address in the installer!
Oh man, we’d have to scrap all the PC versions. That’s most of the market.
Or, use a credit card to press another thousand 2 CD sets for PCs, and print 1000 more of these intricately folding sleeves I designed with the breakaway mailer wrappers, because the adhesives were already used. This epic fail was on me. I took a lozenge and tucked into the office couch.
The Best Kind of Failure
There’s always a new sunrise. We got some excellent reviews from Multimedia World, New Media, Wired–-who all had the Mac versions. The PC Gamer guy never published one, but a couple other Windows reviewers played our first edition game just fine on their C: drives.
Because of the great reviews, we got interest from a publisher (Gaga) in Japan, and Activision in the US, and they signed up to distribute Paparazzi to every Babbage’s, CompUSA and Circuit City store. Of course, they wanted it to be under their own brand, so they replaced my clunker installer with their own proper one and redid the packaging with their own UPC code and all. The future was wide open.
In addition to not being great at testing, we were also not very good at negotiating contracts, so the Museworthy dream crumbled soon after. But that’s a story for another day.
You can still see a pretty cool complete walkthrough of the game here. Kind of NSFW content, but the game resolution is so low, it’s as fuzzy as those distant memories.
The Intellyx Take
I’ve always said that testing is doing. The best testing practice should uncover a few things that developers and even dedicated testers wouldn’t even think about testing.
Has software quality gotten better, in today’s age of automated testing, DevOps automation, and now AI-driven testing? The fact is, our old desktop game was a million times simpler to validate than today’s distributed cloud native applications.
You’ll still need an honest reviewer. Who would have thought he would wake me up again, 30 years later?
Copyright ©2024 Intellyx B.V. Intellyx is an industry analysis and advisory firm focused on enterprise digital transformation. Covering every angle of enterprise IT from mainframes to artificial intelligence, our broad focus across technologies allows business executives and IT professionals to connect the dots among disruptive trends. As of the time of writing, none of the organizations mentioned in this article is an Intellyx customer. No AI chatbot was used to write this article. Image source: Screens from Paparazzi! Tales of Tinseltown, Museworthy, Inc., published 1995.
