When discussing the pros and cons of software development, especially the testing side, much of the focus is placed on the process. But sometimes it helps to see, accurately and clearly, just how much money and time is saved in the end, so BMC commissioned Forrester to conduct a study of cost savings and business benefits enabled by BMC AMI DevX Total Test. To do this in a way that would drive home some numbers with unmistakable clarity, Forrester interviewed four BMC customers—a total of seven decision makers—and then compiled their responses into a composite profile, which it calls a Total Economic Impact™ (TEI) study.
Manual testing is both time-consuming and unreliable
Prior to adopting BMC AMI DevX Total Test, the customers surveyed had all relied on manual testing , which presented significant challenges. Because the organizations operated in highly regulated environments and their mainframe code supported business-critical systems, rigorous testing of code changes was required. In some cases, changes were still being made to programs that were decades old, with some containing millions of lines of code.
Manually performing thorough integration and regression testing on such a scale was not only time-consuming—the customers interviewed typically budgeted about three days for testing in each ten-day sprint—it was also unreliable. Given such a high volume of tests, some errors went unnoticed, and in some cases, testers accidentally introduced errors during testing. The pressure to innovate, develop, and deliver code faster exacerbated these issues.
The study also found that not only did manual testing take time and introduce errors, it also took time away from opportunities for innovation, such as enabling developers to write code that supports new business logic. In other words, it’s not just about the time spent doing an outmoded procedure that needs to be addressed, it’s also about what they’re not doing that’s costing a company in time, money, and competitiveness.
For example, from a team perspective, writing test scripts took about as much time as performing a manual test. But unlike a manual test, the test scripts were reusable and could be run automatically. This meant that when a program had to be changed, they were able to save up to 90 percent of the time that had been dedicated to manual testing. Not only was the speed and quality of code development increased, the time saved was reinvested.
The four big numbers
In terms of quantifiable benefits, Forrester estimated the risk-adjusted present value of BMC AMI DevX Total Test over three years to be:
- $4.5 million worth of time savings and increased developer output, or about one day every two weeks.
- Twenty percent fewer bugs—saving $376,000—by finding bugs that manual testing would have missed, and finding them earlier.
- $2.9 million saved with automation efficiencies that allowed more tests in less time, and improved code quality.
- $2.6 million saved by reducing the risk of major mainframe defects and preventing negative customer impact from bugs in production.
From this analysis, Forrester determined that the composite organization yielded benefits of $10.34 million on an investment of $3.4 million, for a net present value (NPV) of $6.95 million and an ROI of 205 percent.
The marketplace
The research was conducted in the banking and financial services industry and the healthcare sector, both in North America and EMEA, with companies that had developer populations in the hundreds, and in one case, numbered 2,000. It is a crucial time for businesses seeking to recover from the impacts of the global pandemic and maximize their potential for long-term growth and sustainability in a period of high-velocity digital transformation. The relevance of mainframe technology in supporting the huge demands of data and speed in every industry is beyond question, but the constant caveat has been the need to update the testing process to bring it more in line with distributed technologies.
As the survey demonstrates, the benefits of incorporating automated testing cover both sides of the marketplace—improving quality and innovation inside a company’s walls while delivering a reliable product to the end users outside.
To learn more about the benefits and savings realized by these customers after adopting automated testing with BMC AMI DevX Total Test, join our October 7 webinar and read the full Forrester TEI report. A one-page infographic summary is also available. To calculate the potential financial impact of Topaz for Total Test for your own organization, check out Forrester’s companion Business Case Builder.
