DevOps and test automation are important elements of an effective software development organization. Some organizations are having difficulty justifying the investment in test automation, particularly for legacy systems that were not automated from the beginning.
This article describes some benefits and a financial model that has been used to justify investment in test automation. I am not going to discuss the technology to use for test automation, In general though,there are four major types of tests that can be automated:
- User Interface tests – Automate the entry and display of data through user interface screens.
- Unit tests – Tests individual objects and methods within a system.
- Load and volume tests – Generate high volume tests simulating large batch processes and high user volume.
- Service testing – Tests that exercise the major service components in a system without going through the user interface.
Look for more discussion on these test categories in future posts.
Why Automation
Manual testing alone is not sufficient in a volatile, continuously changing, and complex system. Continuous integration is desirable but it must be accompanied by continuous testing in order to be effective.
The impacts of high defect rates are very significant:
- High support overhead
- Elongated release delivery cycles
- Lower development velocity
- Poor customer satisfaction
Benefits of automation
By using automation effectively you can significantly increase test coverage and implement true continuous integration and regression testing from coding through software delivery
Some of the benefits are:
- Decrease cycle time and improve development throughput by providing continuous regression testing tools to the developers. Finding defects at the earliest stages is by far the most efficient and least expensive.
- Enable deeper levels of functional testing within a development cycle. Critical tester time is not consumed running trivial regressions tests, freeing them to run higher-value, more complex tests.
- Improve quality of the code base leading to higher customer satisfaction and reduced support overhead.
- Mitigates higher levels of expected defects if you are taking on of new developers regularly.
- Provide a safeguard for quality when you are making significant process changes.
Creating Flow
Agile in general and Kanban specifically recommend creating flow through a continuous deployment strategy – this dramatically improves the speed of feature delivery to customers.
A proper Agile test and integration strategy approaches testing as a cumulative and continuous cycle where features are driven from Specification through Deployment. Regression testing will provide much higher quality but must be automated in order to be truly effective
The excerpts below describe the reality of the requirements for our test systems in today’s world:
“Applications change, grow and gain complexity over their useful life. The feature set grows steadily over time. Therefore the number of tests that are needed for adequate coverage is also constantly increasing.. That is why manual testing can’t keep up – unless you constantly increase test resources and cycle time, your test coverage will constantly decline. Automation can help this by allowing you to accumulate your test cases over the life of the application so that both existing and new features can always be tested.”
“The irony is that greatest risk to the user is in the existing features, not the new ones! If something the customer is already doing stops working – or worse, starts doing the wrong thing – then you could halt operations. The loss of a new feature may be inconvenient or even embarrassing, but it is unlikely to be devastating.”
– Automated Testing Handbook by Linda Hayes
The Impact of Defects on Throughput
If teams are struggling to meet planned sprint goals, it is important to understand that defects usually make up the majority of “UNPLANNED WORK” – i.e. the work involved in fixing defects is not generally factored into the estimating process. The impact on resource utilization and delivery time frames are significant.
Automated regression tests run on team servers and integration test servers will lessen this impact enormously by enabling detection very early in the development cycle.
The following chart shows the impact of defects on the cycle time of a real team whose productivity was observed at various defect levels.
Defects Found by Phase
It is well known that defects cost more to fix the later they are found in the development cycle. The worst case scenario is when it is actually found by a customer. The way I justify test automation is not that it will reduce the total defects in the code, but that it will change the distribution of where they are found. This will lower the cost to find and fix them. Think about the difference in cost between a developer spotting a missing “if” clause before he compiles the code on his desktop versus a system outage to a high volume production financial network. There is no comparison in terms of direct cost and customer impact.
Savings from redistributing defects
If a project was to achieve the redistribution of defects shown above. It would result in some significant cost savings.
To determine the cost savings, I first made some assumptions. They are:
- A blended cost of $65 US for all resources, onshore and offshore.
- Industry standard data for the multiplication factors to use for determining the cost of a defect at each stage,
- A total defect rate of 1000 items per release cycle
- The same number of total defects will be found in both cases, they are just found earlier in the development cycle in the second model.
Redistributing the defects to the target shown above would result in a savings of approximately $950K US ($2.66M minus $1.7M) per release cycle. This is shown below.
Summary
The benefits from test automation can be significant. This capability is essential to an effective Lean and Agile development organization. I hope this helps other to justify this critical investment.
Thanks to Bennet Vallet for his contribution to this article. He can be found at Bennet Vallet on LinkedIn.
streaming Agile 