A Survey on Continuous Testing

A survey was conducted to find out the level of adoption and the maturity of testing that exists in the software industry. The survey named Continuous Testing Report was sponsored and conducted jointly by Capgemini, Sogeti and CA.

Continuous Testing is one of the critical pieces in fulfilling a critical DevOps objective. It ensures that the speed of testing stays abreast with the development speed, ensuring faster delivery of user stories.

The results of the survey was as I expected it to be. The pains that we normally go through have come out accentuated. I don’t want to sound pessimistic but knowing that the challenges we face in software projects is common across organizations and industries gives an air of confidence that we are not lagging behind in the race.

Time Spent on Requirements

Majority of respondents have indicated that about 40-70% of the overall project time is spent on gathering requirements, clarifying them and communicating to teams.

This is alarming because the actual development and testing is squeezed, and the shrink indicates that the developers and testers are not getting enough time to put in their best efforts on quality control into production.

A co-located agile setup with a Product Owner and possibly a Business Analyst working with the teams is the best way forward to arrest such time and effort losses.

Test Environments

This is my favorite one. Test environments. I have faced plenty of delays and have missed deadlines thanks to the environment setup time.

According to the survey report, 81% of the respondents have indicated that they have lost a third of their project time trying to get the test environments right.

The test environment setup is one of the critical pieces and not getting it right stops the pipeline from flowing. And this can have drastic effects on the delay of feedbacks, duplication of efforts and the worst of all, it could start giving a flavor of waterfall with a number of features going into UAT or Production as a bundled release.

The fix is not simple. Virtualization has helped but the answers do not lie in creating environments at the drop of a hat but rather turning to containers to duplicate the efforts of test environment setup.

The survey further notes that about 36% of the respondents feel that wait times and the financials behind environment setup is a critical challenge with no easy way out.

Test Case Automation

To implement continuous testing, one of the first things that we must achieve is to automate testing all the user stories. Once we have automated test cases, then the job would be to put them in an orchestration loop.

The survey indicates that only 22% of the test cases are automated today. Of course, this is the average across the board but I did expect this number to be in the highest quadrant as automating test cases is fairly standardized today.

Until the test automation goes closer to 100% target, testers will always play catch-up and developers will always find reasons to pin the blame for delays on testers.

Survey Methodology

The Continuous Testing Report study was finalized based on 500 interviews that were conducted in June and July 2018. Survey respondents were in senior positions (decision makers) and based across eight countries.

Only organizations with over 1000 employees were chosen for this survey. Respondents came from varied industries. 40% of the respondents came from organizations that have 1000+ employees, 40% from 5000+ employees and the remaining from 10,000+ employees.

Read the complete report here.