Software testing is expensive yet
an important phase to ensure quality of a system. Testing can be done manually
and also automatically. Manual testing requires human effort for the execution
of test cases. While automated testing automatically executes test cases on the
system following some standard software solution. The objective is to provide
an overview on existing studies on the effect of test automation on the
software cost, time and quality.
is possibly the very expensive and time consuming phase of software
development. According to estimation, testing phase consumes over 50% of the
total project resources 1. Additional expense to software producers except,
testing cost, is cost related to poor quality and malfunctioning programs and
errors 2. According to estimation 2, due to poor testing and errors
detected by the customers in U.S. software producers lost about 21.2 billion
dollars. If the expense caused by errors to the users is added in this, the
cost rises to 59.5 billion dollars, out of which 22.2 billion can be saved by
proper testing of the system 2. So improving the testing process is an
effective process to ensure quality and reduce cost.
Testing phase can be considered as an important step that
can help to improve software’s cost, quality and time to market. Testing of a
software system has two categories: a) Manual Testing and b) Automated Testing.
In Manual Testing human tester initiates various tests over the system and then
observe the behavior of the
System and then prepare the
reports and estimate the quality of the system according to observation 3,
4. While, in automation testing some standard software solutions is used
which helps to control the execution of test cases on Software Under Test (SUT)
5. Automation testing has direct impact on the software cost, time and
3. BACKGROUND STUDY
explains test automation as a manner to reduce or totally remove the
involvement of humans in testing process by replacing it with automatic tools
6. Test Automation has may benefits, but it also have some limitations 7.
Table 1: Benefits and
Limitations of Software
Test Automation 7
quality of product
replace manual testing
Demand hard strategy planning
Difficult to maintain
Long maturation time
May failure to
achieve expected goals
Need very skilled
Often cause false
Two things must be kept in mind before performing
test automation: (1) to know well what will be automated; (2) to know the
difference between what can be, cannot be and is worth to be automated 7.
Mostly software organizations miscalculate their implementation process by
creating non-viable expectations about test automation 8.
b) Basic Software Test Automation Lifecycle:
Software test Automation is not a single step
activity. In fact, it is a complete set of multiple activities, it is a process
and tools require proper planning and separate resources 9. The author also
explains that a simple test automation process is composed of five activities
He also says that for a successful automation process the team must be properly
trained and sufficient resources must be provided. Dustin et al. 5 ,
developed a lifecycle methodology to bring systematic approach to the automation
process named Automated Testing Lifecycle Methodology (ATLM). This lifecycle
has 6 stages: 1) decision to automate test; 2) test tool acquisition; 3)
automated testing introduction process; 4) test planning, design and
development; 5) execution management of test; 6) test program, review and
assessment. Each stage require separate resources for its execution. A basic
automation lifecycle is presented in 6, 10 & 9 composed of four
phases: Selection ,Modeling, Execution and
4. LITERATURE REVIEW
The process of software testing can be roughly divided into
manual and automated testing. Ramler & Wolfmaier 11 presented difference
between automation and manual testing and suggest that manual testing is suited
for detecting new and unexpected errors, on the other hand automated testing is
used for the prevention of further errors in the modules.
The total cost of testing in the life time of project both
for manual and automated testing is compared if the cost of automation testing
is less than the cost of the manual execution of testing then perform
automation testing 6.
Industrial software or other large software are developed
iteratively that is in multiple versions. As the system grows more
functionalities are added in the software. Multiple developers work together to
develop a single software. With each new release a new functionality is added
in the system this makes it impossible to test each modified version of all
developers. This is time consuming and requires more resources ultimately
result in high cost. So it is difficult to decide which pre-existing
functionalities requires testing 12.
In 13 Damm et al. explains how
test driven development and test automation can be combined and used in
industry. They presented a cost effective approach, using C++ to write both
software and test tools, for early defect detection. In many cases resource saving is the
important discussion for test automation. A simple solution present by S.
Berner et al. 14 is to implement test automation only on test cases and on
those tasks that are implemented manually before. Kasurinen et al. find that in
industry 26% of the test cases are
tested using automation process and this process is mostly
used for quality control and
quality assurance 16.Test case automation helps the organization to increases
the number of test cases they execute and also increases the frequency of test
execution 6. In 11 Ramler & Wolfmaier presented a cost based model
which helps to decide automation strategy and to reduce testing cost,
automation is an effective solution, suggested by them. All of the papers as discussed
above, only explain the importance and significance of test automation process
on the software but none of them explain the impact of automation process on
the software. In 12, Kumar et al.
develop a formula to calculate the impact of test automation on software time,
quality and cost.he observes positive impact of automation on cost and time and
in most of the cases quality is also improved. Test automation also decreases
overall testing time.
5. AIM OF RESEARCH
Testing is an important process in software development
cycle to ensure quality of a system and deliver a system with minimum defects
as possible. Because zero defect is not possible and it must not be a goal
instead satisfying user quality expectations must be the goal 16. Testing is
an expensive and time consuming process and also requires larger amount of
resources. To minmize the cost and time advancements in the field of testing are
made one of the popular change is automation testing that is automate the
testing phase. The aim of this research is to understand how software cost and
time needed for testing is affected and also what is its impact on software
6. RESEARCH QUESTIONS
Q1: How test automation effect software cost and time?
Q2: How test automation effect the quality of the system?
This paper helps its reader to
understand test automation its life cycle, benefits of test automation and
limitations of software test automation.one can understand the impact of test
automation on software cost and time required for automation process. This
paper explain how quality of a software is affected by software test
Software testing has key
importance for verification and validation of a system. The reason is testing
ensures the quality of the system and also testing consumes almost 60% of the
system cost. Automation testing is a simple solution to reduce software cost
for testing but it requires expertise and detailed knowledge about the process.
It also increases the efficiency of the testing process.
The literature review
shows that automation testing not only reduces cost and time required for
testing but also increases the quality of the system to a significant level.
A survey can be conducted in order to
views of professionals about test
and its impact on the software cost, time
1 E. Kit, Software Testing in the Real
Process, Reading Mass,USA: Addison-Wesley, 1995.
2 G. Tassey, The Economic Impacts of
Inadequate Infrastructure for Software Testing, National Institute of Standards
and Technology: Diane publishing, 2002.
3 B. Hetzel, The Complete Guide to Software
Testing, MA, USA: QED Information Sciences, 1988.
4 C. S. &. T. B. Glenford Myers, The
Art of Software Testing, John Wiley & Sons, 2011.
5 J. R. &. J. P. Elfriede Dustin,
Automated Software Testing: Introductio, Management and Performance,
Addison-Wesley Professional, 1999.
6 C. S. &. M. Gilpin, Evaluating
Automated Functional Testing Tools, Forrester Research, 2005.
7 K. R. K. M. K. P. *. M. V. M. Dudekula
Mohammad Rafi, “Benefits and LImitations of Automated Software Testing:
Systematic literature review and practitioner survey,” in 7th International Workshop on Automation of
Software Testing (AST), Zurich, Switzerland, 2012.
8 B. Pettichord, “Seven Steps to Test
Automation Success,” November 2001.
9 P. Pocatilu, “Automated Software
Testing Process,” Economy
Informatics, vol. 1, pp. 97-99, 2002.
10 D. S. S. E. &. K. L. Kristian
Wiklund, “Impediments for Automated Testing–An Empirical Analysis of a
User Support Discussion Board,” in IEEE
Conference on Software Testing, Verification and Validation, Clevekand. OH, USA, 2014.
11 R. R. &. K. Wolfmaier, “Economic
perspectives in test automation: balancing automated and manual testing with
opportunity cost,” in 6th
Proceedings of Internatioal Workshop on Automation of Software Test, New
York, USA, 2006.
12 D. K. &. K. K. Mishra, “The
Impacts of Test Automation on Software’s Cost, Quality and Time to
Market,” Procedia Computer Science, vol.
79, pp. 8-15, 2016.
13 L. L. &. D. O. Lars-OLA Damm,
“Introducing Test Automation and Test-Driven Development: An Experience
Report,” Electronics Notes in
Theoretical Computer Science (ENTCS), vol. 116, pp. 3-15, 2005.
14 R. W. &. R. K. K. Stefan Berner,
“Observations and lessons learned from automated testing,” in 5th Proceeedings of the 27th International
Conference in Software Engineering, New York, USA, 2005.
15 O. T. &. K. S. Jussi Kasurinen,
Practice: Empirical Observations,” Advances
in Software Engineering- Special issue on Software Test Automation, no. New
York, USA, 2010.
16 F. F. R. &. S. J. W.E., “Zero
Detection: Quality Comes to Services,” Harvard
Business Review, vol. 68, pp. 105-111, 1990.