\documentclass{article}
\usepackage{fullpage}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage{xcolor}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage[natbibapa]{apacite}
\usepackage{eso-pic}
\AddToShipoutPictureBG{\AtPageLowerLeft{\includegraphics[scale=0.7]{powered-by-Authorea-watermark.png}}}

\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}

\usepackage{authblk}

\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}




\iflatexml
\documentclass[]{article}
\fi
% \usepackage[affil-it]{authblk} % Not (yet) supported by LaTeXML.

\usepackage[]{graphicx}

\usepackage[space]{grffile}

% \usepackage[]{latexsym} % Not (yet) supported by LaTeXML.

% \usepackage[]{textcomp} % Not (yet) supported by LaTeXML.

\usepackage[]{longtable}

\usepackage[]{multirow}

\usepackage[]{booktabs}

\usepackage[]{amsfonts,amsmath,amssymb}
\usepackage[]{amsfonts,amsmath,amssymb}
\usepackage[]{amsfonts,amsmath,amssymb}

\usepackage[]{url}

\usepackage[utf8]{inputenc}

% \usepackage[polish,english]{babel} % Not (yet) supported by LaTeXML.

% \usepackage[]{csquotes} % Not (yet) supported by LaTeXML.

\usepackage[]{graphicx}

\usepackage[]{array}

\usepackage[]{multirow}

\usepackage[]{amsmath}

% \usepackage[]{lipsum} % Not (yet) supported by LaTeXML.

% \usepackage[]{biblatex} % Not (yet) supported by LaTeXML.

% First macro on next line not (yet) supported by LaTeXML: 
% \addbibresource{references.bib}
\newcommand\blfootnote[1][]{%
  \begingroup
  \renewcommand\thefootnote{}\footnote{#1}%
  \addtocounter{footnote}{-1}%
  \endgroup
}


\begin{document}

\title{Assessment of Defect Prediction Cost Effectiveness of an Industrial
Software Development Project Supported by DePress Framework}


\author[ ]{Jarosław Hryszko}

\affil[ ]{}
\vspace{-1em}


\date{}

\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup

\selectlanguage{english}
\begin{abstract}
The potential benefits of using DePress (Defect Prediction in Software
Systems) Extensible Framework tool for commercial software development
projects have not been previously investigated. Documented cases of
machine-learning usage in industrial applications for defect prediction
purposes, are rare. Due to these facts, representatives of Wroclaw
University of Technology and Volvo Group began a long-term cooperation
for that purpose. As achieving a positive result of research described
in this paper could trigger more interest from business stakeholders, we
decided to verify the cost effectiveness of DePress framework usage for
defect prediction purposes and investigate, if the defect prediction
technique would positively impact software development projects by
generating profits. To meet this goal, we proposed using a defect
prediction-based quality assurance (QA) effort allocation strategy based
on the Pareto principle and Boehm's law. Then, based on real life data
collected from an actual, industrial software development project using
DePress Extensible Framework, we conducted a defect prediction and
simulated potential quality assurance costs based on the best prediction
result and proposed QA strategy. Results of the simulation have been
optimistic and have resulted in continued usage of DePress based defect
prediction for actual industrial projects run by Volvo Group.%
\end{abstract}%



++

\textbf{Short, edited version of this paper was published with Lech
Madeyski as ''Assessment of the Software Defect Prediction Cost
Effectiveness in an Industrial Project'', in Software Engineering:
Challenges and Solutions, vol. 504 of Advances in Intelligent Systems
and Computing, pp. 77-90, Springer, 2017. DOI:
10.1007/978-3-319-43606-7\_6}

\textbf{}

\section*{Introduction}\label{introduction}

The first attempts to use machine learning for software development
quality assurance, were made in the early 1990's (Munson et al.
1992)~\cite{Munson1992}. Since then, this approach has
gradually improved. Why then, has it not gained wider popularity in
commercial projects so far? One reason is the wide variety of data
necessary to perform a prediction as well as the many different sources
of data which are encountered in commercial software development
projects. Until recently, conducting a prediction on a chosen project,
required time-consuming preparation of a special, additional program (or
programs) that acquired the desired data and then put it through a
suitable preprocessing procedure. One obstacle was the need to gather
data from varying sources (tools, processes, methodologies, etc.) as
well as the data itself having differing sizes and measurements. Also,
different machine learning mechanisms have a different degree of
effectiveness depending on the choice of data sources and data itself
(experimentally selected). Software defect prediction has been
considered to be too complex a process, too cost and time-consuming, and
there have been no known solutions for wrapping it into one, universal,
defect prediction application which could be used for different
projects.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/depress-bw/depress-bw}
\caption{{DePress tool%
}}
\end{center}
\end{figure}

To fill that gap, Madeyski and
Majchrzak~\cite{Madeyski2014} proposed using a new tool
called DePress. DePress (Defect Prediction for Software Systems), based
on the KNIME framework~\cite{KNIMESite}, allows the
creation of graphical workflows and uses an intuitive, user-friendly
interface (Figure~\ref{fig:workflow}). Possible
applications of DePress vary from software measurement, product and
process improvement, quality assurance processes, integrating data from
various tools and other sources for the sake of data analyses as well as
its main task: KNIME's data mining capabilities for software defect
prediction purposes. Being intuitive and highly customizable, DePress
makes itself a perfect tool which can be conveniently utilized (thanks
to its user-friendly interface) in different commercial software
development projects for defect prediction introduction and is freely
available for any purposes under its GNU General Public
License~\cite{DePressSite}.

Potential benefits from usage of DePress framework for commercial
software development projects have not been
investigated~\cite{Madeyski2014}. To fill that gap, our
study aimed to answer the following research questions:

\textbf{RQ1:} \emph{What is the highest level of prediction recall
achievable by DePress tool in basic configuration mode, using an
industrial project's data?}

The possible benefit varies, depending on the potential prediction
effectiveness. This implies the need of first verifying what is the
highest recall value of the defect prediction process handled entirely
by the DePress tool. As DePress can be highly customizable thanks to its
plugin-based architecture, as well as being open source can make
possible adjustments complex and expensive. Therefore, we decided to
restrict the DePress usage only to its basic set-up.

\textbf{RQ2:} \emph{Will usage of the DePress framework pay off for an
industrial project?}

To answer this question, we had to compare the costs of introducing and
using the DePress based defect prediction to the potential benefits
generated by its introduction. To achieve this, we used values such as
return on investment (ROI) and benefit-cost ratio (BCR).

\textbf{RQ3:} \emph{How cost effective is the DePress framework for
defect prediction for an industrial software development project and
what is the highest possible prediction recall achievable with that
tool?}

If the answer to the first research question is positive, the next step
is to verify what will be the profit from the highest prediction recall
achievable using DePress' basic defect prediction set-up.

\section*{Project Context}\label{project-context}

Volvo Group, one of the leading automotive companies, was invited to
take part in this research. The primary motivation for Volvo Group's
interest was to verify, if their company can use DePress and its
software defect prediction to increase quality and cost-effectiveness of
quality assurance (QA) in their software development projects.

\subsection*{Target Software}\label{target-software}

An important criteria for selecting the proper software development
project for the purpose of this research, was that such a project would
be mature enough, to provide historical information which can be used as
a source of training data. A special survey was conducted by Volvo Group
to select the most suitable candidates for initial research.

During our previous research, we recognized elements occurring in
projects that hindered or prevented completion of a
prediction~\cite{Hryszko2015}. The most important of
these elements are:

\begin{itemize}
\tightlist
\item
  Unavailability of data necessary to create a prediction model;
\item
  Lack of possibility to link code changes with defects;
\item
  The lack of, or incorrect version of available data;
\item
  Inappropriate use of a version control system.
\end{itemize}

One of the projects finally selected as a business case subject, was an
initiative which develops and maintains an application called Texas, a
project which provides all the data necessary to achieve the highest
possible recall of prediction such as:

\begin{itemize}
\tightlist
\item
  Information on defects found;
\item
  Registered code changes as a result of defects;
\item
  Code metrics.
\end{itemize}

In the considered project, it is possible to clearly distinguish which
code changes apply only to bug fixes. This is possible thanks to the
practice adopted by its developers: any modification of source code is
committed to the code repository with an appropriate comment. In the
case of a modification resulting from the fix of a particular defect, a
unique identifier is included in the comment. Moreover, such a feature
is an example of the correct usage of a version control system.
Information on the number of subsequent versions of the software are
available, and the naming of each version is standardized.

From an organizational perspective, Texas is a document management tool
that supports the Vehicle Type Approval (VTA) certification process for
vehicle components and completed vehicles produced by Volvo Group. The
VTA certificate confirms that the production samples of a design will
meet specified performance standards. Key users of the Texas tool are
Volvo Group's Certification Managers and engineers, as well as their
brand representatives and European market companies. The certification
process is crucial for the company, as availability of Volvo Group's
products directly depend on it, making the reliability, development and
maintenance of the Texas software highly important. Defects in the
application can delay the work of certification engineers, which is
unacceptable as such delays affect the scheduled dates for approval of
certificate documents and publication process and, therefore have a
negative impact on the date of product availability.

From a technical perspective, Texas is a Java-based application written
using the Java Enterprise Edition computing
platform~\cite{JEESite}. The main development
environment used for the development process is Eclipse Integrated
Development Environment~\cite{EclipseSite} and Apache
Maven~\cite{MavenSite}, its main build automation tool.
To assemble a complete application, 13 different Maven projects needed
to be built by default. For the purpose of this research, 21 consecutive
software releases (versions) were available, ranging from Texas version
4.0.0 to version 6.0.0. The source code is stored in a code repository
managed by Apache Subversion (SVN) version control
system~\cite{SVNSite}.

Within the considered project, we can observe three stages of the
software life-cycle (project phases):

\begin{enumerate}
\tightlist
\item
  Development
\item
  Testing
\item
  Post-release
\end{enumerate}

\subsection*{Related Work}\label{related-work}

The first publication related to an industrial application for defect
prediction was published in 1997 by Khoshgoftaar et
al.~\cite{Khosh1997}. It was a case study of quality
modeling for a very large telecommunications system. The authors of that
work used a neural network to model a future fault proneness for a
real-world system managed by a telecommunication company. However, the
final results were not used to support the company's quality assurance
procedures. Two other publications of Khoshgoftaar and Seliya from
2004~\cite{Khosh2004} and
2005~\cite{Khosh2005} continued with the previous
concept and focus on commercial data analysis, but were not applied to a
real-world environment. A similar approach is common for most research
projects utilizing industrial data - examples can be found in different
publications by
Ostrand~\cite{Ostrand1},~\cite{Ostrand3},
Tosun~\cite{Tosun2} and
Turhan~\cite{Turhan1}, \cite{Turhan2}.
Examples of industrial applications of information gathered by using
defect prediction can be found in such publications as
\cite{Wong}, \cite{Succi} and
\cite{Klas}. Complete cases describing the introduction
of defect prediction in industrial environments were presented by
Ostrand~\cite{Ostrand2}, Li et
al.~\cite{Li} and Tosun~\cite{Tosun1}.

\section*{Model Construction}\label{model-construction}

\subsection*{Training and Dataset
Evaluation}\label{training-and-dataset-evaluation}

Two factors related to the dataset used, greatly affect the quality of
defect prediction:

\begin{itemize}
\tightlist
\item
  Size of the dataset;
\item
  Number of defects found in the dataset's source release.
\end{itemize}

Size of the dataset is important, because in machine learning,
effectiveness of the process in most cases is proportional to the size
of the dataset: the larger the dataset, the greater the efficiency of
the machine learning mechanism.

Number of defects has a direct relation to the possible occurrence of a
class imbalance problem -- such as when the total number of data
instances from a single class of data (in this case defective) is far
less than the total number of another class of instances
(non-defective). If a relatively low number of defects were detected for
a relatively large application in a particular release, we should expect
a large class imbalance.

Considering the above factors, we analyzed each available release for
its size and number of defects registered. For the size determinant, we
selected the number of separate code modules (Java classes -- not to be
confused with data classes). To obtain the number of modules per each
release, code metrics had to be collected. Collecting code metrics has
to be conducted during the process of building projects (in this case by
Maven tool). To minimize impact on the Texas project team work, we
decided to copy the source code and build each release, project by
project, locally within the Eclipse Integrated Development Environment.
To collect the metrics during the build, Eclipse Metrics 2 tool was
used. Eclipse Metrics 2 is a CPL-license based Free Software tool, which
works as an Eclipse IDE plugin~\cite{Metrics2Site} and
was created as continuation of Eclipse Metrics -- original metrics
collection Eclipse plugin~\cite{MetricsSite}. Eclipse
Metrics 2 permits the collection of different kinds of code metrics
(Table~\ref{tab:eclipse_metrics}) and exports them to
an XML file. The metrics data can then be read by a dedicated DePress
node (also called Eclipse Metrics) and converted into DePress' internal
data format.

Java class-level code metrics measured by Eclipse Metrics 2 tool

Metric name

Metrics

NumberOfOverridenMethods

Total number of methods in Java class that are overridden from an
ancestor class

NumberOfAttributes

Total number of Java class' attributes

NumberOfChildren

Total number of Java class' direct subclasses

NumberOfMethods

Total number of methods in Java class

DepthOfInheritanceTree

In the inheritance hierarchy, distance from Java class object

LackOfCohesionOfMethods

Cohesiveness of a Java class calculated with the Henderson-Sellers
method

NumberOfStaticMethods

Total number of static methods in Java class

SpecializationIndex

Relation between number of methods and depth of inheritance tree

WeightedMethodsPerClass

Sum of the cyclomatic complexity measured for all methods in Java class

NumberOfStaticAttributes

Total number of Java class' static attributes

The best source for obtaining information about the number of defects
registered in each release is a tool used for defect tracking. In the
case of the Texas project, the software used for that purpose was JIRA
created by the Atlassian company~\cite{JiraSite}.
Similarly to Eclipse Metrics, JIRA also allows the export of defect data
into an XML file, and that file can then be parsed by a DePress node
called Jira Offline.

When transferred to DePress, size and defect data can be aggregated with
a few simple steps using the KNIME framework capabilities. Results for
each historical Texas release are presented in
Table~\ref{tab:bugs_per_release}.

After analyzing the data presented in
table~\ref{tab:bugs_per_release}, we decided to use
the dataset based on release 4.0.0, both for training and evaluation:
the highest number of detected defects were compared to number of code
modules available for analysis which made release 4.0.0 the most
suitable for the above purposes. For further research, releases 5.0.0
and 6.0.0 can also be used. Other releases -- with a significantly lower
number of registered defects to a similar amount of modules available
for analysis (defect per module ratio
\textbackslash{}(\textless{}0,1\textbackslash{})) made us assume that
strong a class imbalance would be observed. Because of this we decided
not do any further investigation of these releases.

Defects and number of modules per release

Release number

Defects

Modules

Defect per module ratio

4.0.0

837

744

1,125

4.0.1

19

685

0,03

4.0.2

23

673

0,03

4.1.0

45

597

0,08

4.1.1

17

546

0,03

4.2.0

54

608

0,09

4.3.0

10

660

0,02

4.3.1

8

608

0,01

4.4.0

29

616

0,05

4.5.0

11

747

0,01

5.0.0

270

744

0,36

5.1.0

15

593

0,03

5.2.0

23

606

0,04

5.4.0

35

582

0,06

5.5.0

42

582

0,07

5.6.2

10

612

0,02

5.6.3

1

612

0,002

6.0.0

462

688

0,67

Datasets for both purposes -- training and validation, were constructed
by splitting 4.0.0 data, using a built-in DePress' function called
Stratified Sampling, so that after the division, the ratio between code
parts classified differently would be preserved in both created
datasets.

\subsection*{Objective Variable And Class Imbalance
Counteraction}\label{objective-variable-and-class-imbalance-counteraction}

\par\null

For the purpose of this research, we chose two-value (``0'' and ``1'')
objective variable to distinguish the fault-prone module, where at least
one defect occurred (``1''), from the fault-free modules, where no
defects were observed (``0'').

To counteract against any possible class imbalance, we decided to
randomly remove some of the majority class instances. To follow our
initial approach, we used the basic mechanisms built into DePress/KNIME
by constructing a workflow as shown in
Figure~\ref{fig:class_imb_wrkf}. First, the dataset
is split into two parts, by classifying rows of two different sets,
depending on the objective variable value (``1'' or ``0''), using the
Row Splitter KNIME node. Then, the majority class instances were reduced
to reach exactly the same number of instances of minority class,
achieved by random sampling done with the Row Sampling node. Finally,
two equal size record sets were merged by Concatenate node to make one
dataset with an equal ratio between the classes as a result.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/class-imb-wrkf-bw/class-imb-wrkf-bw}
\caption{{Class imbalance counteraction%
}}
\end{center}
\end{figure}

\subsection*{Predictor Variables
Selection}\label{predictor-variables-selection}

\par\null

The total available dataset which can be used for machine learning, can
be modified to minimize the prediction error. Various techniques can be
used for this purpose, such as removing any unnecessary information, but
the most commonly used technique is Feature Selection -- a tool for
building a set of code metric types from all data collected by the
Eclipse Metrics 2 tool
(Table~\ref{tab:eclipse_metrics}), which results in
the smallest prediction error. During the Feature Selection process,
when the metric types set is relatively small it can be based on
different exhaustive search algorithms, covering all possible
combinations of the available metric kinds. For each combination
indicated by the algorithm, the process of prediction model creation and
its validation is conducted, and then the prediction error is
determined. The metrics sub-set that results in the smallest prediction
error, is then selected.

When there are a large number of available types of metrics, usage of an
exhaustive search algorithm is very expensive.
Pendharkar~\protect\hyperlink{Pendharkar2010}{(Pendharkar 2010)}
estimates that when an algorithm of such a type would be applied to a
data set consisting of 21 different metric types, it would take
approximately 120 years of calculation by a 900 MHz RISC processor based
computer to complete the Feature Selection process.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/feature-sel-bw/feature-sel-bw}
\caption{{Feature selection workflow%
}}
\end{center}
\end{figure}

The KNIME framework has a built-in Feature Selection functionality based
on the backward elimination
algorithm~\cite{KNIMEDocumentation}. It consists of a
loop constructed by two nodes
(Figure~\ref{fig:feature_sel}): Backward Feature
Elimination Start and Backward Feature Elimination End. In the loop
created by these nodes, the appropriate machine learning mechanism
should be placed, along with any additional supporting nodes, if needed.
In the particular example shown, Naive Bayes-based classifier nodes are
used -- respectively: Learner and Predictor. To divide input data for
training and validation sets, node Partitioning is used.

The backward elimination approach used in KNIME framework is carried out
in
\textbackslash{}(\textbackslash{}frac\{n\textbackslash{}times(n+1)\}\{2\}-1\textbackslash{})
iterations, where \textbackslash{}(n\textbackslash{}) is total number of
features (columns) in the input dataset (input table), and in this case
the total number of types of metrics collected
(\textbackslash{}(n=10\textbackslash{})):

\begin{enumerate}
\tightlist
\item
  In the first iteration, the loop is executed with all features
  (columns): the dataset is divided into two sets, a model is created by
  the Learner node using the first set, then validated by the Predictor
  node using the second set;
\item
  In the next nine (\textbackslash{}(n-1\textbackslash{})) iterations
  each input column is omitted once. Model creation and its validation
  is performed for each iteration and prediction results are collected;
\item
  The Backward Feature Elimination End node discards the column that
  influenced the result the least;
\item
  The process repeats until one feature (column) is left.
\end{enumerate}

Then the Backward Feature Elimination Filter node is used to filter the
actual dataset, using the best feature combination found as a result of
the above process.

\subsection*{Prediction Models}\label{prediction-models}

Due to fact that DePress, tool used in our research is based on data
mining framework KNIME, various types of fault-prone module predictors
can be used for the purpose of research. Since prediction results are
categorical (faulty or not-faulty), we decided to test classifiers often
used in software defect prediction~\cite{Hall2012,
Moser2008, Khosh1995, Selby1988}, which are available in the basic
package of KNIME:

\begin{itemize}
\tightlist
\item
  Naive Bayes,
\item
  Probabilistic Neural Network,
\item
  Decision Tree.
\end{itemize}

More information about build-in KNIME classifiers can be found in KNIME'
documentation ~\cite{KNIMEDocumentation}.

Prediction results -- modules marked as defect-prone or non-defect
prone, can be compared against actual data describing defect-prone
module distribution and were used to build the confusion matrix (Table
\ref{tab:conf_matrix}) -- a commonly used tool for
performance comparison across categorical
studies~\cite{Hall2012}.

Confusion Matrix

Code units

Predicted faulty

Predicted not faulty

Actually faulty

True positives (TPs)

False negatives (FNs)

Actually not faulty

False positives (FPs)

True negatives (TNs)

As the prediction performance measure for business case purposes was
chosen, the percentage proportion of code units predicted as defective
that were actually defective, are known as the prediction
recall~\protect\hyperlink{Weiss1987}{(Weiss 1987)}, defined as:

\begin{equation}
Rec=\frac{TPs}{TPs+FNs}\\
 \end{equation}

\section*{Assessment Method}\label{assessment-method}

To investigate the cost effectiveness of the defect prediction technique
applied to an industrial software development project using DePress
framework, we agreed to follow the plan below:

\begin{enumerate}
\tightlist
\item
  Creation of a QA effort allocation strategy, based on defect
  prediction provided by DePress;
\item
  Analysis of actual, real-life costs of quality assurance for the
  selected release (4.0.0);
\item
  Creation of prediction models for the chosen release;
\item
  Selection of highest prediction recall;
\item
  Usage of an effort allocation strategy, based on the prediction
  characterized by the with highest recall achieved, to simulate a
  prediction-based quality assurance usage scenario;
\item
  Comparison of actual (without DePress tool used) and simulated (with
  DePress tool) QA costs.
\end{enumerate}

\subsection*{Quality Assurance Effort Allocation
Strategy}\label{quality-assurance-effort-allocation-strategy}

\par\null

The Pareto principle can be observed in software quality (20\% of
program modules are responsible for 80\% of
defects)~\protect\hyperlink{Endres03}{(Endres 2003,}
\protect\hyperlink{Pressman2010}{Pressman 2010,}
\protect\hyperlink{Iqbal2009}{Iqbal 2009)}. Hence, any effective way to
recognize the mentioned 20\% of ``high risk'' computer code will help
with such quality assurance (QA) efforts allocation, so that the maximum
number of software defects will be eliminated using the available
resources within a limited time period. Additionaly, in 1976 Boehm
proved, that defect fixing costs grow in every following phase of the
software project~\protect\hyperlink{Boehm1976}{(Boehm 1976)}. That
observation, which is widely called Boehm's
Law~\protect\hyperlink{Endres03}{(Endres 2003)}, results in another
important consequence of smart quality assurance efforts allocation: the
earlier the QA actions will take place, the better it is from the
perspective of the software development project's budget.

Considering the above facts, we proposed a strategy which would use the
prediction model to indicate as much as possible the mentioned 20\% of
software modules responsible for 80\% of the bugs, therefore helping to
integrate as much as possible the QA efforts into the coding stage of
the software creation, while defect fixing cost is still relatively low.
Such an approach should ideally decrease the total cost of bug fixing in
the project and create savings for the total project's
budget~\protect\hyperlink{Slaughter1998}{(Slaughter 1998)}.

If we denote \textbackslash{}(M\_\{total\}\textbackslash{}) as the total
number of testable program's modules and
\textbackslash{}(H\_\{total\}\textbackslash{}) as the total number of
discoverable defects, we can say that
\textbackslash{}(0,8H\_\{total\}\textbackslash{}) comes from
\textbackslash{}(0,2M\_\{total\}\textbackslash{}).

We can expect that:

\begin{equation}
0<Rec<1\\
 \end{equation}

Where \textbackslash{}(Rec\textbackslash{}) is measured as the highest
possible recall of defect prediction performed by DePress framework,
being the number of modules correctly indicated (predicted) as
responsible for 80\% of discoverable defects
\textbackslash{}(M\_\{i\}\textbackslash{}) should be:

\begin{equation}
M_{i}=0.2\times Rec\times M_{total}\\
 \end{equation}

Then, based on the Pareto principle, we should expect that if the
machine learning mechanism will be able to point out the ``high risk''
20\% of software modules with measured recall
(\textbackslash{}(Rec\textbackslash{})), the number of defects which can
be avoided by allocation of the best quality assurance efforts on the
first (development) project's phase, shall be:

\begin{equation}
H_{1}^{\prime}{}=0.8\times Rec\times H_{total}\\
 \end{equation}

Number of defects to be fixed in second and third phase of the project:

\begin{equation}
H_{2+3}^{\prime}{}=H_{total}-H_{1}^{\prime}{}\\
 \end{equation}

\subsection*{Return on Investment}\label{return-on-investment}

To investigate if usage of DePress framework will pay off, we shall use
the Return on Investment (ROI) factor:

\begin{equation}
ROI=\frac{Benefit-Investment}{Investment}\\
 \end{equation}

If the investment will not pay off, the ROI factor is negative,
otherwise positive. In our evaluation of defect prediction
cost-effectiveness we will focus on potential benefits that method will
generate:

\begin{equation}
Benefit=C_{total}-C_{total}^{\prime}{}\\
 \end{equation}

Where
\textbackslash{}(C\_\{total\}\^{}\{\textbackslash{}prime\}\{\}\textbackslash{})
is the simulated total quality assurance cost in the project with defect
prediction applied, and \textbackslash{}(C\_\{total\}\textbackslash{})
is the actual QA cost in the project, without the defect prediction
used.

\textbackslash{}(Investment\textbackslash{}) is defined as the total
cost of defect prediction introduction. Moreover,
\textbackslash{}(NetReturn\textbackslash{}) is calculated as
\textbackslash{}(Benefit\textbackslash{}) reduced by
\textbackslash{}(Investment\textbackslash{}):

\begin{equation}
NetReturn=C_{total}-(C_{total}^{\prime}{}+Investment)=Benefit-Investment\\
 \end{equation}

\subsection*{Benefit Cost Ratio}\label{benefit-cost-ratio}

To analyze potential benefits from the usage of defect prediction, we
will use the Benefit Cost Ratio (BCR) factor:

\begin{equation}
BCR=\frac{Benefit}{Investment}\\
 \end{equation}

Values larger than 1 for the BCR mean a monetary gain from DePress based
defect prediction usage, values smaller than 1 -- a loss.

\section*{Quality Assurance Costs: Actual and
Simulated}\label{quality-assurance-costs-actual-and-simulated}

\subsection*{Actual Project's Quality Assurance
Costs}\label{actual-projects-quality-assurance-costs}

Volvo Group's policy did not allow us to publish the real costs of work
invested in the project. For the purpose of research, we agreed that the
man-hour cost of work by a software developer
\textbackslash{}(C\_\{d\}\textbackslash{}) will be marked as:

\begin{equation}
C_{d}=x\\
 \end{equation}

In that case, the average man-hour cost of work by a software tester
\textbackslash{}(C\_\{t\}\textbackslash{}) shall be, calculated
according to current labor market data
rates~\protect\hyperlink{wynagrodzenia}{(wynagrodzenia.pl 2015)}:

\begin{equation}
C_{t}=0.85x\\
 \end{equation}

That means, that when the tester and code developer are working together
on bug fixing during the later stages of the project (not in the coding
phase), the average cost per man-hour should be:

\begin{equation}
C_{d+t}=\frac{C_{d}+C_{t}}{2}=0.925x\\
 \end{equation}

Other costs, such as infrastructure and hardware, will remain constant
for the real-life and alternative (prediction-based) scenario, so they
will be omitted.

Average defect fixing costs

Phase

1.Development

2.Testing

3.Post-release

Team members involved

Developer

Developer Tester

Developer Tester

Average fixing time per one defect {[}hours{]},
\textbackslash{}(T\textbackslash{})

1

3

3

Assumed cost of man-hour \textbackslash{}(C\_\{hour\}\textbackslash{})

\textbackslash{}(C\_\{d\}\textbackslash{})

\textbackslash{}(C\_\{d+t\}\textbackslash{})

\textbackslash{}(C\_\{d+t\}\textbackslash{})

Cost per one defect
\textbackslash{}(C\_\{defect\}=T\textbackslash{}times
C\_\{hour\}\textbackslash{})

x

2.775x

2.775x

Time spent on project work was traced by every team member using the
JIRA tool. As a result of analysis of that data, we could obtain an
average of the total time spent on fixing a single defect for each phase
of the project (Table~\ref{tab:costs}). The amount of
time spent on quality assurance, together with the number of hours spent
and number of defects fixed, divided by phases, are shown in
Table~\ref{tab:scenar1}.

Actual resources consumed on defect fixing

Phase

1.Development

2.Testing

3.Post-release

Number of defects discovered
\textbackslash{}(H\_\{phase\}\textbackslash{})

190

383

264

QA cost per one defect \textbackslash{}(C\_\{defect\}\textbackslash{})

x

2.775x

2.775x

QA cost per phase
\textbackslash{}(C\_\{phase\}=H\_\{phase\}\textbackslash{}times
C\_\{defect\}\textbackslash{})

190x

1063x

733x

According to the data in Table~\ref{tab:scenar1}, the
total number of defects discovered in release 4.0.0 are:

\begin{equation}
H_{total}=\sum H_{phase}=190+383+264=837\\
 \end{equation}

Accordingly, the total quality assurance cost is:

\begin{equation}
C_{total}=\sum C_{phase}=190x+1063x+733x=1985x\\
 \end{equation}

The ratio between defects fixed in testing and those fixed during the
post-release stages is:

\begin{equation}
\frac{H_{2}}{H_{3}}=\frac{383}{264}\approx\frac{3}{2}\\
 \end{equation}

\subsection*{Model Construction and
Prediction}\label{model-construction-and-prediction}

\par\null

To create a prediction model that will be able to classify new source
code, a machine learning process must be performed using training data
already categorized respectively, and taken from an earlier version of
the program, in which the area of ``high risk'' can be easily identified
by investigating which code areas were changed most because of bug
fixes. In the case of the Texas Project we used the previously described
method, that all of the source code changes caused by fixing defects,
were labeled with unique defect identifiers, which were recorded in the
software responsible for managing bugs in the project. Both version
control system, and defect tracker application used in the project make
it possible to obtain the desired data by exporting it to a file. The
DePress framework consists of appropriate modules allowing for
conversion of the data from both those programs to a universal internal
format. Log entries from the repository, take into account all changes
in the source code. All recorded changes resulting from bug fixes, are
recognized by the module Marker Parser. These entries are then compared
with data from the defect management system by the module Issues per
Artefact (Figure~\ref{fig:issues_per_artefact}). The
effect of this module's usage are statistics describing the frequency of
code changes resulting from bug fixes for each of the code modules. Due
to this comparison it is easy to recognize the most defective areas
which are known to be ``high risk''. In the case investigated (release
4.0.0), as an area of ``high risk'', we pointed out all the modules that
have to be modified to be able to fix more than 2 registered defects.
These modules accounted for 22.4\% of all modules and were responsible
for 80.36\% of all registered errors.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/issues-per-artefact-bw/issues-per-artefact-bw}
\caption{{Workflow for combining defect and code change information%
}}
\end{center}
\end{figure}

Code metrics collected for each code module were classified in
accordance with the applied approach -- ``high risk'' modules were
marked as ``1'' and those not belong to this group as ``0''. Ratio of
modules labeled as ``1'' to those labeled as ``0'' was 1:15,52, which
indicates a class imbalance problem.

Categorized data was divided into two equal sets by stratified sampling.
One of the sets was stored for validation of created prediction models,
the second was used for preparation of samples based on three selected
classifiers -- Naive Bayes, Decision Tree and Probabilistic Neural
Network.

For each classifier, four different experimental setup preparations were
possible, thanks to the module-based architecture of the DePress tool:

\begin{itemize}
\tightlist
\item
  Without feature selection, with class imbalanced dataset,
\item
  Without feature selection, with class balanced dataset,
\item
  With feature selection, with class imbalanced dataset,
\item
  With feature selection, with class balanced dataset.
\end{itemize}

When needed, feature selection was carried out by KNIME's functionality
presented in
section~\ref{subsec:predictorvariablesselection} and
class balance was achieved by using mechanism presented in
section~\ref{subsec:objectivevariable}.

Prediction results: Recall and F-measure (in brackets) values for all
experimental set-ups

Classifier

Without Feature Selection

With Feature Selection

Class Imbalance

Class Balance

Class Imbalance

Class Balance

Naive Bayes

0.409 (0.237)

0.783 (0.621)

0.318 (0.412)

0.783 (0.766)

Decision Tree

0.273(0.279)

0.682(0.667)

0.227(0.357)

0.652(0.682)

Probabilistic Neural Network

0.091(0.167)

0.783(0.72)

0.136(0.24)

0.783(0.74)

\subsection*{Prediction Recall}\label{prediction-recall}

Using the approach described in the previous section, defect prediction
was performed and its recall measured
(Table~\ref{tab:pred_results}) for all four
experimental set-ups, classifiers and samples. The best prediction
results (highest recall values) were obtained for the imbalanced class
sample, but with the feature selection step. At this point, we were able
to answer \textbf{RQ1:} \emph{What is the highest level of prediction
recall achievable by DePress tool in basic configuration, using an
industrial project's data?} From all results, the highest recall for
highest F-measure values in our experiment
(\textbackslash{}(Rec\textbackslash{})) got prediction based on the
Naive Bayes algorithm:

\begin{equation}
Rec=0.783\\
 \end{equation}

\subsection*{Prediction-based costs
simulation}\label{prediction-based-costs-simulation}

For the purpose of cost simulation in this scenario, where defect
prediction is introduced to the project using DePress framework, we
agreed that:

\begin{itemize}
\tightlist
\item
  Total number of discoverable defects in release
  4.0.0~(\ref{htotal}) is a constant value;
\item
  Average fixing cost per one defect
  (Table~\ref{tab:costs}) is also true for the
  considered scenario;
\item
  Information on location of ``high risk'' software modules, with recall
  \textbackslash{}(Rec\textbackslash{}), will be available in the first
  phase of the project;
\item
  Ratio~(\ref{hrest}) is preserved.
\end{itemize}

Considering the highest recall value achieved for release 4.0.0 as a
result of the prediction models
creation~(\ref{highRec}) and the total number of
discovered defects in that release~(\ref{htotal}),
based on the proposed strategy~(\ref{strategy}) we
should expect, that the number of software issues which can be solved by
allocation of the best quality assurance practices in the first,
development phase of the project is:

\begin{equation}
H_{1}^{\prime}{}=0.8\times 0.783\times 837=524\\
 \end{equation}

Regarding the number of defects which are expected to be found in later
phases of the project~(\ref{hrest}):

\begin{equation}
H_{2+3}^{\prime}{}=837-524=313\\
 \end{equation}

As we assumed that ratio~(\ref{hrest}) is preserved,
the number of defects which are expected to be found in the project's
second and third phase (connected) are:

\begin{equation}
H_{2}^{\prime}{}=313\times 0.6=188\\
 \end{equation} \begin{equation}
H_{3}^{\prime}{}=313\times 0.4=125\\
 \end{equation}

Considering the above values, we simulated quality assurance costs in
the assumption that the machine learning mechanism will be able to point
out the ``high risk'' 20\% of software modules with measured
recall~(\ref{highRec}), and the best quality assurance
efforts will be allocated to the development phase to avoid the
calculated number of defects~(\ref{h1prim}). Results of
that simulation are presented in
Table~\ref{tab:scenar2}.

Simulated QA costs, with defect prediction used

Phase

1.Development

2.Testing

3.Post-release

Number of defects fixed
\textbackslash{}(H\_\{phase\}\^{}\{\textbackslash{}prime\}\{\}\textbackslash{})

525

188

125

QA cost per one defect \textbackslash{}(C\_\{defect\}\textbackslash{})

x

2.775x

2.775x

QA cost per phase
\textbackslash{}(C\_\{phase\}\^{}\{\textbackslash{}prime\}\{\}=H\_\{phase\}\^{}\{\textbackslash{}prime\}\{\}\textbackslash{}times
C\_\{defect\}\textbackslash{})

524x

522x

347x

Total quality assurance cost in this scenario will be:

\begin{equation}
C_{total}^{\prime}{}=\sum C_{phase}^{\prime}{}=524x+522x+347x=1393x\\
 \end{equation}

\subsection*{Cost of investment}\label{cost-of-investment}

Costs of defect prediction introduction were calculated as the sum of
such elementary costs:

\begin{itemize}
\tightlist
\item
  Tool acquisition and installation costs,
\item
  Training time costs,
\item
  Data collection cost,
\item
  Defect prediction preparation cost.
\end{itemize}

Tool acquisition and installation costs of the case investigated shall
be considered as zero costs. In Volvo's organization, DePress is freely
available via an internal application installation system called FAROS.
DePress can be ordered and installed on user's computers without any
additional costs for the project.

Training time costs -- support of DePress and defect prediction is
handled by their internal Development \& Runtime Support organization,
which is already pre-paid by project, whether such a tool and technology
will be used or not. Due to that fact, any training cost is the cost per
man-hour for each developer, which will be responsible for the analysis.
After measuring time spent on training a single person, we can state
that: a developer needs to spend a maximum of 4 hours on training, 1-2
hours of general introduction plus another 1-2 hours of training in
prediction preparation.

Data collection cost is mostly the man-hour cost of exporting the proper
data from data sources and code metrics generation for two selected
releases. Data export, if used (direct-connection DePress nodes also can
be used, which will collect the required data in real-time), will not
take more that one man-hour and in most cases that activity should take
no more than a few minutes. For code metrics collection purposes using
Eclipse Metrics plugin, each of two releases need to be build locally,
which include -- downloading release from the remote repository, local
build and XML metrics files collection. After measuring time spent on
that activity, we found that it took no more than 2 man-hours.

Defect prediction preparation cost is the man-hour cost of defect
prediction workflow preparation using the DePress tool. As generic
workflows for most technologies used in the organization are
pre-installed with the DePress tool, only correction actions are
expected, if any. Results of time measurement say that preparation of a
proper DePress workflow should not take more than one man-hour.

Summary of investment costs is presented in
Table~\ref{tab:costs-table}.

Defect Prediction Investment Costs

Activity

Time required {[}hours{]}

Cost {[}man-hours{]}

DePress tool acquiring and installation costs

0

0

Training time costs

4

4x

Data collection cost

3

3x

Defect prediction preparation cost

1

x

TOTAL (Investment)

8

8x

\section*{Results}\label{results}

Here, with respect to research questions \textbf{RQ2} and \textbf{RQ3}
we summarize the results of our simulation. Research question
\textbf{RQ1} was answered in
section~\ref{model_construction}.

\textbf{RQ2:} \emph{Will usage of the DePress framework in industrial
project pay off?}

Simulation shows, that we should expect
Benefit~(\ref{benefit}) from DePress usage in the
project:

\begin{equation}
Benefit=1985x-1393x=592x\\
 \end{equation}

Accordingly, expected Return on Investment~(\ref{roi}):

\begin{equation}
ROI=\frac{592x-8x}{8x}=73\\
 \end{equation}

As ROI is positive, we can state that investment will pay off.

\textbf{RQ3:} \emph{How cost effective is the DePress framework for
defect prediction in an industrial software development project with the
highest possible prediction recall achievable for that tool?}

As shown in Table~\ref{tab:costs-table}, the expected
total investment cost of DePress tool-based defect prediction
application in software development project is:

\begin{equation}
Investment=8x\\
 \end{equation}

Benefit Cost Ratio~(\ref{bcr}) calculated
using~(\ref{benefit})
and~(\ref{investment}) values:

\begin{equation}
BCR=\frac{592x}{8x}=74\\
 \end{equation}

Such BCR value shows that we should expect a high monetary gain from
DePress tool usage for supporting quality assurance with defect
prediction. Moreover, NetReturn~(\ref{netreturn}) from
the simulated defect prediction application is:

\begin{equation}
NetReturn=592x-8x=584x\\
 \end{equation}

Answering research question \textbf{RQ3}, when the defect prediction
application strategy proposed in
section~\ref{strategy_par} is applied, such an
approach can result in reduction of final QA costs by almost 40\%, when
recall of the prediction model will be 0,952, and after fixing 76\% of
the detectable bugs~(\ref{h1prim}) while still in the
first, developmental phase of the project. Graphical comparison of
quality assurance costs -- actual and simulated, with defect prediction
introduced, is shown in
Figure~\ref{fig:cost_comparison}.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/continuous-simulation/continuous-simulation}
\caption{{Costs of quality assurance in considered project: actual compared to
simulated, when DePress-based defect prediction is used%
}}
\end{center}
\end{figure}

\section*{Threats to Validity}\label{threats-to-validity}

In this paper, defects are not distinguished according to their severity
(minor, major, etc.) and use an average, fixed time for each single
defect. Omitting the severity measure in defect prediction studies is a
frequent practice~\cite{Menzies2010}, however it can be
important when simulated QA cost calculation will be compared to
real-life values. In our simulation we assumed equal severity for each
defect, which is reflected in an equal, average cost
(Table~\ref{tab:costs}). However, when we apply the
proposed effort allocation strategy into a real-life environment, we can
deal with the situation, of when defects left undetected until the later
phases of testing and after-deployment will be characterized by higher
severities than defects resolved while within the coding phase; such a
situation would negatively impact overall quality assurance costs, when
DePress tool would be used for defect prediction purposes, in comparison
to simulated values.

In our simulation, quality assurance cost is calculated based on the
number of defects expected to be fixed by QA efforts at each phase,
multiplied by the average cost to fix a single defect, based on actual
project data. Then, simulated costs are compared to actual costs. Rahman
et.al.~\cite{Rahman2014} argue, that a more efficient
way of comparing quality assurance efforts, when the defect prediction
models are involved, is by a comparison of AUCEC (Area Under Cost
Efficiency Curve) values~\cite{Arisholm2010}. The
approach followed in our paper was motivated by the fact that the
comparison of cost values (actual and simulated) is considered to be
more readable by business stakeholders.

In this paper we do not consider any costs not traced by the JIRA
system. There was no technical possibility to obtain cost information on
quality assurance actions taken outside the project's team during the
post-release phase, an example being the service desk team responsible
for contact with an end user, who found a new defect in the
after-deployment phase.

For the purpose of simulation, we used the same initial conditions as in
an actual project. There were a like number of detectable defects as
well as a similar ratio between defects fixed in testing and those fixed
in the post-release stages~(\ref{hrest}). In the actual
defect prediction using the proposed effort allocation strategy, we
should expect that the aforementioned conditions would be different, for
example if the actual application will consider different software
release and/or development project.

\section*{Conclusions}\label{conclusions}

Cost effectiveness within the simulated scenario is strictly related to
the quality of prediction when measured with recall
(\textbackslash{}(Rec\textbackslash{})). Based on the simulation
presented, it is possible to calculate the
\textbackslash{}(NetReturn\textbackslash{}) of using a proposed quality
assurance effort allocation strategy for a series of defect prediction
recall values (solid line on
Figure~\ref{fig:qa_vs_recall}). In such a way, we can
observe how \textbackslash{}(NetReturn\textbackslash{}) depends on
\textbackslash{}(Rec\textbackslash{}) in terms of the proposed QA effort
allocation strategy. However, any
\textbackslash{}(NetReturn\textbackslash{}) coefficient values for
\textbackslash{}(Rec\textbackslash{}) lower than 0,5 should not be
considered, as below that point, the efficiency of defect prediction is
similar to that of random guessing.

One can argue, that it is improper for one to assume, that it is
possible for programmers to predict and fix 78\% of all detectable
defects while still in the developmental phase, only due to the fact
that code parts actually responsible for these defects are known the
developers, thanks to defect prediction. Please notice, that in our
simulation we considered only bugs actually fixed by programmers. The
only difference is that in the actual project (not in simulation), more
defects were found by testers and end-users during the second and third
project phases, which caused higher average bug fixing costs in these
phases. Nevertheless, the defects were finally fixed by the developers.
Moreover, we could go a step further and modify our effort allocation
strategy to use DePress for prediction model creation, aimed at finding
code responsible for 100\% of all detectable defects. In such a case,
the relation between the expected number of defects fixed in the
developmental stage, and prediction recall would be:

\begin{equation}
H_{1}^{\prime}{}=Rec\times H_{total}\\
 \end{equation}

Theoretically, in such a case we can expect that the
\textbackslash{}(NetReturn\textbackslash{}) value should be even higher
than calculated based on the proposed strategy (dotted line on
Figure~\ref{fig:qa_vs_recall}). However, as the
proposed effort allocation strategy is based on the Pareto principle
(only 20\% of code is responsible for 80\% of defects), we should
consequently expect that to find and remove the remaining 20\% of
defects, an additional 80\% of code would need to be inspected,
respectively. In such a scenario, by covering even 100\% of code by all
necessary precautions to remove every detectable defect may cost
significantly more than the actual, real quality assurance in the
considered project release (4.0.0). We expect that the remaining
approximately 20\% of detectable defects can be effectively found and
removed during further phases of the project, such as in testing phase,
with better cost effectiveness, than with comprehensive quality
assurance works during coding phase, focused on finding and eliminating
100\% of detectable defects.\selectlanguage{english}
\begin{figure}[h!]
\begin{center}
\includegraphics[width=0.98\columnwidth]{figures/netreturn-coefficient/netreturn-coefficient}
\caption{{NetReturn coefficient depending on recall value%
}}
\end{center}
\end{figure}

In industrial software development projects, quality assurance (defect
fixing) consumes a significant amount of time and resources. By using
the defect prediction technique, project members can obtain information
on possible defect-prone elements of the software, before defects will
occur, to optimally plan their quality assurance process. What is
proposed in this paper, is a simple effort allocation strategy which is
based on DePress framework-driven defect prediction, Pareto principle
and Boehm Law, and which eliminates most of the quality assurance work
during after-development project's phases. Such an approach
significantly increase quality assurance costs in development phase,
however overall, QA costs will decrease
(Figure~\ref{fig:cost_comparison}) in comparison to
actual, real-life costs observed in the investigated project, as
significantly less discoverable defects are left to be fixed in later
phases, where, according to Boehm's Law, bug-fixing costs are
considerably higher. At the same time, we need to mention the low
investment costs, when a Knime-based DePress framework will be used for
defect prediction purposes. Low investment costs and high recall of even
simple defect prediction performed by DePress, can result with high
\textbackslash{}(NetReturn\textbackslash{}) of DePress-aided quality
assurance planned basing on proposed effort allocation strategy.

\textbackslash{}printbibliography

\selectlanguage{english}
\FloatBarrier
\end{document}

