Text
A SPICE Based Process Assessment Model for Public
Financial and Physical Resource Management Process
Ebru
Gökalp
[1], Onur Demirörs
Informatics Institute, Middle East
Technical University, Ankara, Turkey
The necessity of transformation for efficient and effective management of Public Financial and Physical
Resource Management (PFPRM) process
has become increasingly critical in the governmental organizations. However, as a result of literature review, it is observed that there is a lack of a guideline for process
capability determination and improvement of PFPRM process. ISO/IEC 15504,
also termed Software Process
Improvement and Capability dEtermination (SPICE), is used as a baseline to
generate capability/maturity models for different specific domains/sectors, based
on the observed benefits obtained in software organizations. We have utilized
the same approach for the government domain, and process definition of PFPRM
based on the requirements of SPICE is developed. As a result, PFPRM
process can be assessed based on SPICE to be consistently applied, managed, and
controlled across governmental agencies. A multiple case study, including the assessments of three organization’s
PFPRM process capability level is performed. The assessment results are used to
develop road-maps for implementing PFPRM process improvement in the study. The findings show that the proposed approach is
applicable for identifying the current state of the process capability and the
gaps with the assessed capability level of the PFPRM process performed in
governmental organizations.
Keywords: SPICE, Government, Financial Management,
Physical Resource Management, Process Capability Determination, ISO/IEC 15504
1. Introduction
The governments are under
pressure to improve the service performance with limited budget. A key
government priority is to considerably reduce costs and achieve better value
for money through reforming public financial and physical resources management
(PFPRM) process. While attempting to adopt and comprehend effective,
contributory, and transparent administration measures, it is confronted with
the challenge of transformation and the need to reengineer governmental
processes and systems [1]. In the wake of the financial
problems faced by the government during the 2000s, the necessity of
transformation for efficient and effective management of PFPRM process has
become increasingly critical in the governmental organizations.
As the government organizations continue to focus on
core competencies and outsource non-core, yet critical functions, these
organizations are depending on PFPRM process as a key enabler to achieve and to
maintain them. Today's government organizations must now manage an increasing number
of contractors and suppliers who are carrying out mission critical functions
for their organizations. Thus, government sector organizational core
competences now include the PFPRM process. As stated in [2], government sector
organizations should be concentrating their attention on measurement of the process
performance of the PFPRM in order to improve its core competencies.
The capability/maturity models are used to assess the
level of capability of the most critical processes of many organizations. There are well-accepted Process
Capability/Maturity Models (PCMMs), such as ISO/IEC 15504- also termed Software
Process Improvement and Capability dEtermination (SPICE) [3–6],CMMI-DEV (Capability Maturity
Model Integration for Development) model [7]. The ISO/IEC 15504 standard has entered a
revision cycle to be transformed from a single, multi-part standard to a set of
related standards using a defined numeric range, the ISO/IEC 330xx series [8]. This transformation is intended
to result in an expanding range of application, and to importantly more open
standards framework. The revised requirements for the standard are aimed to
provide the extension of process assessments to fields outside the ICT domain [9].These PCMMs are used as a
basis for evaluation and comparison for process assessment and/or improvement,
assuming that higher level of process capability or organizational maturity is
associated with better performance. These models are developed for the purpose
of performing assessments of software and systems processes. As a result of the
observed benefits of these PCMMs such as expense savings, increased involvement
of employees, predictable and improved quality and productivity, and
constructing a consistency of process capture and use; customizing them to
different domains other than software development is the subject of growing
interest in the literature. Accordingly, many initiatives based on SPICE are
proposed for various domains: Automotive SPICE [10], Medi SPICE [11], Enterprise SPICE [12], and IT Service Management [13].
We have utilized the same approach for the government
domain by developing the Government Process Capability Determination Model
(Gov-PCDM) [14–19] based on SPICE [3–6]. The major reason for
selecting SPICE as a basis is its well-defined and commonly accepted structure.
The aim of the Gov-PCDM is to provide the base for improving the processes
performed in public organizations. It pursues a standardized and structured
approach for governmental process assessment for performing quality improvement
initiatives in a consistent, repeatable manner, assessed by competent metrics
with a roadmap on what to do to improve quality. In the scope of this study, we
analyze one of the processes defined in Gov-PCDM, the PFPRM process.
The capability levels defined in the SPICE measurement
framework from level zero to level five have been developed to be appropriate
universally to all processes except for level-1 (performed) where the
observable indicators are different for each process, while all the process
attributes from level two to five are common for all processes. In the scope of
this study, the process definition of PFPRM process is developed, including
level-1 process performance indicators as outcomes, base practices and work
products. Thus, the PFPRM process capability level can be assessed based on
SPICE owing to this developed process definition.
In this paper the method for how to develop the
process definition of PFPRM and how to conduct process capability level
assessment is described in detail. Moreover, multiple case studies conducted in
order to check the usefulness and adequacy of the proposed approach, is discussed.
As part of the case study, the capability levels of the PFPRM process performed
in three governmental institutions are assessed, roadmaps for improving the
process to the next capability levels are derived and the results of
each assessment are presented. Over the presentation, the results are discussed
with practitioners in meetings. Then, follow-up interviews are conducted after the meeting to
ask
if the results correctly represent the state of the process. Finally,
the results are analyzed in the study.
This paper is divided into five sections. In the
second section, a brief literature review about PFPRM process improvement is
given. The method for the development of PFPRM process definition and capability
assessment is provided in the third section. In the fourth section, the design
and implementation of the multiple case studies conducted in three governmental
organizations are presented and the analyzed results as well as the derived
road-maps to improve the assessed processes are given. The conclusions and future studies are given in
the last section.
2. Literature
Review
The
literature reflects an increasing flow in the research stream of public sector performance assessment and benchmarking as stated in [20]. Hong et
al. [21] also identified a growing
need for assessment and benchmarking studies of complex business practices and proliferation
of research studies in the area of public sector processes. Fettke et al. [22] demonstrated the use of a
business process maturity model to improve service response efficiency and
effectiveness in public administration.
The
use of maturity models for PFPRM process, such as contracting and procurement
sub-processes, has seen some, but limited, application in the public sector.
Raymond [23] stated that the necessity for
PFPRM process best practices is increasing, especially in the areas of value
for money, ethics, competition, transparency, and accountability. Rendon [24] developed Contract Management
Maturity Model to assess and measure a US Air Force contracting agency’s
procurement process and identified process improvement initiatives. Møller et
al. [25] developed and applied a public
procurement maturity model for the Denmark government. The use of their model
will improve standardization, consistency, and transparency of Danish public
procurement organization practices. Waterman and Knight [26] explored using a capability
maturity model for conducting self-assessments in a case study on U.K.
government procurement departments. Concha et al. [27] introduced the e-Government
Procurement Observatory Maturity Model (eGPO-MM) to measure government
e-procurement portals status across the Latin American region and to enable
development of an improvement road map for eGP in each participating country.
We have also performed research studies related to
customization of SPICE for the government domain: A literature review that
results in the necessity of a structured process improvement model regarding
the specialties of the government domain [17], an exploratory case study to check if
the customization of SPICE for government domain is applicable [14], a study of proposing a SPICE
based process improvement method for government domain [15], development
of a generic process definition for governmental agency-specific processes and
conducting case studies to investigate the appropriateness [18,19], and a study of development of
a process definition for one of the seven management and support processes [16]. The initial findings of our
studies show the applicability and usefulness of the approach of SPICE-based
government process capability determination model of Gov-PCDM.
The
literature review can be summarized by concluding that there is a growing
research stream in the benefits and challenges of measuring PFPRM process. Moreover,
the literature review identified a research gap in that there is limited
research in the use of maturity models for assessing PFPRM process. Of the
research identified in the review, none of the maturity models identified is based
on key PFPRM process base practices, but on other procurement and contracting
functions. Additionally, they are not based on one of the well-accepted PCMMs.
Thus, they do not provide assessments of PAs applicable universally to all processes, such as process control, process measurement,
etc. Lastly, none of
them is implemented in the Turkish governmental organizations. The research in
this study aims to fill this gap by providing a SPICE-based process
capability determination model of Gov-PCDM [14–19] for PFPRM process capability determination
to perform the process improvement in a structured and standardized way. The application of the proposed approach in case study
settings is performed in Turkish governmental organizations.
2
3. Method
The
method followed to develop the process definition of the PFPRM process and to
conduct the process capability level assessment is defined in this section.
3.1 Development
of the Process Definition of PFPRM Process
The process
definition, based
on the requirements defined in ISO/IEC 15504-2 [3] is characterized by process purpose (the objective of
accomplishing the process), outcomes (the observable consequences expected from successfully
performing the process), base practices (a list of actions that may be used
for the achievements of the outcomes), and work products (individually recognizable bodies
of information created and stored for human use).
The
development of the process definition of PFPRM is illustrated in Figure 1. As a
first step, documents related to business rules and policies of the government
for the PFPRM, as well as existing quality improvement models and standards are
investigated. The process definition of PFPRM is developed by harmonizing
existing quality improvement models and standards as FEAF (Federal Enterprise
Architecture Framework) [28], APQC (American Process
Qualification Center) [29], and SPICE-ISO/IEC 15504- Part 2 [3], based on the business rules and
policies of the PFPRM stated in laws, decree-laws, guidelines and regulations,
as Public Procurement Law, Public Financial Management and Control Law, and Budget
Preparation Guideline. The process definition is defined based on the requirements stated in SPICE-ISO/IEC
15504- Part 2 [3]. Process purpose is defined based
on Business Reference Model (BRM) defined in FEAF [28] and business rules and policies
stated in governmental documents. Outcomes and work products are defined based
on the business rules and policies. Base practices and sub-base practices are adapted
from APQC [29]. The draft version of PFPRM
definition is developed by one of the authors.
The process
definition draft is formally reviewed by five PFPRM process owners who are
working in the finance department in government agencies. They are requested to provide verbal and
written feedback on the following questions:
(1)
Are the major elements of the process definition of PFPRM; such as, outcomes,
base practices, and work products well defined and articulated?
(2)
Is there any missing information in the process definition of PFPRM?
The
feedbacks are used to refine and revise the model. The revised version of the
process definition of PFPRM is reviewed and approved by the management with
executive responsibility within two organizational units by two people and one
of the authors who has both professional and academic experience in using.
Consequently, the final version of the process definition of PFPRM which is
given in Table 1 is obtained.
Fig. 1. The Development of Process
Definition of PFPRM
The
process definition is needed to perform PFPRM process capability level
assessment based on SPICE. It is used in level-1 assessment to check if the
process is performed. The method followed for the capability level assessment
is defined in the following sub-section.
Table 1. The Process Definition of PFPRM Process
Purpose
| The
purpose of public financial and physical resource management is to deploy
and use the government’s resources, facilities and assets.
|
Outcomes
| As a result of
successful implementation of the financial and physical resource management
process;
1. Financial and
physical resource strategy and policies are established.
2. The detailed
financial plan (budget) containing cost estimates for consumed resources
and, where applicable, revenue projections for fees* received is generated.
3. Procurements of
goods, services or works are performed based on public procurement law.
4. Finance and
accounting transactions are handled for procured goods/services or works and
receivable where applicable.
5. Physical
resources are acquired, constructed and disposed.
6. Warehouse used
for storing tangible physical resources is managed.
7. Reports
including internal and external financial information are generated.
8. Internal and
external audits are conducted.
* Many government services issue
licenses and permits and collect an associated fee.
|
Base
Practices
| MGRSP1.BP1:
Establish and maintain a strategy and policies for financial and physical
resource management: Establish and
maintain a strategy and policies for financial and physical resource
management. [Outcome: 1]
1.1.1. Build a strategic plan to support business
objectives
1.1.2. Design capital structure
MGRSP1.BP2:
Perform budgeting: Prepare
periodic detailed budgets and plans and financial forecasts, according to
established strategy and policies.[Outcome:1, 2]
1.2.1 Develop annual budget proposal
1.2.2 Get approval for the budget from
ministry of finance
1.2.3 Develop periodic detailed
financial plan/budgets and forecasts based on the approved budget
1.2.4 Allocate resources
1.2.5 Manage financial risk
1.2.6. Manage fee administration, where
applicable
|
Table 1. The Process Definition of PFPRM Process (Continued)
Base Practices
Base
Practices
| MGRSP1.BP3: Procure goods/services or works: Purchase goods/services or works based on
the public procurement law [Outcome:2,3]
1.3.1 Elicitate need and requirements.
1.3.2 Prepare technical contract.
1.3.3 Conduct market research to calculate the
approximate cost.
1.3.4 Determine tender procedure.
1.3.5 Prepare documents related to tender including
proposal evaluation criteria.
1.3.6 Obtain approval for the tender.
1.3.7 Define tender committee.
1.3.8 Publish invitation for bidding.
1.3.9 Review
tender documents.
1.3.10 Receive tender proposals.
1.3.11 Apply evaluation criteria to select a provider,
negotiate contract terms and conditions to resolve open items and select the
contractor.
1.3.12 Invite the selected contractor to sign the
contract.
1.3.13 Monitor
contractor performance.
1.3.14 Close the
contract after ensuring that each party’s performance meets contractual
requirements.
MGRSP1.BP4: Process finance and accounting
transactions: Process all the transactions
related to purchasing products/services, paying, and receiving. [Outcome: 3,4]
1.4.1
Process payable accounts
1.4.2
Process receivable, credit, and collections accounts
MGRSP1.BP5: Manage physical
resources: Establish requirements and standards for physical items which are
acquired, constructed and disposed. [Outcome: 5, 6]
1.5.1 Acquire and deploy assets
1.5.2 Manage facilities
1.5.3 Manage physical risk
1.5.4 Dispose nonproductive physical
assets
MGRSP1.BP6: Operate Warehousing Collect, receive, and store assets,
according to established strategy and procedures. [Outcome: 6, 7]
1.6.1 Track inventory deployment
1.6.2 Receive, inspect, and store
deliveries
1.6.3 Track product availability
1.6.4 Record checkouts of store
1.6.5 Track inventory accuracy
1.6.6 Track third-party logistics,
storage and shipping performance
1.6.7 Manage physical finished goods
inventory
|
Table 1. The Process Definition of PFPRM Process (Continued)
Base Practices
| MGRSP1.BP7: Report information: Report transactions to accounting department (internal) and court of
accounts (external). [Outcome: 7]
1.7.1 Provide external financial information
1.7.2 Provide internal financial information
MGRSP1.BP8: Conduct internal and
external audits:
Determine compliance of performed process with the requirements, plans, laws
and procedures, as appropriate. [Outcome: 8]
1.8.1 Develop and implement audit strategy
1.8.2 Plan audit
1.8.3 Perform Auditing
1.8.4 Identify corrective actions from the audit report
1.8.5 Track actions for audit report
|
Work Products
|
Inputs
| Outputs
|
Public Financial Management and Control Law [Outcomes:
1,8]
|
|
Budget Preparation Guideline [Outcomes: 2]
|
|
Public Procurement Law [Outcomes: 3]
|
|
Public procurement Contracts Law [Outcomes: 3]
|
|
Asset Legislation [Outcomes: 5,6]
|
|
Regulation on Prepayment Procedures and Principles [Outcomes:
2,3,4]
|
|
Regulation on the Principles and Procedures of Internal Control and
Preliminary Financial Control [Outcomes: 7,8]
|
|
| Budget Proposal
[outcome:2]
|
| Detailed financial
plan [outcome:2]
|
Payment Order Document [outcome:4]
|
|
| Warehouse Documents for
(asset request stock-in, stock out, inspection and acceptation) [outcome:6]
|
| Appropriation Transfer
Document [outcome:7]
|
| Audit Report [outcome
8]
|
| | |
The measurement framework is based
on SPICE. Process capability
is classified into six levels in ISO/IEC 15504-2 [3] as Level 0: Incomplete; Level 1: Performed; Level 2: Managed; Level 3:
Established; Level 4: Predictable; Level 5: Optimizing.
The
measure of capability is based upon a set of PAs. Process capability indicators
are the means of achieving the capabilities addressed by the considered PAs.
Evidence of process capability indicators supports the judgment of the degree
of achievement in the PA.
The
PA of Level-1 is Process Performance which is a measure of the extent to which
the process purpose is achieved. The developed PFPRM process definition given in Table
1 is used for the Level-1 assessment. The base practices and work products
defined in process definition are used as ‘process
performance indicators’. For the assessments of levels 2 to 5, exactly the
same PAs, ‘generic practices indicators’, ‘generic resources indicators’ and ‘generic work products indicators’ as the
exemplar PAM provided by the ISO/IEC 15504-5 [6] are used.
The capability
level of each process instance is determined by rating PAs. For example, to
determine whether a process has achieved capability level-1 or not, it is
necessary to determine the rating achieved by PA 1.1 (Process performance). A
process that fails to achieve capability level-1 is at capability level 0. Each
PA is measured by an ordinal rating that represents the extent of achievement
of the PA, as F (Fully) meaning
86% to 100% of achievement of the PAs, L (Largely)
meaning 51% to 85% of
achievement of PAs, P (Partially)
meaning 16% to 50% of
achievement of PAs, or N (Not) meaning 1% to 15% of achievement of
PAs, achieved. A process is defined to be at capability level n if all PAs below level
n are rated as F and the level n PA(s) are rated as F or L, as defined in
ISO/IEC 15504-2 [3].
The
assessment procedures for details of the activities of assessment planning,
briefing of the participants, collecting and validating as well as reporting
are based on ISO/IEC 15504-3 [4].
4. Multiple
Case Study
To
evaluate the applicability of the proposed approach in different cases, we
applied multiple case study research. The multiple case study design,
implementation and analysis of findings are given in this section.
4.1 Multiple
Case Study Design
The research
strategy of this study matches many of the qualitative research attributes [30]. The data is needed to be
collected in its natural settings; the assessor is the key instrument in
collecting the data, there are various forms of data and inductive data
analysis is needed to be conducted.
The case studies are conducted
by the protocol template proposed by Yin [31].
The
objective of the study is to investigate if the proposed approach can be utilized for the assessment
of the process capability level determination of PFPRM and for the achievement
of roadmaps for PFPRM process improvement.
The following research
question is defined in accordance with the
objective above: “How suitable it is to use the approach of SPICE based
government process capability determination model with the purpose of
identifying the current state of the PFPRM process capability and the gaps with
the assessed capability level, as well as how well it provides roadmaps for
improving the PFPRM process capability of the governmental organizations?”
The
measure used in the research is the capability level of the governmental processes.
Case Selection
Strategy is to
select organizations that have been subject to one of our previous studies
where we analyzed their processes. We needed
organizations for which we have already investigated the strengths and
weaknesses of the processes. From
this perspective, the organizations will enable us better to observe if the
approach capable of revealing these strengths and weaknesses and indicating the
capability level of the process. Another criterion for the selection of the
process is the necessity of process improvement. The last criterion for the case study selection is
having the judgement of the organization on the requirement of the studies for
the process capability level determination and generating a roadmap for process
improvement. Without such a judgement, the study might be considered as a
burden as part of the organizations’ daily studies. The success of the process capability level
assessment is highly dependent on the process participants’ contributions.
Field Procedures
and Data Collection is based on the
ISO/IEC 15504. The
assessment procedures for details of the activities of assessment planning,
briefing of the participants, collecting and validating as well as reporting
are based on ISO/IEC 15504-3: Guidance on Performing an
Assessment [4]. This
ensured assessment planning, assessment performing, data collection and
creating documents in a standard format. The process
assessment team consists of the participants in the organization responsible
for the quality assurance and authors of the study, one of the authors is a
formally certified ‘competent’ assessor.
A
visit is planned to be performed in order to make each process assessment, for
which evidence gathering techniques as inspections of the documents (the
documentation as law, decree-law, policies, or other documents using for the
process) and as conducting semi-structured interviews with the people involved
(participants) in the actual execution of the process. The assessment team will
use this information to create the assessment report for each agency.
Sources
of evidences are the
process capability level assessment interviews, follow-up interviews conducted
with process stakeholders after sharing the assessment results in a meeting,
and the information-gathering documents specifically defined for the process. The
assessment team gathers information independently from the documents to be evaluated
(so the documents as law, decree-law, policies, or other documents using for the
process are inspected) and the person involved (participant) in the actual
execution of this process. Then, the formal assessments are conducted through
semi structured group interviews with process participants, and the direct
evidences are analyzed. After the process capability level assessment
interviews, process capability level is determined, and assessment reports are
prepared. Following, assessment results are shared and discussed with process
participants in a meeting by the assessment team. The follow-up interviews are
conducted to obtain process participants’ opinion on the assessment results
after the meeting.
The measure of the process capability
level is based upon a set of PAs as defined in the sub-section 3.1 process capability
level assessment. The rating will be performed based on gathered evidences from
inspected documents and process capability level assessment interviews by the
assessment team.
4.2 Multiple
Case Study Implementation
Multiple case studies are performed in the Turkish
Republic Ministry of Development, North Cyprus Turkish Republic Ministry of
Health and, North Cyprus Turkish Republic Ministry of Labor and Social Security
in order to evaluate the adequacy of the proposed approach. Case Studies are
selected among the organizations that have been subject to one of our previous
studies where we analyzed their PFPRM processes. We have already known the strengths
and weaknesses of the PFPRM process performed in these organizations.
Organization-1: Turkish Republic Ministry of
Development plans and guides the country’s development process in a macro
approach and focuses on the coordination of policies and strategy development.
It has 38 departments, 818 employees. An information system, named E-budget,
integrated with the Turkish Republic Ministry of Finance and Public Contract
Institute is used to coordinate the financial activities. The finance department is responsible for
carrying out all works related to procurement, contracting, and physical
resources management.
Organization-2: North Cyprus Turkish Republic Ministry
of Health is in charge of regulating the health care system. 87 people are
working in the ministry.
Organization-3: North Cyprus Turkish Republic Ministry of Labor and Social Security is
responsible for labor and social security affairs. 141 employees are working in
the ministry. An information system, named E-finance, integrated with the North
Cyprus Turkish Republic Ministry of Finance is used to coordinate the financial
activities in Organization-2 and Organization-3. The finance department is responsible for
carrying out all works related to procurement, and physical resources management.
The
process capability level assessments were performed by the assessment teams
consisting of four people for each organizations; two were participants in the
organizations responsible for the quality assurance, and the authors, one of
whom is a formally certified ‘competent’ assessor. After inspections of the related
documents, the assessment team conducted the process capability level
assessment interviews with six, two, and three PFPRM process owners, within
175, 85, and 95 minutes in Organization-1, 2 and 3 respectively. Five of the PFPRM
process owners have more than five years’ work experiences. Three of them have
three years’ work experiences, three of them have less than two years’ work
experience.
Table 2 PFPRM Process Assessment Results in Organization-1
Attribute
| Evidences
| Assessment Value
|
1.1 Process
Performance
| The process clearly
achieves its purpose by maintaining financial and physical resource
management.
| Fully Achieved
|
2.1
Performance Management
| The performance is
planned and managed informally, but they are not adjusted. Performance
quality criteria are not defined and not monitored. There is no evidence of meeting, reviews and
corrective actions. The e-budget system is used to manage the interfaces.
| Largely Achieved
|
2.2
Work
Product
Management
| Work products are
defined, revisions of the work
products are stored in information systems but their appropriate review and approval
criteria are not identified. Additionally, they are not reviewed.
|
Largely Achieved
|
3.1
Process Definition
| The process is
defined in the related governmental laws, but guidance for the process is not
provided. The sequence and
interaction of activities are not documented. Effectiveness of the process is monitored with performance metrics
defined in the yearly performance plan.
| Largely Achieved
|
3.2
Process Deployment
| Governmental regulations
define specific requirements of the process, required human resources and
information are available, allocated and used. However, necessary
competencies for performing the process are not defined. Data about the
performance of the process to demonstrate its suitability and effectiveness
is not collected.
| Partially Achieved
|
The
execution of the process are nearly the same in Organization-2 and
Organization-3. Thus, the evidences for the capability level assessment are
common. Summary of the evidences are presented in Table 2 for organization-1 and
Table 3 for organizations 2 and 3. More details of the assessments are provided
in the assessment reports [32–34].
Table 3 PFPRM Process Assessment Results in Organization-2 and Organization-3
Attribute
| Evidences
| Assessment Value
|
1.1 Process
Performance
| A strategy and
policies for financial and physical resource management are established.
Budgeting and procurement are performed based on laws. However, base
practices for physical resource management and warehouse operations are
partially achieved.
| Largely Achieved
|
2.1
Performance Management
| The performance is
planned informally, roles are defined in the regulations, and resources are
made available to perform the process. The e-finance system is used to manage
the interfaces. However, performance objectives are not defined and adjusted.
Performance quality criteria are not defined and not monitored. There is no evidence of meeting, reviews and
corrective actions.
| Largely Achieved
|
2.2
Work
Product
Management
| Work products are in
a standard form, but the requirements for documentation and control of the
work products are not defined. Quality criteria, appropriate review and
approval criteria are not identified. Additionally, they are not reviewed and
adjusted.
|
Partially Achieved
|
3
The result of the PFPRM process
assessment is that the capability level of the process performed is Level 2 in
the Organization-1 and Level-1 in Organizations-2 and Organization-3 with the
rationale based on collected and validated evidence as given in Table 4.
Table 4 PFPRM Process Assessment
Results
Process
Attributes
| Org.-1
| Org.-2
| Org.-3
|
P.A.1.1 Process Performance
| F
| L
| L
|
P.A.2.1 Performance Management
| L
| L
| L
|
P.A.2.2 Work Product Management
P.A.3.1 Process Definition
| L
L
| P
-
| P
|
P.A. 3.2 Process Deployment
| P
|
|
|
….
|
|
|
|
Result
| Level-2
Managed
| Level-1
Performed
| Level-1 Performed
|
4.3 Analysis
of Findings
The capability
levels of the PFPRM processes are determined after conducting the assessments
and rating the PAs based on the gathered evidences. The roadmaps to improve the
capability level of the processes are derived from the assessment evidences
based on ISO/IEC 15504-4 [5]. The assessment results are presented with
senior managers of the organizations, process participants in a meeting. The
ratings for each PA and evidences for that rating is explained. The derived
guideline for process improvement is also shared. The derived guideline for the process
capability level improvement and the follow-up interview results are given in
this sub-section.
4.3.1 Guideline for Improvement Capability of the
Process
In order to improve the capability
level of the PFPRM process performed in Organization-1 to the next capability
level, level 3, assessment values of the PAs should be as follows; Performance
and Work Product Management attributes: Fully Achieved, Process Definition and
Deployment attributes: Largely or Fully Achieved. For PFPRM process capability
level improvement in organizations 2 and 3 to the next capability level, level
2, assessment values of the PAs should be as follows; Process Performance
attribute: Fully Achieved, Performance and Work Product Management attributes:
Largely or Fully Achieved.
The roadmap to
improve the capability level of the processes is derived from the assessment
evidences summarized in Table 2 and Table 3. The aim is to turn negative
evidences into positive suggestions of
process capability indicators supporting the judgment of the degree of
achievement of the PA. For example; for the performance management attribute;
the second indicator (Generic Practice 2.1.2) is to plan and monitor the
performance of the process to fulfill the identified objectives. Negative
evidence, observed during the assessment for this indicator is that the process
work product reviews are not planned. Thus, necessity of reviewing work products
is indicated in the guideline. Some base practices are not fully achieved in Organizations
2 and 3. In order to improve the process capability level, all of base
practices should be fully achieved. Thus, the base practices which are observed
and validated as not fully achieved are indicated in the guideline listed as
below. The first four items are common for three organizations. The process
management item is for guideline of Organization-1. The items indicating
deployment of the expense, assets, and warehouse management systems and
internal and external auditing mechanism are for guideline of Organizations 2 and
3 in addition to four common items.
Organization-1,
Organization-2, and Organization-3
1. Work-product Management
- Define
requirement of work product
- Define
quality criteria
- Define
appropriate review and approval of work products. And also, review the work
products based on this definition.
- Define
relations between work products
- Deploy
review and adjustment mechanism
2. Configuration Management
- Provide
a change control mechanism for the work product
- Manage
versions of the work products
3. Workflow Management
- Define
activities, tasks, responsible employees, authorities and resources
- Define
sequence and interaction of activities
4. Risk Management
- Define
risks related to fulfill the objective of the process
- Manage
financial risk
- Develop
problem and issue management mechanism
- Define
how to adjust the objective when needed
Organization-1
5. Process Management
- Verify
the conformance of defined process with standard process requirements
officially.
- Identify
and collect data to monitor process performance
- Monitor
and adjust the process performance indicators if necessary
Organization-2
& Organization-3
5. Deploy Expense Management System
6. Deploy Asset Management System
7. Deploy Warehouse Management System
8. Deploy Internal and External
Auditing Mechanism
4.3.2 Follow-up Interviews with Process Participants
In
order to check the usefulness and adequacy of the proposed approach, the
follow-up interviews were conducted with process participants and senior
managers of the organizations after sharing the assessment results in a
meeting. The interviews took about 10 minutes. The open-ended structured
questionnaire given in Table 4 is utilized. We used 5 points scale in the first
two questions. It includes options as: strongly agree (5 points), agree (4
points),
neutral (3 points), disagree (2 points), strongly disagree (1 points). The median of
the responses is calculated from 8, 4, and 6 people (6, 2 and 3 of them are
process owners) for Organization-1, 2, 3 respectively.
The
findings in the conducted interviews support our proposed approach. Since the
median of the responses for the first two questions are calculated as 4 in 5
points Likert scale. It is obtained from the interview results that they think
that achieving a road map to guide what to do for increasing process capability
is useful, all of the suggestions indicated in the guideline will improve the
process performance of the PFPRM process. They also confirm that process
definition of PFPRM process covers all outcomes of the process. While answering
the last question, two of the respondents pointed out some possible improvement
areas such as interoperability of involved parties; such as public procurement
agency to keep track of stages in procurements in Organization-1. However, this is out of our scope and is
primarily related to e-government initiatives. They reported the main benefits of the
assessment as realizing the need for measuring PFPRM process capability level
in its specific environment. They stated that the biggest contribution
to the improvement of the process is provided by developing risk management
system and monitoring the effectiveness and suitability of the process.
Table 4 Results of Interview with the Stakeholders for PFPRM Process
Question
| Survey Type
| Response
|
Q1) Are measuring PFPRM
process capability and obtaining guideline for improvement useful?
| 5
points Likert scale
| Median:
4
|
Q2) Do you think that applying these
suggestions will improve the PFPRM process performance?
| 5
points Likert scale
| Median:
4
|
Q3) Is there any information you want to
add in process definition of PFPRM process? Please write, if any.
| Open-end
| No.
|
Q4) Is there any missing item(s) in
guideline for improvement list? Please write, if any.
| Open-end
| Interoperability
|
As
a result of the application of case study research, some possible threats to
validity arise. Activities to overcome these threats were planned during the design
phase of the case study. The activities performed to avoid threats are
explained in this sub-section.
– Construct
validity considers if the constructs in the case
study are well-structured or subjective to the judgment. To avoid these problems, the
information is collected from the participants with different roles (process
owner, process stakeholder, and executive member) and from multiple sources,
including documentation (Laws, decree-laws, regulations), interviews and
observations of the participants.
–
As for the internal validity, application of multiple case study is remarkably
significant to overcome this threat. A logical chain of evidence was established
while performing the study and reporting the results. The evidences collected
during the case studies were given in detail in the assessment reports [32–34]. Different sources of evidences
were utilized for analysis of the results and answering the research question.
The resulting outputs were validated by the related stakeholders by conducting
the interviews.
– External
validity deals with the
concern of the generalizability of the case study results. We conducted multiple
case studies where we can apply replication logic to overcome this threat. We
initially applied the approach in Organization-1. These first application results were
reviewed, approved, and the protocol and the field procedure of the case study
were refined. Then, the replication material of the case study was applied to Organization-2
and Organization-3. It is ensured that the replication logic was applied consistently
through the cases and consistent outputs could be achieved through multiple
executions of the same or different cases.
–
Regarding reliability, many activities were performed to avoid reliability
problems and to ensure that other researchers can perform the same study
following the methodology. A case study protocol defined by Yin [31] was followed for each case study,
where the objectives, corresponding research question, plan, sources of the
evidences of the case study are identified. Additionally, the replication
material of the case study was developed. It was observed that following this
material results in similar findings and conclusions.
5. Conclusion
Domain
specific business process improvement models are more than welcomed by various
industries in recent years. Although the concept of process improvement models
is not new, the application of process improvement models to the public sector
has not been extensively studied. The process
improvement of PFPRM process is a key government priority, since government
organizations are under pressure to limit the budgets and to achieve effective,
transparent, and contributory administration measures.
In order to provide PFPRM process improvement, SPICE,
one of the well-accepted PCMMs, is taken as the baseline. Accordingly, Financial
and Physical Resources Management process in the governmental institutions is
defined based on the requirements stated in ISO/IEC 15504-2 [3]. Thus, the PFPRM process
capability level can be assessed based on SPICE owing to this developed process
definition. The method for developing the PFPRM process
definition and for conducting process assessment is described, followed by a multiple
case studies in three organizations, which is performed to check the applicability
of this approach.
The results of
the multiple case studies highlight that the proposed approach is successful at
identifying PFPRM process defects at different process capability levels and is
capable of providing a roadmap for moving the process capability level to the
next step. The interview results show that PFPRM process stakeholders think
that obtaining a roadmap for increasing process capability is useful, and all
of the suggestions indicated in the roadmap will improve the process
performance of the PFPRM process. They also report the main benefits of the
assessment as realizing the need for measuring PFPRM process capability level
in its specific environment.
Overall, the
proposed approach provides a baseline for initiating and maintaining a continuous
process improvement cycle for PFPRM in government organizations. It enables
each government institution to evaluate its PFPRM processes in detail, which in
turn helps identify the current state of its PFPRM process capability as well
as generate a feasible improvement roadmap for moving to the next process
capability level. With this, the institution is also benchmarked against its
peers which are evaluated by using the same approach.
The study contributes to the literature in
terms of extending the boundaries of SPICE by demonstrating its applicability
in the government domain. The
SPICE community supports applicability of the standard to domains other than
software. Another contribution is the observation of the feasibility of the
proposed approach through multiple case studies, whereas existing public
finance or procurement management maturity models in the literature do not
include such applications. The findings of the case study suggest the
applicability of the proposed approach in a wider context within the public
sector.
A limitation of the study is the number of case
studies and countries. The case studies are performed in three organizations
from two countries. Additional case studies in different agencies and countries will improve the
generalizability of the results.
Future
studies include increasing the number of case studies performed in different
countries and agencies. Additionally, the application of the approach to other
governmental processes is planned. The purpose is to acquire feedback from a
representative collection of the case studies in order to improve the
evaluation, refinement and validation of the approach. Another future study comprises development of the PFPRM process
capability self-assessment approach covering a comprehensive set of questions
and alternative answers. Publishing the approach over the internet and the
collection of new assessment data from various government organizations from
different countries and benchmarking the data. Moreover, it is
also planned to develop of a tool for providing self-assessment of PFPRM by
process owners. Thus, an employee working in the finance department can assess
the process to observe the weaknesses and to improve their PFPRM without any
help.
6. References
[1] A. Ertürk,
Influences of HR practices, social exchange, and trust on turnover intentions
of public IT professionals, Public Pers. Manage. 43 (2014) 140–175.
[2] D.N. Burt,
D.W. Dobler, S.L. Starling, World class supply management: The key to supply
chain management, Irwin/McGraw-Hill, 2003.
[3] ISO, ISO/IEC
15504-2: Information technology - Process assessment - Part 2: Performing an
assessment, (2003).
[4] ISO, ISO/IEC
15504-3: Information technology -- Process assessment -- Part 3: Guidance on
performing an assessment, (2004).
[5] ISO, ISO/IEC
15504-4: Information technology -- Process assessment -- Part 4: Guidance on
use for process improvement and process capability determination, (2004).
[6] ISO, ISO/IEC
15504-5: Information technology - Process assessment - Part 5: An exemplar
Process Assessment Model, (2012).
[7] C.P. Team,
CMMI® for Development, Version 1.3, Improving processes for developing better
products and services, Softw. Eng. Inst. (2010).
[8] ISO, ISO/IEC
33000: Information Technology – Process Assessment, International Organization
for Standardization, (2015).
[9] T.P. Rout,
Critical design decisions in the development of the standard for process
assessment, in: Int. Conf. Softw. Process Improv. Capab. Determ., Springer,
2013: pp. 247–251.
[10] S.I.G.
Automotive, Automotive SPICE Process Assessment Model, Final Release, v4. 4
(2010) 46.
[11] F. Mc
Caffery, A. Dorling, Medi SPICE development, J. Softw. Maint. Evol. Res. Pract.
22 (2010) 255–268.
[12] L. Ibrahim,
Improving process capability across your enterprise, in: 4th World Congr.
Softw. Qual. (4WCSQ), Bethesda/USA, 2008.
[13] A.L.
Mesquida, A. Mas, E. Amengual, J.A. Calvo-Manzano, IT Service Management
Process Improvement based on ISO/IEC 15504: A systematic review, Inf. Softw.
Technol. 54 (2012) 239–247.
[14] E. Gökalp, O.
Demirörs, Government process capability model: an exploratory case study, in:
Int. Conf. Softw. Process Improv. Capab. Determ., Springer, 2014: pp. 94–105.
[15] E. Gökalp, O.
Demirörs, Proposing an ISO/IEC 15504 based process improvement method for the
government domain, in: Int. Conf. Softw. Process Improv. Capab. Determ.,
Springer, 2015: pp. 100–113.
[16] E. Gökalp, O.
Demirörs, Developing Process Definition for Financial and Physical Resource
Management Process in Government Domain, in: M.P. Clarke, V.R. O’Connor, T.
Rout, A. Dorling (Eds.), Softw. Process Improv. Capab. Determ. 16th Int. Conf.
SPICE 2016, Dublin, Ireland, June 9-10, 2016, Proc., Springer International
Publishing, Cham, 2016: pp. 169–180. doi:10.1007/978-3-319-38980-6_13.
[17] E. Gökalp, O.
Demirörs, Kamu Kurumları için Süreç Yetenek Modeli Geliştirilmesi., in: UYMS,
2014.
[18] E. Gökalp, O.
Demirörs, Towards a Process Capability Assessment Model for Government Domain,
in: M.P. Clarke, V.R. O’Connor, T. Rout, A. Dorling (Eds.), Softw. Process
Improv. Capab. Determ. 16th Int. Conf. SPICE 2016, Dublin, Ireland, June 9-10,
2016, Proc., Springer International Publishing, Cham, 2016: pp. 210–224.
doi:10.1007/978-3-319-38980-6_16.
[19] E. Gökalp, O.
Demirörs, ISO/IEC 15504 Standardının Devlet Kurumları için Uyarlanması, (2015).
[20] D.
Maheshwari, M. Janssen, Measurement and benchmarking foundations: Providing
support to organizations in their development and growth using dashboards, Gov.
Inf. Q. 30 (2013) S83–S93.
[21] P. Hong, D.
Dobrzykowski, Y. Won Park, P. Hong, S.W. Hong, J. Jungbae Roh, K. Park,
Evolving benchmarking practices: a review for research perspectives,
Benchmarking An Int. J. 19 (2012) 444–462.
[22] J. Zwicker,
P. Fettke, P. Loos, Business process maturity in public administrations, in:
Handb. Bus. Process Manag. 2, Springer, 2010: pp. 369–396.
[23] J. Raymond,
Benchmarking in public procurement, Benchmarking An Int. J. 15 (2008) 782–793.
[24] R.G. Rendon,
Procurement process maturity: Key to performance measurement, J. Public
Procure. 8 (2008) 200.
[25] M. Møller, J.
Hedegaard, K. Petersen, A. Vendelbo, S. Jakobsen, Development model for public
procurement in a Danish context, in: Proc. from 4th Int. Public Procure. Conf.,
2006.
[26] J. Waterman,
L. Knight, Achieving continuous improvement through self-assessment, in: Int.
Public Procure. Conf., 2010.
[27] G. Concha, H.
Astudillo, M. Porrua, C. Pimenta, E-Government procurement observatory,
maturity model and early measurements, Gov. Inf. Q. 29 (2012) S43–S50.
[28] C. I. O.
Council, Federal Enterprise Architecture Consolidated Reference Model Document
Version 2.3, 2007.
[29] American
Productivity & Quality Center (APQC), Process Classification Framework,
Washington, DC, 2012.
[30] J.W.
Creswell, Research design: Qualitative, quantitative, and mixed methods approaches,
Sage publications, 2013.
[31] R.K. Yin,
Case study research: Design and methods, Sage publications, 2013.
[32] E. Gökalp,
Technical Report of Financial and Physical Resource Management Process
Assessment in North Cyprus Turkish Republic Ministry of Labor and Social
Security, 2016.
[33] E. Gökalp,
Technical Report of Financial and Physical Resource Management Process
Assessment in North Cyprus Turkish Republic Ministry of Health, 2016.
[34] E. Gökalp,
Technical Report of Financial and Physical Resource Management Process
Assessment in Ministry of Development in Turkey, 2016.