Text
A SPICE Based Process Assessment Model for Public Financial and Physical Resource Management Process
Ebru Gökalp[1], Onur Demirörs
Informatics Institute, Middle East Technical University, Ankara, Turkey
 
Abstract
The necessity of transformation for efficient and effective management of Public Financial and Physical Resource Management (PFPRM) process has become increasingly critical in the governmental organizations. However, as a result of literature review, it is observed that there is a lack of a guideline for process capability determination and improvement of PFPRM process. ISO/IEC 15504, also termed Software Process Improvement and Capability dEtermination (SPICE), is used as a baseline to generate capability/maturity models for different specific domains/sectors, based on the observed benefits obtained in software organizations. We have utilized the same approach for the government domain, and process definition of PFPRM based on the requirements of SPICE is developed. As a result, PFPRM process can be assessed based on SPICE to be consistently applied, managed, and controlled across governmental agencies. A multiple case study, including the assessments of three organization’s PFPRM process capability level is performed. The assessment results are used to develop road-maps for implementing PFPRM process improvement in the study. The findings show that the proposed approach is applicable for identifying the current state of the process capability and the gaps with the assessed capability level of the PFPRM process performed in governmental organizations.
Keywords: SPICE, Government, Financial Management, Physical Resource Management, Process Capability Determination, ISO/IEC 15504
1. Introduction
The governments are under pressure to improve the service performance with limited budget. A key government priority is to considerably reduce costs and achieve better value for money through reforming public financial and physical resources management (PFPRM) process. While attempting to adopt and comprehend effective, contributory, and transparent administration measures, it is confronted with the challenge of transformation and the need to reengineer governmental processes and systems [1]. In the wake of the financial problems faced by the government during the 2000s, the necessity of transformation for efficient and effective management of PFPRM process has become increasingly critical in the governmental organizations.
As the government organizations continue to focus on core competencies and outsource non-core, yet critical functions, these organizations are depending on PFPRM process as a key enabler to achieve and to maintain them. Today's government organizations must now manage an increasing number of contractors and suppliers who are carrying out mission critical functions for their organizations. Thus, government sector organizational core competences now include the PFPRM process. As stated in [2], government sector organizations should be concentrating their attention on measurement of the process performance of the PFPRM in order to improve its core competencies.
The capability/maturity models are used to assess the level of capability of the most critical processes of many organizations. There are well-accepted Process Capability/Maturity Models (PCMMs), such as ISO/IEC 15504- also termed Software Process Improvement and Capability dEtermination (SPICE) [3–6],CMMI-DEV (Capability Maturity Model Integration for Development) model [7]. The ISO/IEC 15504 standard has entered a revision cycle to be transformed from a single, multi-part standard to a set of related standards using a defined numeric range, the ISO/IEC 330xx series [8]. This transformation is intended to result in an expanding range of application, and to importantly more open standards framework. The revised requirements for the standard are aimed to provide the extension of process assessments to fields outside the ICT domain [9].These PCMMs are used as a basis for evaluation and comparison for process assessment and/or improvement, assuming that higher level of process capability or organizational maturity is associated with better performance. These models are developed for the purpose of performing assessments of software and systems processes. As a result of the observed benefits of these PCMMs such as expense savings, increased involvement of employees, predictable and improved quality and productivity, and constructing a consistency of process capture and use; customizing them to different domains other than software development is the subject of growing interest in the literature. Accordingly, many initiatives based on SPICE are proposed for various domains: Automotive SPICE [10], Medi SPICE [11], Enterprise SPICE [12], and IT Service Management [13].
We have utilized the same approach for the government domain by developing the Government Process Capability Determination Model (Gov-PCDM) [14–19] based on SPICE [3–6]. The major reason for selecting SPICE as a basis is its well-defined and commonly accepted structure. The aim of the Gov-PCDM is to provide the base for improving the processes performed in public organizations. It pursues a standardized and structured approach for governmental process assessment for performing quality improvement initiatives in a consistent, repeatable manner, assessed by competent metrics with a roadmap on what to do to improve quality. In the scope of this study, we analyze one of the processes defined in Gov-PCDM, the PFPRM process.
The capability levels defined in the SPICE measurement framework from level zero to level five have been developed to be appropriate universally to all processes except for level-1 (performed) where the observable indicators are different for each process, while all the process attributes from level two to five are common for all processes. In the scope of this study, the process definition of PFPRM process is developed, including level-1 process performance indicators as outcomes, base practices and work products. Thus, the PFPRM process capability level can be assessed based on SPICE owing to this developed process definition.
In this paper the method for how to develop the process definition of PFPRM and how to conduct process capability level assessment is described in detail. Moreover, multiple case studies conducted in order to check the usefulness and adequacy of the proposed approach, is discussed. As part of the case study, the capability levels of the PFPRM process performed in three governmental institutions are assessed, roadmaps for improving the process to the next capability levels are derived and the results of each assessment are presented. Over the presentation, the results are discussed with practitioners in meetings. Then, follow-up interviews are conducted after the meeting to ask if the results correctly represent the state of the process. Finally, the results are analyzed in the study.
This paper is divided into five sections. In the second section, a brief literature review about PFPRM process improvement is given. The method for the development of PFPRM process definition and capability assessment is provided in the third section. In the fourth section, the design and implementation of the multiple case studies conducted in three governmental organizations are presented and the analyzed results as well as the derived road-maps to improve the assessed processes are given. The conclusions and future studies are given in the last section.
2. Literature Review
The literature reflects an increasing flow in the research stream of public sector performance assessment and benchmarking as stated in [20]. Hong et al. [21] also identified a growing need for assessment and benchmarking studies of complex business practices and proliferation of research studies in the area of public sector processes. Fettke et al. [22] demonstrated the use of a business process maturity model to improve service response efficiency and effectiveness in public administration.
The use of maturity models for PFPRM process, such as contracting and procurement sub-processes, has seen some, but limited, application in the public sector. Raymond [23] stated that the necessity for PFPRM process best practices is increasing, especially in the areas of value for money, ethics, competition, transparency, and accountability. Rendon [24] developed Contract Management Maturity Model to assess and measure a US Air Force contracting agency’s procurement process and identified process improvement initiatives. Møller et al. [25] developed and applied a public procurement maturity model for the Denmark government. The use of their model will improve standardization, consistency, and transparency of Danish public procurement organization practices. Waterman and Knight [26] explored using a capability maturity model for conducting self-assessments in a case study on U.K. government procurement departments. Concha et al. [27] introduced the e-Government Procurement Observatory Maturity Model (eGPO-MM) to measure government e-procurement portals status across the Latin American region and to enable development of an improvement road map for eGP in each participating country.
We have also performed research studies related to customization of SPICE for the government domain: A literature review that results in the necessity of a structured process improvement model regarding the specialties of the government domain [17], an exploratory case study to check if the customization of SPICE for government domain is applicable [14], a study of proposing a SPICE based process improvement method for government domain [15], development of a generic process definition for governmental agency-specific processes and conducting case studies to investigate the appropriateness [18,19], and a study of development of a process definition for one of the seven management and support processes [16]. The initial findings of our studies show the applicability and usefulness of the approach of SPICE-based government process capability determination model of Gov-PCDM.
The literature review can be summarized by concluding that there is a growing research stream in the benefits and challenges of measuring PFPRM process. Moreover, the literature review identified a research gap in that there is limited research in the use of maturity models for assessing PFPRM process. Of the research identified in the review, none of the maturity models identified is based on key PFPRM process base practices, but on other procurement and contracting functions. Additionally, they are not based on one of the well-accepted PCMMs. Thus, they do not provide assessments of PAs applicable universally to all processes, such as process control, process measurement, etc. Lastly, none of them is implemented in the Turkish governmental organizations. The research in this study aims to fill this gap by providing a SPICE-based process capability determination model of Gov-PCDM [14–19] for PFPRM process capability determination to perform the process improvement in a structured and standardized way.  The application of the proposed approach in case study settings is performed in Turkish governmental organizations.
2  
3. Method
The method followed to develop the process definition of the PFPRM process and to conduct the process capability level assessment is defined in this section.
 
3.1 Development of the Process Definition of PFPRM Process
The process definition, based on the requirements defined in ISO/IEC 15504-2 [3] is characterized by process purpose (the objective of accomplishing the process), outcomes (the observable consequences expected from successfully performing the process), base practices (a list of actions that may be used for the achievements of the outcomes), and work products (individually recognizable bodies of information created and stored for human use).
The development of the process definition of PFPRM is illustrated in Figure 1. As a first step, documents related to business rules and policies of the government for the PFPRM, as well as existing quality improvement models and standards are investigated. The process definition of PFPRM is developed by harmonizing existing quality improvement models and standards as FEAF (Federal Enterprise Architecture Framework) [28], APQC (American Process Qualification Center) [29], and SPICE-ISO/IEC 15504- Part 2 [3], based on the business rules and policies of the PFPRM stated in laws, decree-laws, guidelines and regulations, as Public Procurement Law, Public Financial Management and Control Law, and Budget Preparation Guideline. The process definition is defined based on the requirements stated in SPICE-ISO/IEC 15504- Part 2 [3]. Process purpose is defined based on Business Reference Model (BRM) defined in FEAF [28] and business rules and policies stated in governmental documents. Outcomes and work products are defined based on the business rules and policies. Base practices and sub-base practices are adapted from APQC [29]. The draft version of PFPRM definition is developed by one of the authors.
The process definition draft is formally reviewed by five PFPRM process owners who are working in the finance department in government agencies. They are requested to provide verbal and written feedback on the following questions:
(1) Are the major elements of the process definition of PFPRM; such as, outcomes, base practices, and work products well defined and articulated?
(2) Is there any missing information in the process definition of PFPRM?
The feedbacks are used to refine and revise the model. The revised version of the process definition of PFPRM is reviewed and approved by the management with executive responsibility within two organizational units by two people and one of the authors who has both professional and academic experience in using. Consequently, the final version of the process definition of PFPRM which is given in Table 1 is obtained.
 
Fig. 1. The Development of Process Definition of PFPRM
 
The process definition is needed to perform PFPRM process capability level assessment based on SPICE. It is used in level-1 assessment to check if the process is performed. The method followed for the capability level assessment is defined in the following sub-section.
 
Table 1. The Process Definition of PFPRM Process     

Purpose
The purpose of public financial and physical resource management is to deploy and use the government’s resources, facilities and assets.
 
Outcomes
As a result of successful implementation of the financial and physical resource management process;
1. Financial and physical resource strategy and policies are established.
2. The detailed financial plan (budget) containing cost estimates for consumed resources and, where applicable, revenue projections for fees* received is generated.
3. Procurements of goods, services or works are performed based on public procurement law.
4. Finance and accounting transactions are handled for procured goods/services or works and receivable where applicable.
5. Physical resources are acquired, constructed and disposed.
6. Warehouse used for storing tangible physical resources is managed.
7. Reports including internal and external financial information are generated.
8. Internal and external audits are conducted.
* Many government services issue licenses and permits and collect an associated fee.
 
Base Practices
MGRSP1.BP1: Establish and maintain a strategy and policies for financial and physical resource management: Establish and maintain a strategy and policies for financial and physical resource management. [Outcome: 1]
1.1.1. Build a strategic plan to support business objectives
1.1.2. Design capital structure
 
MGRSP1.BP2: Perform budgeting:  Prepare periodic detailed budgets and plans and financial forecasts, according to established strategy and policies.[Outcome:1, 2]
1.2.1 Develop annual budget proposal
1.2.2 Get approval for the budget from ministry of finance
1.2.3 Develop periodic detailed financial plan/budgets and forecasts based on the approved budget
1.2.4 Allocate resources
1.2.5 Manage financial risk
1.2.6. Manage fee administration, where applicable
 
 
 
 
 
 
 
Table 1. The Process Definition of PFPRM Process (Continued)

Base Practices
 
 
 
Base Practices
MGRSP1.BP3: Procure goods/services or works:   Purchase goods/services or works based on the public procurement law  [Outcome:2,3]
1.3.1 Elicitate need and requirements.
1.3.2 Prepare technical contract.
1.3.3 Conduct market research to calculate the approximate cost.
1.3.4 Determine tender procedure.
1.3.5 Prepare documents related to tender including proposal evaluation criteria.
1.3.6 Obtain approval for the tender.
1.3.7 Define tender committee.
1.3.8 Publish invitation for bidding.
1.3.9  Review tender documents.
1.3.10 Receive tender proposals.
1.3.11 Apply evaluation criteria to select a provider, negotiate contract terms and conditions to resolve open items and select the contractor.
1.3.12 Invite the selected contractor to sign the contract.
1.3.13  Monitor contractor performance.
1.3.14  Close the contract after ensuring that each party’s performance meets contractual requirements.
 
MGRSP1.BP4: Process finance and accounting transactions: Process all the transactions related to purchasing products/services, paying, and receiving. [Outcome: 3,4]
1.4.1 Process  payable accounts
1.4.2 Process receivable, credit, and collections  accounts
 
MGRSP1.BP5: Manage physical resources: Establish requirements and standards for physical items which are acquired, constructed and disposed. [Outcome: 5, 6]
1.5.1 Acquire and deploy assets
1.5.2 Manage facilities
1.5.3 Manage physical risk
1.5.4 Dispose nonproductive physical assets
 
MGRSP1.BP6: Operate Warehousing Collect, receive, and store assets, according to established strategy and procedures. [Outcome: 6, 7]
1.6.1 Track inventory deployment
1.6.2 Receive, inspect, and store deliveries
1.6.3 Track product availability
1.6.4 Record checkouts of store
1.6.5 Track inventory accuracy
1.6.6 Track third-party logistics, storage and shipping performance
1.6.7 Manage physical finished goods inventory
 
 
Table 1. The Process Definition of PFPRM Process (Continued)

Base Practices
 
 
 
Base Practices
MGRSP1.BP7: Report information: Report transactions to accounting department (internal) and court of accounts (external). [Outcome: 7]
1.7.1 Provide external financial information
1.7.2 Provide internal financial information
 
MGRSP1.BP8: Conduct internal and external audits: Determine compliance of performed process with the requirements, plans, laws and procedures, as appropriate. [Outcome: 8]
1.8.1 Develop and implement audit strategy
1.8.2 Plan audit
1.8.3 Perform Auditing
1.8.4 Identify corrective actions from the audit report
1.8.5 Track actions for audit report
 
Work Products
Inputs
Outputs
Public Financial Management and Control Law [Outcomes: 1,8]
 
Budget Preparation Guideline [Outcomes: 2]
 
Public Procurement Law [Outcomes: 3]
 
Public procurement Contracts Law [Outcomes: 3]
 
Asset Legislation [Outcomes: 5,6]
 
Regulation on Prepayment Procedures and Principles [Outcomes: 2,3,4]
 
Regulation on the Principles and Procedures of Internal Control and Preliminary Financial Control [Outcomes: 7,8]
 
 
Budget Proposal [outcome:2]
 
Detailed financial plan [outcome:2]
Payment Order Document [outcome:4]
 
 
Warehouse Documents for (asset request stock-in, stock out, inspection and acceptation) [outcome:6]
 
Appropriation Transfer Document [outcome:7]
 
Audit Report [outcome 8]
3.2 Process Capability Level Assessment
The measurement framework is based on SPICE. Process capability is classified into six levels in ISO/IEC 15504-2 [3] as Level 0: Incomplete; Level 1: Performed; Level 2: Managed; Level 3: Established; Level 4: Predictable; Level 5: Optimizing.
The measure of capability is based upon a set of PAs. Process capability indicators are the means of achieving the capabilities addressed by the considered PAs. Evidence of process capability indicators supports the judgment of the degree of achievement in the PA.
The PA of Level-1 is Process Performance which is a measure of the extent to which the process purpose is achieved. The developed PFPRM process definition given in Table 1 is used for the Level-1 assessment. The base practices and work products defined in process definition are used as ‘process performance indicators’. For the assessments of levels 2 to 5, exactly the same PAs, ‘generic practices indicators’, ‘generic resources indicators’ and ‘generic work products indicators’ as the exemplar PAM provided by the ISO/IEC 15504-5 [6] are used.
The capability level of each process instance is determined by rating PAs. For example, to determine whether a process has achieved capability level-1 or not, it is necessary to determine the rating achieved by PA 1.1 (Process performance). A process that fails to achieve capability level-1 is at capability level 0. Each PA is measured by an ordinal rating that represents the extent of achievement of the PA, as F (Fully) meaning 86% to 100% of achievement of the PAs, L (Largely) meaning 51% to 85% of achievement of PAs, P (Partially) meaning 16% to 50% of achievement of PAs, or N (Not) meaning 1% to 15% of achievement of PAs, achieved. A process is defined to be at capability level n if all PAs below level n are rated as F and the level n PA(s) are rated as F or L, as defined in ISO/IEC 15504-2 [3].
The assessment procedures for details of the activities of assessment planning, briefing of the participants, collecting and validating as well as reporting are based on ISO/IEC 15504-3 [4].
4. Multiple Case Study
To evaluate the applicability of the proposed approach in different cases, we applied multiple case study research. The multiple case study design, implementation and analysis of findings are given in this section.
4.1 Multiple Case Study Design
The research strategy of this study matches many of the qualitative research attributes [30]. The data is needed to be collected in its natural settings; the assessor is the key instrument in collecting the data, there are various forms of data and inductive data analysis is needed to be conducted.
The case studies are conducted by the protocol template proposed by Yin [31].
The objective of the study is to investigate if the proposed approach can be utilized for the assessment of the process capability level determination of PFPRM and for the achievement of roadmaps for PFPRM process improvement.
The following research question is defined in accordance with the objective above: “How suitable it is to use the approach of SPICE based government process capability determination model with the purpose of identifying the current state of the PFPRM process capability and the gaps with the assessed capability level, as well as how well it provides roadmaps for improving the PFPRM process capability of the governmental organizations?”
The measure used in the research is the capability level of the governmental processes.
Case Selection Strategy is to select organizations that have been subject to one of our previous studies where we analyzed their processes. We needed organizations for which we have already investigated the strengths and weaknesses of the processes. From this perspective, the organizations will enable us better to observe if the approach capable of revealing these strengths and weaknesses and indicating the capability level of the process. Another criterion for the selection of the process is the necessity of process improvement. The last criterion for the case study selection is having the judgement of the organization on the requirement of the studies for the process capability level determination and generating a roadmap for process improvement. Without such a judgement, the study might be considered as a burden as part of the organizations’ daily studies.  The success of the process capability level assessment is highly dependent on the process participants’ contributions.
Field Procedures and Data Collection is based on the ISO/IEC 15504. The assessment procedures for details of the activities of assessment planning, briefing of the participants, collecting and validating as well as reporting are based on ISO/IEC 15504-3: Guidance on Performing an Assessment [4]. This ensured assessment planning, assessment performing, data collection and creating documents in a standard format. The process assessment team consists of the participants in the organization responsible for the quality assurance and authors of the study, one of the authors is a formally certified ‘competent’ assessor.
A visit is planned to be performed in order to make each process assessment, for which evidence gathering techniques as inspections of the documents (the documentation as law, decree-law, policies, or other documents using for the process) and as conducting semi-structured interviews with the people involved (participants) in the actual execution of the process. The assessment team will use this information to create the assessment report for each agency.
Sources of evidences are the process capability level assessment interviews, follow-up interviews conducted with process stakeholders after sharing the assessment results in a meeting, and the information-gathering documents specifically defined for the process. The assessment team gathers information independently from the documents to be evaluated (so the documents as law, decree-law, policies, or other documents using for the process are inspected) and the person involved (participant) in the actual execution of this process. Then, the formal assessments are conducted through semi structured group interviews with process participants, and the direct evidences are analyzed. After the process capability level assessment interviews, process capability level is determined, and assessment reports are prepared. Following, assessment results are shared and discussed with process participants in a meeting by the assessment team. The follow-up interviews are conducted to obtain process participants’ opinion on the assessment results after the meeting.
The measure of the process capability level is based upon a set of PAs as defined in the sub-section 3.1 process capability level assessment. The rating will be performed based on gathered evidences from inspected documents and process capability level assessment interviews by the assessment team.
4.2 Multiple Case Study Implementation
Multiple case studies are performed in the Turkish Republic Ministry of Development, North Cyprus Turkish Republic Ministry of Health and, North Cyprus Turkish Republic Ministry of Labor and Social Security in order to evaluate the adequacy of the proposed approach. Case Studies are selected among the organizations that have been subject to one of our previous studies where we analyzed their PFPRM processes. We have already known the strengths and weaknesses of the PFPRM process performed in these organizations.
Organization-1: Turkish Republic Ministry of Development plans and guides the country’s development process in a macro approach and focuses on the coordination of policies and strategy development. It has 38 departments, 818 employees. An information system, named E-budget, integrated with the Turkish Republic Ministry of Finance and Public Contract Institute is used to coordinate the financial activities. The finance department is responsible for carrying out all works related to procurement, contracting, and physical resources management.
Organization-2: North Cyprus Turkish Republic Ministry of Health is in charge of regulating the health care system. 87 people are working in the ministry.
Organization-3: North Cyprus Turkish Republic Ministry of Labor and Social Security is responsible for labor and social security affairs. 141 employees are working in the ministry. An information system, named E-finance, integrated with the North Cyprus Turkish Republic Ministry of Finance is used to coordinate the financial activities in Organization-2 and Organization-3. The finance department is responsible for carrying out all works related to procurement, and physical resources management.
The process capability level assessments were performed by the assessment teams consisting of four people for each organizations; two were participants in the organizations responsible for the quality assurance, and the authors, one of whom is a formally certified ‘competent’ assessor. After inspections of the related documents, the assessment team conducted the process capability level assessment interviews with six, two, and three PFPRM process owners, within 175, 85, and 95 minutes in Organization-1, 2 and 3 respectively. Five of the PFPRM process owners have more than five years’ work experiences. Three of them have three years’ work experiences, three of them have less than two years’ work experience.
Table 2 PFPRM Process Assessment Results in Organization-1
Attribute
Evidences
Assessment Value
1.1 Process
Performance
The process clearly achieves its purpose by maintaining financial and physical resource management.
Fully Achieved
2.1 Performance Management
The performance is planned and managed informally, but they are not adjusted. Performance quality criteria are not defined and not monitored. There is no evidence of meeting, reviews and corrective actions. The e-budget system is used to manage the interfaces.
Largely Achieved
2.2 Work
Product
Management
 
Work products are defined, revisions of the work products are stored in information systems but their appropriate review and approval criteria are not identified. Additionally, they are not reviewed.
 
Largely Achieved
 
3.1 Process Definition
The process is defined in the related governmental laws, but guidance for the process is not provided. The sequence and interaction of activities are not documented. Effectiveness of the process is monitored with performance metrics defined in the yearly performance plan.
Largely Achieved
3.2 Process Deployment
Governmental regulations define specific requirements of the process, required human resources and information are available, allocated and used. However, necessary competencies for performing the process are not defined. Data about the performance of the process to demonstrate its suitability and effectiveness is not collected.
 
Partially Achieved
The execution of the process are nearly the same in Organization-2 and Organization-3. Thus, the evidences for the capability level assessment are common. Summary of the evidences are presented in Table 2 for organization-1 and Table 3 for organizations 2 and 3. More details of the assessments are provided in the assessment reports [32–34].
Table 3 PFPRM Process Assessment Results in Organization-2 and Organization-3
Attribute
Evidences
Assessment Value
1.1 Process
Performance
A strategy and policies for financial and physical resource management are established. Budgeting and procurement are performed based on laws. However, base practices for physical resource management and warehouse operations are partially achieved.
Largely Achieved
2.1 Performance Management
The performance is planned informally, roles are defined in the regulations, and resources are made available to perform the process. The e-finance system is used to manage the interfaces. However, performance objectives are not defined and adjusted. Performance quality criteria are not defined and not monitored. There is no evidence of meeting, reviews and corrective actions.
Largely Achieved
2.2 Work
Product
Management
Work products are in a standard form, but the requirements for documentation and control of the work products are not defined. Quality criteria, appropriate review and approval criteria are not identified. Additionally, they are not reviewed and adjusted.
 
Partially Achieved
3 
The result of the PFPRM process assessment is that the capability level of the process performed is Level 2 in the Organization-1 and Level-1 in Organizations-2 and Organization-3 with the rationale based on collected and validated evidence as given in Table 4.
Table 4 PFPRM Process Assessment Results
Process Attributes
Org.-1
Org.-2
Org.-3
P.A.1.1 Process Performance
F
L
L
P.A.2.1 Performance Management
L
L
L
P.A.2.2 Work Product Management
P.A.3.1 Process Definition
L
L
P
-
P
 
P.A. 3.2 Process Deployment
P
 
 
….
 
 
 
Result
Level-2
Managed
Level-1
Performed
Level-1 Performed
 
4.3 Analysis of Findings
 The capability levels of the PFPRM processes are determined after conducting the assessments and rating the PAs based on the gathered evidences. The roadmaps to improve the capability level of the processes are derived from the assessment evidences based on ISO/IEC 15504-4 [5]. The assessment results are presented with senior managers of the organizations, process participants in a meeting. The ratings for each PA and evidences for that rating is explained. The derived guideline for process improvement is also shared. The derived guideline for the process capability level improvement and the follow-up interview results are given in this sub-section.
4.3.1 Guideline for Improvement Capability of the Process
 
In order to improve the capability level of the PFPRM process performed in Organization-1 to the next capability level, level 3, assessment values of the PAs should be as follows; Performance and Work Product Management attributes: Fully Achieved, Process Definition and Deployment attributes: Largely or Fully Achieved. For PFPRM process capability level improvement in organizations 2 and 3 to the next capability level, level 2, assessment values of the PAs should be as follows; Process Performance attribute: Fully Achieved, Performance and Work Product Management attributes: Largely or Fully Achieved.
 The roadmap to improve the capability level of the processes is derived from the assessment evidences summarized in Table 2 and Table 3. The aim is to turn negative evidences into positive suggestions of process capability indicators supporting the judgment of the degree of achievement of the PA. For example; for the performance management attribute; the second indicator (Generic Practice 2.1.2) is to plan and monitor the performance of the process to fulfill the identified objectives. Negative evidence, observed during the assessment for this indicator is that the process work product reviews are not planned. Thus, necessity of reviewing work products is indicated in the guideline. Some base practices are not fully achieved in Organizations 2 and 3. In order to improve the process capability level, all of base practices should be fully achieved. Thus, the base practices which are observed and validated as not fully achieved are indicated in the guideline listed as below. The first four items are common for three organizations. The process management item is for guideline of Organization-1. The items indicating deployment of the expense, assets, and warehouse management systems and internal and external auditing mechanism are for guideline of Organizations 2 and 3 in addition to four common items.
Organization-1, Organization-2, and Organization-3
1. Work-product Management
- Define requirement of work product
- Define quality criteria
- Define appropriate review and approval of work products. And also, review the work products based on this definition.
- Define relations between work products
- Deploy review and adjustment mechanism
2. Configuration Management
- Provide a change control mechanism for the work product
- Manage versions of the work products
3. Workflow Management
- Define activities, tasks, responsible employees, authorities and resources
- Define sequence and interaction of activities
4. Risk Management
- Define risks related to fulfill the objective of the process
- Manage financial risk
- Develop problem and issue management mechanism
- Define how to adjust the objective when needed
Organization-1
5. Process Management
- Verify the conformance of defined process with standard process requirements officially.
- Identify and collect data to monitor process performance
- Monitor and adjust the process performance indicators if necessary
Organization-2 & Organization-3
5. Deploy Expense Management System
6. Deploy Asset Management System
7. Deploy Warehouse Management System
8. Deploy Internal and External Auditing Mechanism
4.3.2 Follow-up Interviews with Process Participants
In order to check the usefulness and adequacy of the proposed approach, the follow-up interviews were conducted with process participants and senior managers of the organizations after sharing the assessment results in a meeting. The interviews took about 10 minutes. The open-ended structured questionnaire given in Table 4 is utilized. We used 5 points scale in the first two questions. It includes options as: strongly agree (5 points), agree (4 points), neutral (3 points), disagree (2 points), strongly disagree (1 points). The median of the responses is calculated from 8, 4, and 6 people (6, 2 and 3 of them are process owners) for Organization-1, 2, 3 respectively.
The findings in the conducted interviews support our proposed approach. Since the median of the responses for the first two questions are calculated as 4 in 5 points Likert scale. It is obtained from the interview results that they think that achieving a road map to guide what to do for increasing process capability is useful, all of the suggestions indicated in the guideline will improve the process performance of the PFPRM process. They also confirm that process definition of PFPRM process covers all outcomes of the process. While answering the last question, two of the respondents pointed out some possible improvement areas such as interoperability of involved parties; such as public procurement agency to keep track of stages in procurements in Organization-1. However, this is out of our scope and is primarily related to e-government initiatives.  They reported the main benefits of the assessment as realizing the need for measuring PFPRM process capability level in its specific environment. They stated that the biggest contribution to the improvement of the process is provided by developing risk management system and monitoring the effectiveness and suitability of the process.
 Table 4 Results of Interview with the Stakeholders for PFPRM Process
Question
Survey Type
Response
Q1) Are measuring PFPRM process capability and obtaining guideline for improvement useful?
5 points Likert scale
Median: 4
Q2) Do you think that applying these suggestions will improve the PFPRM process performance?
5 points Likert scale
Median: 4
Q3) Is there any information you want to add in process definition of PFPRM process? Please write, if any.
Open-end
No.
Q4) Is there any missing item(s) in guideline for improvement list? Please write, if any.
Open-end
Interoperability
4.3.3 Threats to Validity
As a result of the application of case study research, some possible threats to validity arise. Activities to overcome these threats were planned during the design phase of the case study. The activities performed to avoid threats are explained in this sub-section.
Construct validity considers if the constructs in the case study are well-structured or subjective to the judgment. To avoid these problems, the information is collected from the participants with different roles (process owner, process stakeholder, and executive member) and from multiple sources, including documentation (Laws, decree-laws, regulations), interviews and observations of the participants.
– As for the internal validity, application of multiple case study is remarkably significant to overcome this threat. A logical chain of evidence was established while performing the study and reporting the results. The evidences collected during the case studies were given in detail in the assessment reports [32–34]. Different sources of evidences were utilized for analysis of the results and answering the research question. The resulting outputs were validated by the related stakeholders by conducting the interviews.
External validity deals with the concern of the generalizability of the case study results. We conducted multiple case studies where we can apply replication logic to overcome this threat. We initially applied the approach in Organization-1. These first application results were reviewed, approved, and the protocol and the field procedure of the case study were refined. Then, the replication material of the case study was applied to Organization-2 and Organization-3. It is ensured that the replication logic was applied consistently through the cases and consistent outputs could be achieved through multiple executions of the same or different cases.
– Regarding reliability, many activities were performed to avoid reliability problems and to ensure that other researchers can perform the same study following the methodology. A case study protocol defined by Yin [31] was followed for each case study, where the objectives, corresponding research question, plan, sources of the evidences of the case study are identified. Additionally, the replication material of the case study was developed. It was observed that following this material results in similar findings and conclusions.
5. Conclusion
Domain specific business process improvement models are more than welcomed by various industries in recent years. Although the concept of process improvement models is not new, the application of process improvement models to the public sector has not been extensively studied. The process improvement of PFPRM process is a key government priority, since government organizations are under pressure to limit the budgets and to achieve effective, transparent, and contributory administration measures.
In order to provide PFPRM process improvement, SPICE, one of the well-accepted PCMMs, is taken as the baseline. Accordingly, Financial and Physical Resources Management process in the governmental institutions is defined based on the requirements stated in ISO/IEC 15504-2 [3]. Thus, the PFPRM process capability level can be assessed based on SPICE owing to this developed process definition. The method for developing the PFPRM process definition and for conducting process assessment is described, followed by a multiple case studies in three organizations, which is performed to check the applicability of this approach.
The results of the multiple case studies highlight that the proposed approach is successful at identifying PFPRM process defects at different process capability levels and is capable of providing a roadmap for moving the process capability level to the next step. The interview results show that PFPRM process stakeholders think that obtaining a roadmap for increasing process capability is useful, and all of the suggestions indicated in the roadmap will improve the process performance of the PFPRM process. They also report the main benefits of the assessment as realizing the need for measuring PFPRM process capability level in its specific environment.
Overall, the proposed approach provides a baseline for initiating and maintaining a continuous process improvement cycle for PFPRM in government organizations. It enables each government institution to evaluate its PFPRM processes in detail, which in turn helps identify the current state of its PFPRM process capability as well as generate a feasible improvement roadmap for moving to the next process capability level. With this, the institution is also benchmarked against its peers which are evaluated by using the same approach.
The study contributes to the literature in terms of extending the boundaries of SPICE by demonstrating its applicability in the government domain. The SPICE community supports applicability of the standard to domains other than software. Another contribution is the observation of the feasibility of the proposed approach through multiple case studies, whereas existing public finance or procurement management maturity models in the literature do not include such applications. The findings of the case study suggest the applicability of the proposed approach in a wider context within the public sector.
A limitation of the study is the number of case studies and countries. The case studies are performed in three organizations from two countries. Additional case studies in different agencies and countries will improve the generalizability of the results.
Future studies include increasing the number of case studies performed in different countries and agencies. Additionally, the application of the approach to other governmental processes is planned. The purpose is to acquire feedback from a representative collection of the case studies in order to improve the evaluation, refinement and validation of the approach. Another future study comprises development of the PFPRM process capability self-assessment approach covering a comprehensive set of questions and alternative answers. Publishing the approach over the internet and the collection of new assessment data from various government organizations from different countries and benchmarking the data. Moreover, it is also planned to develop of a tool for providing self-assessment of PFPRM by process owners. Thus, an employee working in the finance department can assess the process to observe the weaknesses and to improve their PFPRM without any help.
6. References
[1] A. Ertürk, Influences of HR practices, social exchange, and trust on turnover intentions of public IT professionals, Public Pers. Manage. 43 (2014) 140–175.
[2] D.N. Burt, D.W. Dobler, S.L. Starling, World class supply management: The key to supply chain management, Irwin/McGraw-Hill, 2003.
[3] ISO, ISO/IEC 15504-2: Information technology - Process assessment - Part 2: Performing an assessment, (2003).
[4] ISO, ISO/IEC 15504-3: Information technology -- Process assessment -- Part 3: Guidance on performing an assessment, (2004).
[5] ISO, ISO/IEC 15504-4: Information technology -- Process assessment -- Part 4: Guidance on use for process improvement and process capability determination, (2004).
[6] ISO, ISO/IEC 15504-5: Information technology - Process assessment - Part 5: An exemplar Process Assessment Model, (2012).
[7] C.P. Team, CMMI® for Development, Version 1.3, Improving processes for developing better products and services, Softw. Eng. Inst. (2010).
[8] ISO, ISO/IEC 33000: Information Technology – Process Assessment, International Organization for Standardization, (2015).
[9] T.P. Rout, Critical design decisions in the development of the standard for process assessment, in: Int. Conf. Softw. Process Improv. Capab. Determ., Springer, 2013: pp. 247–251.
[10] S.I.G. Automotive, Automotive SPICE Process Assessment Model, Final Release, v4. 4 (2010) 46.
[11] F. Mc Caffery, A. Dorling, Medi SPICE development, J. Softw. Maint. Evol. Res. Pract. 22 (2010) 255–268.
[12] L. Ibrahim, Improving process capability across your enterprise, in: 4th World Congr. Softw. Qual. (4WCSQ), Bethesda/USA, 2008.
[13] A.L. Mesquida, A. Mas, E. Amengual, J.A. Calvo-Manzano, IT Service Management Process Improvement based on ISO/IEC 15504: A systematic review, Inf. Softw. Technol. 54 (2012) 239–247.
[14] E. Gökalp, O. Demirörs, Government process capability model: an exploratory case study, in: Int. Conf. Softw. Process Improv. Capab. Determ., Springer, 2014: pp. 94–105.
[15] E. Gökalp, O. Demirörs, Proposing an ISO/IEC 15504 based process improvement method for the government domain, in: Int. Conf. Softw. Process Improv. Capab. Determ., Springer, 2015: pp. 100–113.
[16] E. Gökalp, O. Demirörs, Developing Process Definition for Financial and Physical Resource Management Process in Government Domain, in: M.P. Clarke, V.R. O’Connor, T. Rout, A. Dorling (Eds.), Softw. Process Improv. Capab. Determ. 16th Int. Conf. SPICE 2016, Dublin, Ireland, June 9-10, 2016, Proc., Springer International Publishing, Cham, 2016: pp. 169–180. doi:10.1007/978-3-319-38980-6_13.
[17] E. Gökalp, O. Demirörs, Kamu Kurumları için Süreç Yetenek Modeli Geliştirilmesi., in: UYMS, 2014.
[18] E. Gökalp, O. Demirörs, Towards a Process Capability Assessment Model for Government Domain, in: M.P. Clarke, V.R. O’Connor, T. Rout, A. Dorling (Eds.), Softw. Process Improv. Capab. Determ. 16th Int. Conf. SPICE 2016, Dublin, Ireland, June 9-10, 2016, Proc., Springer International Publishing, Cham, 2016: pp. 210–224. doi:10.1007/978-3-319-38980-6_16.
[19] E. Gökalp, O. Demirörs, ISO/IEC 15504 Standardının Devlet Kurumları için Uyarlanması, (2015).
[20] D. Maheshwari, M. Janssen, Measurement and benchmarking foundations: Providing support to organizations in their development and growth using dashboards, Gov. Inf. Q. 30 (2013) S83–S93.
[21] P. Hong, D. Dobrzykowski, Y. Won Park, P. Hong, S.W. Hong, J. Jungbae Roh, K. Park, Evolving benchmarking practices: a review for research perspectives, Benchmarking An Int. J. 19 (2012) 444–462.
[22] J. Zwicker, P. Fettke, P. Loos, Business process maturity in public administrations, in: Handb. Bus. Process Manag. 2, Springer, 2010: pp. 369–396.
[23] J. Raymond, Benchmarking in public procurement, Benchmarking An Int. J. 15 (2008) 782–793.
[24] R.G. Rendon, Procurement process maturity: Key to performance measurement, J. Public Procure. 8 (2008) 200.
[25] M. Møller, J. Hedegaard, K. Petersen, A. Vendelbo, S. Jakobsen, Development model for public procurement in a Danish context, in: Proc. from 4th Int. Public Procure. Conf., 2006.
[26] J. Waterman, L. Knight, Achieving continuous improvement through self-assessment, in: Int. Public Procure. Conf., 2010.
[27] G. Concha, H. Astudillo, M. Porrua, C. Pimenta, E-Government procurement observatory, maturity model and early measurements, Gov. Inf. Q. 29 (2012) S43–S50.
[28] C. I. O. Council, Federal Enterprise Architecture Consolidated Reference Model Document Version 2.3, 2007.
[29] American Productivity & Quality Center (APQC), Process Classification Framework, Washington, DC, 2012.
[30] J.W. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches, Sage publications, 2013.
[31] R.K. Yin, Case study research: Design and methods, Sage publications, 2013.
[32] E. Gökalp, Technical Report of Financial and Physical Resource Management Process Assessment in North Cyprus Turkish Republic Ministry of Labor and Social Security, 2016.
[33] E. Gökalp, Technical Report of Financial and Physical Resource Management Process Assessment in North Cyprus Turkish Republic Ministry of Health, 2016.
[34] E. Gökalp, Technical Report of Financial and Physical Resource Management Process Assessment in Ministry of Development in Turkey, 2016.
 



[1] Corresponding author. email: ebruligokalp@gmail.com