Step 2. Analysis of actual practices
This step in the implementation of change model aimed to assess current care in targeted settings by identifying and using measurement indicators.30 Indicators can be divided by structure, process or outcomes and are measurable elements that help evaluate quality of care and identify where changes are most needed.31
Measuring outcomes for mental health problems is a worldwide challenge32 and Canada’s health care system is no exception.33 Compared to other fields of care, many mental health quality measures are not necessarily available in electronic medical records or manageable to use for research and quality improvement initiatives.32 Consequently, we conducted an analysis of actual practices with qualitative data sources instead of analyzing actual performance using quality indicators.
Data sources for this step of the process were collected from recruited participants in each FMG (Table 2) through face-to-face interviews (≈ 45 to 60 minutes) and non-participatory observation of a nurse-patient encounter. We also retrieved and examined relevant documents on professional activities and collaborative care: documents on specific programs for people with LTCs or CMDs, educational materials, clinical nursing mental health assessment canvases, and health care professionals referral forms.
The data collection period lasted from December 2018 to April 2019. Interviews with primary care providers and nurses were the first data to be collected (n=25), followed by interviews with patients (n=7) and observations (n=7). Field notes were written after each interview and observation to describe first impressions and links to previous interviews/observations. The data from each FMG were separately analyzed and interviews and observations were audio-recorded and transcribed by a professional. Finally, two authors (EE, PR) also listened two to three recordings to validate whether interviews with primary care providers and nurses were capturing all elements of the practice. Analysis of actual practices was done in two main stages.
First, we documented current collaborative practices in each FMG, including nursing activities and challenges in the delivery of collaborative care to patients with CMDs and LTCs. We conducted (AG, JBH) a thematic analysis of interview transcriptions,34 with NVivo 12.0, and then schematized results to illustrate the collaborative care process and relationships between professional activities (see Additional File 1 for an example).
Second, we assessed the level of achievement and quality of professional activities involved in the CCM using two analysis tables built from the previously mentioned AIMS Center tool.26,27 These two tables respectively summarize the main activities of the CCM and the care manager’s activities (see additionnal files 2-3). Throughout this stage, tables were updated.
For each activity, the first author identified the professional who performed the activity and how it was done, the level of achievement, the resources and competencies required as well as any barriers to carrying out the activity. Triangulation of patient interviews, observations of nurse-patient encounter and documents from each FMG were integrated to enrich the assessment of each activity. The analysis was done in an iterative way and results obtained during the quality assessment were validated with the author who conducted a thematic analysis. Finally, preliminary results were presented to all members of the research team to validate results and to make decisions about how to deal with activities that were more difficult to assess. The team decided to ask for the opinion of local working groups regarding the assessment of these activities (step 4).
Discussion on lessons learned. Analysis of actual practices using qualitative data (verbatim from observations and interviews, schemas, and documents) provides a thorough understanding of current collaborative care and how professional activities are delivered, helping to identify potential activities that could be improved (targets for change). A limitation of this approach was the amount of data to collect and to manage in a short period of time. In addition, in the context of our study, we didn’t use quality indicators measuring collaborative mental health care to assess the actual performance as recommended in the Grol & Wensing model. Indeed, using quantitative data from the registry would have helped to better prioritize targets for change.
Assessing all professional activities involved in the CCM can be a time consuming challenge. The CCM includes many activities that are sometimes harder to evaluate because of their subjective nature or require specific and valid measurement tools adapted for such an analysis. For instance, patient’s engagement in their care was more difficult to assess. Consequently, to measure the level of engagement of patients it may be relevant to combine qualitative data sources with a valid questionnaire. For future research, an example of a valid tool that might help to better assess patients’ engagement in their care, is thePatient Assessment of Chronic Illness Care (PACIC). 35 This is a validated self-reported tool to measure the extent to which patients with LTCs receive care that is congruent with Wagner’s Chronic Care Model.35 However, to be more specific in assessing satisfaction with and engagement in both mental health and physical care, it might be useful to adapt the PACIC.
Furthermore, the Assessment of Chronic Illness Care (ACIC) might also be a useful tool to assess the overall level of achievement and quality of professional activities.36 The ACIC was built to help identify areas for improvement in the care of people with LTCs.36 The ACIC covers many principles and professional activities of the “Patient-Centered Integrated Behavioral Health Care Principles & Task Checklist” developed by the AIMS Center, but lacks specificity for mental health treatment and services. For further study, it might be interesting to adapt the ACIC by combining elements of this previous version that are specific to mental health care (e.g. psychiatric consultation or psychotherapy).