The generation of consensus guidelines for carrying out process evaluations in rehabilitation research

Background: Although in recent years there has been a strong increase in published research on theories (e.g. realist evaluation, normalization process theory) driving and guiding process evaluations of complex interventions, there is limited guidance to help rehabilitation researchers design and carry out process evaluations. This can lead to the risk of process evaluations being unsystematic. This paper reports on the development of new consensus guidelines that address the specific challenges of conducting process evaluations alongside clinical trials of rehabilitation interventions.

Methods: A formal consensus process was carried out based on a modified nominal group technique, which comprised two phases. Phase I was informed by the findings of a systematic review, and included a nominal group meeting with an expert panel of participants to rate and discuss the proposed statements. Phase II was an in depth semi-structured telephone interviews with expert panel participants in order to further discuss the structure and contents of the revised guidelines. Frequency of rating responses to each statement was calculated and thematic analysis was carried out on all qualitative data.

Results: The guidelines for carrying out process evaluations within complex intervention rehabilitation research were produced by combining findings from Phase I and Phase II. The consensus guidelines include recommendations that are grouped in seven sections. These sections are theoretical work, design and methods, context, recruitment and retention, intervention staff, delivery of the intervention and results. These sections represent different aspects or stages of the evaluation process.

Conclusion: The consensus guidelines here presented can play a role at assisting rehabilitation researchers at the time of designing and conducting process evaluations alongside trials of complex interventions. The guidelines break new ground in terms of concepts and theory and works towards a consensus in regards to how rehabilitation researchers should go about carrying out process evaluations and how this evaluation should be linked into the proposed trials. These guidelines may be used, adapted and tested by rehabilitation researchers depending on the research stage or study design (e.g. feasibility trial, pilot trial, etc.).

1. Robinson L, Francis J, James P, Tindle N, Greenwell K, Rodgers H. Caring for
carers of people with stroke: developing a complex intervention following
the Medical Research Council framework. Clin Rehabil. 2005;19:560–71.
2. Redfern J, McKevitt C, Wolfe CDA. Development of complex interventions in
stroke care: a systematic review. Stroke. 2006;37:2410–9.
3. Medical Research Council. Developing and evaluation complex
interventions: new guidance. London: Medical Research Council; 2008.
4. Horner S, Rew L, Torres R. Enhancing intervention fidelity: a means of
strengthening study impact. J Spec Pediatr Nurs. 2006;11(2):80–9.
5. Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurse Res. 2004;
53(1):63–6.
6. Kerns SEU, Prinz RJ. Critical issues in the prevention of violence-related
behavior in youth. Clin Child Fam Psychol Rev. 2002;5:133–60.
7. Hart T, Bagiella E. Design and implementation of clinical trials in
rehabilitation research. Arch Physic Med Rehab. 2012;93:S117–26.
8. Townsend E. Enabling occupation: an occupational therapy perspective.
Ottawa: Canadian Association of Occupational Therapists; 2002.
9. Dejong G, Horn SD, Gassaway JA, Slavin MD, Dijkers MP. Toward a
taxonomy of rehabilitation interventions: using an inductive approach to
examine the “black box” of rehabilitation. Arch Physic Med Rehab. 2004;85:
678–86.
10. WHO. World report on disability. Malta: World Health Organization. 2011.
https://www.who.int/disabilities/world_report/2011/report.pdf. Accessed 12
Dec 2017.
11. Richards DA. The complex interventions framework. In: Richards DA,
Hallberg IR, editors. Complex interventions in health. An overview of
research methods. Abingdon, Oxon: Routledge; 2015. p. 1–15.
12. Masterson-Algar P, Burton CR, Rycroft-Malone J, Sackley C, Walker M.
Towards a programme theory for fidelity in the evaluation of complex
interventions. J. Eval. Clin. Pract. 2014;20(4):445–52.
13. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for
cluster-randomised trials of complex interventions: a proposed framework
for design and reporting. Trials. 2013;14:15–25.
14. Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process
evaluation in complex public health intervention studies: the need for
guidance. J Epidemiol Community. 2014;68(6):585–6.
15. Moore G, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process
evaluation of complex interventions: Medical Research Council guidance.
London: MRC Population Health Science Research Network; 2014.
16. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process
evaluation of complex interventions: Medical Research Council guidance.
BMJ. 2015;350:h1258.
17. Graham I, Logan J, Harrison M, Straus SE, Tetroe J, Caswell W, et al. Lost in
knowledge translation: time for a map? The Journal of Continuing
Education in the Health Professions. 2006;26:13–24.
18. Harrison M, Legare F, Graham I, Fervers B. Adapting clinical practice
guidelines to local context and assessing barriers to their use. CMAJ. 2010;
182(2):78–84.
19. Rycroft-Malone J. Formal consensus: the development of a national clinical
guideline. Qual Health Care. 2001;10:238–44.
20. Grimshaw J, Freemantle N, Wallace S, et al. Developing and implementing
clinical practice guidelines. Qual Health Care. 1995;4:55–64.
21. Grimshaw JM, Russell IT. Achieving health gains through clinical guidelines:
I. developing scientifically valid guidelines. Qual Assurance Health Care.
1993;2:243–8.
22. Francke AL, Smit MC, de Veer AJ, Mistiaen P. Factors influencing
implementation of clinical guidelines for healthcare professionals: a
systematic meta-review. BMD Med. Inform. Decis. Mak. 2008;8:38–49.
23. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi
techniques. Int J Clin Pharm. 2016. https://doi.org/10.1007/s11096-016-0257-x.
24. Potter M, Gordon S, Hamer P. The nominal group technique: a useful
consensus methodology in physiotherapy research. N Z J Physiother. 2004;
32(3):126–30.
25. Jones J, Hunter D. Consensus methods for medical and health services
research. BMJ. 1995;311:376–80.
26. Bloor M, Frankland J, Thomas M, Robson K. Focus groups in social research.
London: Sage Publications Ltd; 2001.
27. Masterson-Algar P, Burton CR, Rycroft-Malone J. A mixed-evidence synthesis review
of process evaluations in neurological rehabilitation: recommendations for future
research. BMJ Open. 2016. https://doi.org/10.1136/bmjopen-2016-013002.
28. Trickey H, Harvey I, Wilcox G. Formal consensus and consultation: a
qualitative method for development of a guideline for dementia. Qual
Health Care. 1998;7:192–9.
29. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol.
2006;3(2):77–101.
30. Byng R, Norman I, Redfern S, Jones R. Exposing the key functions of a
complex intervention for shared care in mental health: case study of a
process evaluation. BMC Health Serv Res. 2008;8:274–84.
31. Mann C, Shaw A, Guthrie B, Wye L, Man M, Hollinghurst S, et al. Protocol for
a process evaluation of a cluster randomised controlled trial to improve
management of multimorbidity in general practice: the 3D study. BMJ
Open. 2016. https://doi.org/10.1136/bmjopen-2016-011260
32. Masterson-Algar P, Burton C, Brady MC, Nicoll A, Clarke CE, Rick C, et al.
Protocol for the process evaluation of the PD COMM trial: assessing the
effectiveness of two types of SLT for people with Parkinson’s disease. Trials.
2017;18:397–405.
33. Clarke DJ, Godfrey M, Hawkins R, Sadler E, et al. Implementing a training
intervention to support caregivers after stroke: a process evaluation
examining the initiation and embedding of programme change. Implement
Sci. 2013. https://doi.org/10.1186/1748-5908-8-96.
34. Scianni A, Teixeira-Salmela LF, Ada L. Challenges in recruitment, attendance
and adherence of acute stroke survivors to a randomized trial in Brazil: a
feasibility study. Rev Bras Fisioter. 2012;16:40–455.
35. Chung JC. An intergenerational reminiscence programme for older adults
with early dementia and youth volunteers: values and challenges. Scand J
Caring Sci. 2009;23:259–64.
36. Mayo NE, MacKay-Lyons MJ, Scott SC, et al. A randomized trial of two
home-based exercise programmes to improve functional walking poststroke. Clin Rehabil. 2013;27:659–71.
37. Letts L, Dunal L. Tackling evaluation: applying a programme logic model to
community rehabilitation for adults with brain injury. Can J Occup Ther.
1995;62:268–77.
38. Cohn S, Clinch M, Bunn C, Stronge P. Entangled complexity: why complex
interventions are just not complicated enough. J Health Serv Res Policy.
2013;18(1):40–3.
39. Wolf S, Winstein C, Miller JP, Thompson PA, Taub E, Uswatte G, Morris D,
Blanton S, Nichols-Larsen D. The EXCITE trial: retention of improved upper
extremity function among stroke survivors receiving CI movement therapy.
Lancet Neurol. 2008;7(1):33–40.
40. Schulz KF, Altman DG, Moher D, CONSORT Group, et al. CONSORT 2010
statement: updated guidelines for reporting parallel group randomized
trials. BMJ. 2010;340:c332.
41. Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions:
template for intervention description and replication (TIDieR) checklist and
guide. BMJ. 2014;348:g1687.
42. Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Process evaluation in
randomised controlled trials of complex interventions. BMJ Health Serv Res.
2006;332:413–6.
43. Pawson R, Tilley N. Realistic Evaluation. London: Sage Publications; 1997.
44. Morrison DDM. Replicating an intervention: the tension between fidelity
and adaptation. AIDS Educ Prev. 2009;21(2):128–40.
45. Song M, Happ MB, Sandelowski M. Development of a tool to assess fidelity
to a psycho-educational intervention. J Adv Nurs. 2010;66(3):673–82.
46. Elliot D, Mihalic S. Issues in disseminating and replicating effective
prevention programs. Prev. Sci. 2004;5:47–53.
47. Mihalic S. The importance of implementation fidelity. Emot & Behav Disord
in Youth. 2004;4:83–105.
48. O’Cathain A, Murphy E, Nicholl J. Multidisciplinary, interdisciplinary, or
dysfunctional? Team working in mixed-methods research. Qual Health Res.
2008;18:1574–85.
49. Audrey S, Holliday J, Parry-Langdon N, Campbell R. Meeting the challenges
of implementing process evaluation within randomized controlled trials: the
example of ASSIST (a stop smoking in schools trial). Health Educ Res. 2006;
21:366–77