Process evaluations in neurological rehabilitation: a mixed-evidence systematic review and recommendations for future research

Objective: To systematically review how process evaluations are currently designed, what methodologies are used and how are they developed alongside or within neurological rehabilitation trials.

Methods: This mixed-methods systematic review had two evidence streams: stream I, studies reporting process evaluations alongside neurorehabilitation trials research and stream II, methodological guidance on process evaluation design and methodology. A search strategy was designed for each evidence stream. Data regarding process evaluation core concepts and design issues were extracted using a bespoke template. Evidence from both streams was analysed separately and then synthesised in a final overarching synthesis proposing a number of recommendations for future research.

Results: A total of 124 process evaluation studies, reporting on 106 interventions, were included in stream I evidence. 30 studies were included as stream II evidence. Synthesis 1 produced 9 themes, and synthesis 2 identified a total of 8 recommendations for process evaluation research. The overall synthesis resulted in 57 ‘synthesis recommendations’ about process evaluation methodology grouped into 9 research areas, including the use of theory, the investigation of context, intervention staff characteristics and the delivery of the trial intervention.

Conclusions: There remains no consensus regarding process evaluation terminology within the neurological rehabilitation field. There is a need for process evaluations to address the nature and influence of context over time. Process evaluations should clearly describe what intervention staff bring to a trial, including skills and experience prior to joining the research. Process evaluations should monitor intervention staff's learning effects and the possible impact that these may have on trial outcomes.

↵Tate DG. The state of rehabilitation research: art or science? Arch Phys Med Rehabil 2006;87:160–6. doi:10.1016/j.apmr.2005.11.013CrossRefPubMedGoogle Scholar
↵Hart T, Bagiella E. Design and implementation of clinical trials in rehabilitation research. Arch Phys Med Rehabil 2012;93:S117–26. doi:10.1016/j.apmr.2011.11.039CrossRefPubMedGoogle Scholar
↵Clark AM. What are the components of complex interventions in healthcare? Theorizing approaches to parts, powers and the whole intervention. Soc Sci Med 2013;93:185–93. doi:10.1016/j.socscimed.2012.03.035CrossRefPubMedGoogle Scholar
↵Townsend E. Enabling occupation: an occupational therapy perspective. Ottawa: Canadian Association of Occupational Therapists, 2012.Google Scholar
↵Medical Research Council. Developing and evaluation complex interventions: new guidance. London: Medical Research Council, 2008.Google Scholar
↵Robinson L, Francis J, James P, et al. Caring for carers of people with stroke: developing a complex intervention following the Medical Research Council framework. Clin Rehabil 2005;19:560–71. doi:10.1191/0269215505cr787oaAbstract/FREE Full TextGoogle Scholar
↵Tilling K, Coshall C, McKevitt C, et al. A family support organiser for stroke patients and their carers: a randomised controlled trial. Cerebrovasc Dis 2005;20:85–91. doi:10.1159/000086511CrossRefPubMedWeb of ScienceGoogle Scholar
↵Redfern J, McKevitt C, Wolfe CDA. Development of complex interventions in stroke care: a systematic review. Stroke 2006;37:2410–19. doi:10.1161/01.STR.0000237097.00342.a9Abstract/FREE Full TextGoogle Scholar
↵Oakley A, Strange V, Bonell C, et al. Process evaluation in randomised controlled trials of complex interventions. Health Serv Res 2006;332:413–16.Google Scholar
↵Moore G, Audrey S, Barker M, et al. Process evaluation of complex interventions: Medical Research Council guidance. London: MRC Population Health Science Research Network, 2014.Google Scholar
↵Steckler A, Linnan L. Process evaluation for public health interventions and research. San Francisco (CA): Jossey-Bass, 2002.Google Scholar
↵Hulscher M, Laurant MG, Grol RP. Process evaluation on quality improvement interventions. Qual Saf Health Care 2003;12:40–6. doi:10.1136/qhc.12.1.40Abstract/FREE Full TextGoogle Scholar
↵Bonell C, Oakley A, Hargreaves J, et al. Assessment of generalisability in trials of health interventions: suggested framework and systematic review. BMJ 2006;333:346–9. doi:10.1136/bmj.333.7563.346FREE Full TextGoogle Scholar
↵Byng R, Norman I, Redfern S, et al. Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. BMC Health Serv Res 2008;8:274–84. doi:10.1186/1472-6963-8-274CrossRefPubMedGoogle Scholar
↵Braun SM, van Haastregt JC, Beurskens AJ, et al. Feasibility of a mental practice intervention in stroke patients in nursing homes; a process evaluation. BMC Neurol 2010;10:74–83. doi:10.1186/1471-2377-10-74PubMedGoogle Scholar
↵Grant A, Treweek S, Dreischulte T, et al. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013;14:15–25. doi:10.1186/1745-6215-14-15CrossRefPubMedGoogle Scholar
↵Centre for Reviews and Dissemination. Systematic reviews: CRD's guidance for undertaking reviews in health care. York: University of York, 2009.Google Scholar
↵EPPI-Centre: EPPI-Centre methods for conducting systematic reviews. 2010. http://eppi.ioe.ac.uk/cms/LinkClick.aspx?fileticket=hQBu8y4uVwI%.3d&... Scholar
↵Harden A, Garcia J, Oliver S, et al. Applying systematic review methods to studies of people's views: an example of public health research. J Epidemiol Community Health 2004;58: 794–800. doi:10.1136/jech.2003.014829Abstract/FREE Full TextGoogle Scholar
↵Oliver S, Harden A, Rees R. An emerging framework for including different types of evidence in systematic reviews for public policy. Evaluation 2005;11:428–46. doi:10.1177/1356389005059383AbstractGoogle Scholar
↵Murta SG, Sanderson K, Oldenburg B. Process evaluation in occupational stress management programs; a systematic review. Am J Health Promot 2007;21:248–54. doi:10.4278/0890-1171-21.4.248Abstract/FREE Full TextGoogle Scholar
↵Cooper Robbins SC, Ward K, Skinner SR. School-based vaccination: a systematic review of process evaluations. Vaccine 2011;29:9588–99. doi:10.1016/j.vaccine.2011.10.033CrossRefPubMedWeb of ScienceGoogle Scholar
↵Yeary KH, Klos LA, Linnan L. The examination of process evaluation use in church-based health interventions: a systematic review. Health Promot Pract 2012;13:524–34. doi:10.1177/1524839910390358Abstract/FREE Full TextGoogle Scholar
↵Kavanagh J, Trouton A, Oakley A, et al. A systematic review of the evidence for incentive schemes to encourage positive health and other social behaviours in young people. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, 2006.Google Scholar
↵Popay J, Roberts H, Sowden A, et al. Guidance on the conduct of narrative synthesis in systematic reviews: Final report. Swindon: ESRC Methods Programme.Google Scholar
↵Gough D, Oliver S, Thomas JThomas J, Harden A, Newman M. Synthesis: combining results systematically and appropriately. In: Gough D, Oliver S, Thomas J, eds. An introduction to systematic reviews. London: Sage, 2012:179–226.Google Scholar
↵Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 2008;8:1–10. doi:10.1186/1471-2288-8-1CrossRefPubMedGoogle Scholar
↵McGinley JL, Martin C, Huxham FE, et al. Feasibility, safety, and compliance in a randomized controlled trial of physical therapy for Parkinson's disease. Parkinsons Dis 2012;2012:795294. doi:10.1155/2012/795294PubMedGoogle Scholar
↵Stephens S, Feldman BM, Bradley N, et al. Feasibility and effectiveness of an aerobic exercise program in children with fibromyalgia: results of a randomized controlled pilot trial. Arthritis Rheum 2008;59:1399–406. doi:10.1002/art.24115CrossRefPubMedWeb of ScienceGoogle Scholar
↵Alwin J, Krevers B, Johansson U, et al. Health economic and process evaluation of AT interventions for persons with dementia and their relatives—a suggested assessment model. Technol Disabil 2007;19:61–71.Google Scholar
↵Alwin J, Persson J, Krevers B. Perception and significance of an assistive technology intervention—the perspectives of relatives of persons with dementia. Disabil Rehabil 2013;35:1519–26. doi:10.3109/09638288.2012.743603Google Scholar
↵Khalil H, Quinn L, van Deursen R, et al. Adherence to use of a home-based exercise DVD in people with Huntington disease: participants’ perspectives. Phys Ther 2012;92:69–82. doi:10.2522/ptj.20100438Abstract/FREE Full TextGoogle Scholar
↵Khalil H, Quinn L, van Deursen R, et al. What effect does a structured home-based exercise programme have on people with Huntington's disease? A randomized, controlled pilot study. Clin Rehabil 2013;27:646–58. doi:10.1177/0269215512473762Abstract/FREE Full TextGoogle Scholar
↵Scianni A, Teixeira-Salmela LF, Ada L. Challenges in recruitment, attendance and adherence of acute stroke survivors to a randomized trial in Brazil: a feasibility study. Rev Bras Fisioter 2012;16:40–5.PubMedGoogle Scholar
↵Scobbie L, McLean D, Dixon D, et al. Implementing a framework for goal setting in community based stroke rehabilitation: a process evaluation. BMC Health Serv Res 2013;13:190–203. doi:10.1186/1472-6963-13-190Google Scholar
↵Döpp CM, Graff MJ, Teerenstra S, et al. A new combined strategy to implement a community occupational therapy intervention: designing a cluster randomized controlled trial. BMC Geriatr 2011;11:13. doi:10.1186/1471-2318-11-13CrossRefPubMedGoogle Scholar
↵King G, Tucker MA, Alambets P, et al. The evaluation of functional, school-based therapy services for children with special needs: a feasibility study. Phys Occup Ther Pediatr 1998;18:1–27.Google Scholar
↵Voigt-Radloff S, Graff M, Leonhart R, et al. WHEDA study: effectiveness of occupational therapy at home for older people with dementia and their caregivers—the design of a pragmatic randomised controlled trial evaluating a Dutch programme in seven German centres. BMC Geriatr 2009;9:44–59. doi:10.1186/1471-2318-9-44PubMedGoogle Scholar
↵Voigt-Radloff S, Graff M, Leonhart R, et al. Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT. BMJ Open 2011;1:e000094doi:10.1136/bmjopen-2011-000094.Abstract/FREE Full TextGoogle Scholar
↵Chung JC. An intergenerational reminiscence programme for older adults with early dementia and youth volunteers: values and challenges. Scand J Caring Sci 2009;23:259–64. doi:10.1111/j.1471-6712.2008.00615.xCrossRefPubMedWeb of ScienceGoogle Scholar
↵Morris DM, Taub E, Macrina DM, et al. A method for standardizing procedures in rehabilitation: use in the extremity constraint induced therapy evaluation multisite randomized controlled trial. Arch Phys Med Rehabil 2009;90:663–8. doi:10.1016/j.apmr.2008.09.576CrossRefPubMedGoogle Scholar
↵Østensjø S, Oien I, Fallang B. Goal-oriented rehabilitation of pre-schoolers with cerebral palsy—a multi-case study of combined use of the Canadian Occupational Performance Measure (COPM) and the Goal Attainment Scaling (GAS). Dev Neurorehabil 2008;11:252–9. doi:10.1080/17518420802525500CrossRefPubMedGoogle Scholar
↵Mayo NE, MacKay-Lyons MJ, Scott SC, et al. A randomized trial of two home-based exercise programmes to improve functional walking post-stroke. Clin Rehabil 2013;27:659–71. doi:10.1177/0269215513476312Abstract/FREE Full TextGoogle Scholar
↵Leuty V, Boger J, Young L, et al. Engaging older adults with dementia in creative occupations using artificially intelligent assistive technology. Assist Technol 2013;25:72–9. doi:10.1080/10400435.2012.715113CrossRefPubMedGoogle Scholar
↵Macht M, Gerlich C, Ellgring H, et al. Patient education in Parkinson's disease: formative evaluation of a standardized programme in seven European countries. Patient Educ Couns 2007;65:245–52. doi:10.1016/j.pec.2006.08.005CrossRefPubMedWeb of ScienceGoogle Scholar
↵Li F, Harmer P, Fisher KJ, et al. Tai chi-based exercise for older adults with Parkinson's disease: a pilot-program evaluation. J Aging Phys Act 2007;15:139–51. doi:10.1123/japa.15.2.139PubMedWeb of ScienceGoogle Scholar
↵Taylor NF, Dodd KJ, McBurney H, et al. Factors influencing adherence to a home-based strength-training programme for young people with cerebral palsy. Physiotherapy 2004;90:57–63. doi:10.1016/j.physio.2003.09.001CrossRefGoogle Scholar
↵Resnick B, Michael K, Shaughnessy M, et al. Exercise intervention research in stroke: optimizing outcomes through treatment fidelity. Top Stroke Rehabil 2011;18(Suppl 1):611–19. doi:10.1310/tsr18s01-611CrossRefPubMedGoogle Scholar
↵Van't Leven N, Graff MJ, Kaijen M, et al. Barriers to and facilitators for the use of an evidence-based occupational therapy guideline for older people with dementia and their carers. Int J Geriatr Psychiatry 2012;27:742–8. doi:10.1002/gps.2782CrossRefPubMedGoogle Scholar
↵Döpp CM, Graff MJ, Rikkert MG, et al. Determinants for the effectiveness of implementing an occupational therapy intervention in routine dementia care. Implement Sci 2013;8:131. doi:10.1186/1748-5908-8-131Google Scholar
↵MacNeil Vroomen J, Van Mierlo LD, Van de Ven PM, et al. Comparing Dutch case management care models for people with dementia and their caregivers: the design of the COMPAS study. BMC Health Serv Res 2012;12:132. doi:10.1186/1472-6963-12-132PubMedGoogle Scholar
↵Whiting DL, Simpson GK, McLeod HJ, et al. Acceptance and commitment therapy (ACT) for psychological adjustment after traumatic brain injury: reporting the protocol for a randomised controlled trial. Brain Impair 2012;13:360–76. doi:10.1017/BrImp.2012.28Google Scholar
↵Borrelli B, Sepinwall D, Ernst D, et al. A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. J Consult Clin Psychol 2005;73:852–60. doi:10.1037/0022-006X.73.5.852CrossRefPubMedWeb of ScienceGoogle Scholar
↵Vluggen TP, van Haastregt JC, Verbunt JA, et al. Multidisciplinary transmural rehabilitation for older persons with a stroke: the design of a randomised controlled trial. BMC Neurol 2012;12:164. doi:10.1186/1471-2377-12-164CrossRefPubMedGoogle Scholar
↵Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 2005;6:134–47. doi:10.1177/1524839904273387Abstract/FREE Full TextGoogle Scholar
↵Letts L, Dunal L. Tackling evaluation: applying a programme logic model to community rehabilitation for adults with brain injury. Can J Occup Ther 1995;62:268–77. doi:10.1177/000841749506200506Abstract/FREE Full TextGoogle Scholar
↵Carroll C, Patterson M, Wood S, et al. A conceptual framework for implementation fidelity. Implement Sci 2007;2:40–9. doi:10.1186/1748-5908-2-40CrossRefPubMedGoogle Scholar
↵Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: are implementation effects out of control? Clin Psychol Rev 1998;18:23–45. doi:10.1016/S0272-7358(97)00043-3CrossRefPubMedWeb of ScienceGoogle Scholar
↵Dusenbury L, Brannigan R, Falco M, et al. A review of research on fidelity of implementation: implications for drug abuse preventions in school settings. Health Educ Res 2003;18:237–56. doi:10.1093/her/18.2.237Abstract/FREE Full TextGoogle Scholar
↵Bate P. Context is everything. In: Perspectives on context. Original research. The Health Foundation—inspiring improvement. London: The Health Foundation, 2014:1–30.Google Scholar
↵Dixon-Woods M. The problem of context in quality improvement. In: Perspectives on context. Original research. The Health Foundation—inspiring improvement. London: The Health Foundation, 2014:87–101.Google Scholar
↵Campbell NC, Murray E, Darbyshire J, et al. Designing and evaluating complex interventions to improve health care. BMJ 2007;334:455–9. doi:10.1136/bmj.39108.379965.BEFREE Full TextGoogle Scholar
↵Elford J, Sherr L, Bolding G, et al. Peer-led HIV prevention among gay men in London: process evaluation. AIDS Care 2002;14:351–60. doi:10.1080/09540120220123739CrossRefPubMedWeb of ScienceGoogle Scholar
↵O′Cathain A, Thomas KJ, Drabble SJ, et al. Maximizing the value of combining qualitative research and randomised controlled trials in health research: the QUAlitative Research in Trials (QUART) study—a mixed methods study. Health Technol Assess 2014;18:1–197.CrossRefPubMedGoogle Scholar
↵Dumas JE, Lynch AM, Laughlin JE, et al. Promoting intervention fidelity. Conceptual issues, methods, and preliminary results from the Early Alliance prevention trial. Am J Prev Med 2001;20(1 Suppl):38–47. doi:10.1016/S0749-3797(00)00272-5CrossRefPubMedWeb of ScienceGoogle Scholar
↵Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurs Res 2004;53:63–6. doi:10.1097/00006199-200401000-00010CrossRefPubMedWeb of ScienceGoogle Scholar
↵Horner S, Rew L, Torres R. Enhancing intervention fidelity: a means of strengthening study impact. J Spec Pediatr Nurs 2006;11:80–9. doi:10.1111/j.1744-6155.2006.00050.xCrossRefPubMedGoogle Scholar
↵Ramsay CR, Wallace SA, Garthwaite PH, et al. Assessing the learning curve effect in health technologies. Lessons from the nonclinical literature. Int J Technol Assess Health Care 2002;18:1–10.PubMedWeb of ScienceGoogle Scholar
↵Cook JA, Ramsay CR, Fayers P. Statistical evaluation of learning curve effects in surgical trials. Clin Trials 2004;1:421–7. doi:10.1191/1740774504cn042oaAbstract/FREE Full TextGoogle Scholar
↵Masterson-Algar P, Burton CR, Rycroft-Malone J, et al. Towards a programme theory for fidelity in the evaluation of complex interventions. J Eval Clin Pract 2014;20:445–52. doi:10.1111/jep.12174CrossRefPubMedGoogle Scholar
↵Ramsay CR, Grant AM, Wallace SA, et al. Assessment of the learning curve in health technologies. A systematic review. Int J Technol Assess Health Care 2000;16:1095–108. doi:10.1017/S0266462300103149CrossRefPubMedWeb of ScienceGoogle Scholar
↵Castro FG, Barrera M, Martinez CR. The cultural adaptation of prevention interventions: resolving tensions between fidelity and fit. Prev Sci 2004;5:41–5. doi:10.1023/B:PREV.0000013980.12412.cdCrossRefPubMedWeb of ScienceGoogle Scholar
↵Morrison DM, Hoppe MJ, Gillmore MR. Replicating an intervention: the tension between fidelity and adaptation. AIDS Educ Prev 2009;21:128–40. doi:10.1521/aeap.2009.21.2.128CrossRefPubMedGoogle Scholar
↵Harshbarger C, Simmons G, Coelho H, et al. An empirical assessment of implementation, adaptation, and tailoring: the evaluation of CDC's National Diffusion of VOICES/VOCES. AIDS Educ Prev 2006;18(4 Suppl A):184–97. doi:10.1521/aeap.2006.18.supp.184CrossRefPubMedWeb of ScienceGoogle Scholar
↵Hawe P, Shiell A, Riley T. In response to Spillane V., Byrne M. C., Byrne M., Leathem C. S., O'Malley M. & Cupples M. E. (2007) Monitoring treatment fidelity in a randomized trial of a complex intervention. Journal of Advanced Nursing 60(3), 343–352. Important considerations for standardizing complex interventions. J Adv Nurs 2008;62:267. doi:10.1111/j.1365-2648.2008.04686.xPubMedGoogle Scholar
↵Song MK, Happ MB, Sandelowski M. Development of a tool to assess fidelity to a psycho-educational intervention. J Adv Nurs 2010;66:673–82. doi:10.1111/j.1365-2648.2009.05216.xCrossRefPubMedGoogle Scholar
↵Schulz KF, Altman DG, Moher D, et al., CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. BMJ 2010;340:c332. doi:10.1136/bmj.c332FREE Full TextGoogle Scholar
↵Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348:g1687. doi:10.1136/bmj.g1687Abstract/FREE Full TextGoogle Scholar

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/