Multi-professional image interpretation: performance in preliminary clinical evaluation of appendicular radiographs

Introduction: The study evaluated the performance of a cohort of healthcare professionals in image interpretation of appendicular radiographic examinations following a preliminary clinical evaluation (PCE) course.

Methods: A systematic review and meta-analysis of image interpretation performance by non-medical staff was undertaken. Thirty three participants (seven radiographers, nineteen nurses, five paramedics, and two physiotherapists) completed an image based Objective Structured Examination (OSE) following a short course on PCE of appendicular radiographs. The case bank comprised 25 appendicular radiographic examinations. Prevalence of abnormal examinations approximated 52% (traumatic conditions). The OSE test scores were analysed with Alternative Free-response Receiver Operating Characteristic (AFROC) calculation of Area Under the Curve (AUC), sensitivity, specificity, and Cohen’s Kappa (k) for multi-reader agreement. Professional subgroup results were compared with Analysis of Variance (ANOVA), and compared to the literature review.

Results: The participant subgroup results for radiographers were sensitivity 79.7%, specificity 92.9%, k=72; AUC 86.5. Nurses sensitivity 76.2%, specificity 80.4%, k=56; AUC 78.3. Paramedics sensitivity 80.3%, specificity 81.6%, k=61; AUC 81.5. Physiotherapists sensitivity 90.4%, specificity 91.6%, k=82; AUC 91.8. ANOVA Samples scores (f=1.5; p=0.23), sensitivity (f=0.5; p=0.65), specificity (f=2.8; p=0.05). The comparison of the radiographers and nurses results to the published literature were encouraging (SROC AUC 93.1 and 91.2, respectively). The paramedics and physiotherapists results could not be compared to published standards or abilities on the task due to a lack of literature available.

Conclusions: The study results of image interpretation and commentary of appendicular radiographs in an academic environment by this multi-professional sample were to a high standard, further work is recommended on a larger sample.

1. Moody, I. 50 years of History. London: The Society of Radiographers; 1970.
2. Society and College of Radiographers. Preliminary Clinical Evaluation and Clinical Reporting by Radiographers: Policy and Practice Guidance. London: Society and College of Radiographers; 2013.
3. Woznitza N, Piper K, Rowe S, West C. Optimizing patient care in radiology through team-working: a case study from the United Kingdom. Radiography 2014; 20(3):258-63.
4. Snaith B, Hardy M. Radiographer abnormality detection schemes in the trauma environment—An assessment of current practice. Radiography 2008; 14(4):277-81.
5. Snaith B, Hardy M, Lewis EF. Reducing image interpretation errors–Do communication strategies undermine this?. Radiography 2014; 20(3):230-4.
6. Lancaster A, Hardy M. An investigation into the opportunities and barriers to participation in a radiographer comment scheme, in a multi-centre NHS trust. Radiography 2012; 18(2):105-8.
7. The Health and Care Professions Council. Standards of Proficiency: Radiographers. London: HCPC; 2013.
8. Stevens BJ, Thompson JD. The impact of focused training on abnormality detection and provision of accurate preliminary clinical evaluation in newly qualified radiographers. Radiography. 2018 Feb 1;24(1):47-51.
9. Hazell L, Motto J, Chipeya L. The Influence of Image Interpretation Training on the Accuracy of Abnormality Detection and Written Comments on Musculoskeletal Radiographs by South African Radiographers, Journal of Medical Imaging and Radiation Sciences 2015, 46, 3, (302).
10. McConnell J, Devaney C, Gordon M. Queensland radiographer clinical descriptions of adult appendicular musculo-skeletal trauma following a condensed education programme, Radiography 2012, 19, pp. 48-55.
11. McConnell J, Devaney C, Gordon M, Goodwin M, Strahan R, Baird M. The impact of a pilot education programme on Queensland radiographer abnormality description of adult appendicular musculo-skeletal trauma. Radiography 2013, 18, pp. 184-190.
12. Hardy M, Culpan G. Accident and emergency radiography: a comparison of radiographer commenting and ‘red dotting’. Radiography 2007, 13, pp. 65-71.
13. Meek S, Kendall J, Porter J, Freij R. Can accident and emergency nurse practitioners interpret radiographs? A multicentre study. J Accid Emerg Med 1998, 15, pp. 105-107.
14. Crane J, Delany C. Physiotherapists in emergency departments: responsibilities, accountability and education. Physiotherapy 2013, 99, pp. 95-100.
15. Overton-Brown P,Anthony D. Towards a partnership in care: nurses' and doctors' interpretation of extremity trauma radiology. J Adv Nurs, 27 (1998), pp. 890-896.
16. Piper KJ, Paterson A. Initial image interpretation of appendicular skeletal radiographs: a comparison between nurses and radiographers. Radiography 2009, 15, pp. 40-48.
17. Coleman L, Piper K. Radiographic interpretation of the appendicular skeleton: a comparison between casualty officers, nurse practitioners and radiographers. Radiography, 15 (2009), pp. 196-202.
18. Mordy M, Barrett C. Interpreting Trauma Radiographs Journal of Advanced Nursing 2003, October, vol 44, 81-87.
19. Hardy M, Barret C. Interpretation of trauma radiographs by radiographers and nurse practitioners in the UK: a comparative study. The British Journal of Radiology 2004, 77, 657-661.
20. Health Education England. Multi-professional framework for England. Health Education England 2017 Leeds.
21. The College of Paramedics and Health Education England. Digital Career Framework 2017. The College of Paramedics 2017, Bridgwater.
22. The Royal College of Emergency Medicine. Emergency Care Advanced Clinical Practitioner Curriculum and Assessment Adult Only/Adult and Paediatric / Paediatric. Version 2.0. The Royal College of Emergency Medicine 2017 London.
23. Chudleigh J. Nurse requested x –rays in A&E Department. Emergency Nurse 2004, vol 11, 32-35.
24. Ward W. Key Issues in Nurse requesting x-rays Emergency Nurse 1999. Feb, Vol 6, No 9.
25. Nursing and Midwifery Council. The Code: Standards of conduct performance and ethics for nurses and midwives (2008). London: Nursing and Midwifery Council.
26. Nursing and Midwifery Council. Record keeping: Guidance for nurses and midwives (2009) London: Nursing and Midwifery Council.
27. Making a Difference (1999). Strengthening the Nursing and Midwifery and Health Visiting Contribution to Health and health care. D.O.H London.
28. Chartered Society of Physiotherapy. Advanced practice in physiotherapy: Understanding the contribution of advanced practice in physiotherapy to transforming lives, maximising independence and empowering populations. Chartered Society of Physiotherapy 2016, London.
29. McPherson P, Kersten P, George S, Lattimer V, Ellis B, Breton A, et al. Extended roles for allied health professionals in the NHS: National Co-ordinating Centre for NHS Service Delivery and Organisation; 2004.
30. Taylor D, Bennett D, Cameron P. A paradigm shift in the nature of care provision in emergency departments. Emergency Medicine Journal 2004, 21, pp. 681-684.
31. Anaf S, Sheppard L. Describing physiotherapy interventions in the emergency department setting: an observational pilot study. Accid Emerg Nurs 2007, 15, pp. 34-39.
32. Kempson S. Physiotherapy in an accident and emergency department. Accid Emerg Nurs 1996, 4, pp. 198-202.
33. Anaf S, Sheppard LA. Physiotherapy as a clinical service in emergency departments: a narrative review. Physiotherapy. 2007 Dec 1;93(4):243-52.
34. The Care Quality Commission. The state of health care and adult social care in England 2016/17 (Presented to Parliament pursuant to section 83(4)(a) of the Health and Social Care Act 2008). Newcastle upon Tyne: House of Commons; 2017.
35. Department of Health. Improving outcomes: A Strategy for Cancer. London: HMSO; 2011.
36. The Care Quality Commission. Radiology review: A national review of radiology reporting within the NHS in England. The Care Quality Commission, Newcastle upon Tyne 2018
37. NHS England. Next steps on the NHS Five Year Forward View (Gateway number 06669). Leeds: NHS England; 2017.
38. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of internal medicine. 2011 Oct 18;155(8):529-36.
39. Zamora J, Abraira V, Muriel A, Khan KS, Coomarasamy A. Meta-DiSc: a software for meta-analysis of test accuracy data. BMC Medical Research Methodology 2006, 6:31. Available from ftp://ftp.hrc.es/pub/programas/metadisc/Metadisc_update.htm 02.11.2016
40. Brealey S. Measuring the effects of image interpretation: an evaluative framework. Clinical Radiol 2001; 56.341-347.
41. Brealey S, Scally AJ. Methodological approaches to evaluating the practice of radiographers’ interpretation of images, A Review. Radiography 2008, 14 (1): 46-54.
42. Piper K, Ryan C, Paterson A. The Implementation of a Radiographic Reporting Service, for trauma examinations of the skeletal system, in 4 National Health Service Trusts. Project Report SPGS 438. South Thames Regional Office (NHSE); 1999.
43. Robinson PJ, Wilson D, Coral A, Murphy A, Verow P. Variation between experienced observers in the interpretation of accident and emergency radiographs. The British Journal of Radiology 1999, 72(856): 323-330.
44. IMAGE Information Systems Ltd. K-PACS (Version 1.6.0). Germany, 2008.
45. Royal College of Radiologists. Picture archiving and communication systems (PACS) and guidelines on diagnostic display devices. 2012. London, UK.
46. (Anonymised for peer review manuscript).
47. Kundel HL, Polansky M. Measurement of observer agreement. Radiology. 2003 Aug;228(2):303-8.
48. Thompson JD, Manning DJ, Hogg P. Analysing data from observer studies in medical imaging research: an introductory guide to free-response techniques. Radiography. 2014 Nov 1;20(4):295-9.
49. Berman L, de Lacey G, Twomey E, Twomey B, Welch T, Eban R. Reducing errors in the accident department: a simple method using radiographers. Br Med J (Clin Res Ed). 1985 Feb 9;290(6466):421-2.
50. Bowman S. Introducing an abnormality detection system by radiographers into an accident and emergency department: an investigation into radiographers’ concerns about the introduction of such a system. Res Radiography. 1991;1:2-0.
51. Renwick IG, Butt WP, Steele B. How well can radiographers triage x ray films in accident and emergency departments?. BMJ. 1991 Mar 9;302(6776):568-9.
52. Loughran CF. Reporting of fracture radiographs by radiographers: the impact of a training programme. Br J Radiol 1994; 67: 945e50
53. Mackay SJ. The impact of a short course of study on the performance of radiographers when highlighting fractures on trauma radiographs: “The Red Dot System”. The British journal of radiology. 2006 Jun;79(942):468-72.
54. Morrison ID, Hendry R, Fell C, Stothard K. An audit of radiographers' accuracy in recording abnormalities on causality radiographs using the red dot protocol. Clinical Radiology. 1999;12(54):862-3.
55. Ross C. The effectiveness of a short course of study on the ability of radiographers to highlight and describe abnormalities on skeletal accident and emergency films. MSc Thesis. South Bank University; 2002.
56. McConnell JR, Webster AJ. Improving radiographer highlighting of trauma films in the accident and emergency department with a short course of study--an evaluation. The British journal of radiology. 2000 Jun;73(870):608-12.
57. Hargreaves J, Mackay S. The accuracy of the red dot system: can it improve with training? Radiography (2003) 9, 283-289.
58. Hazell, L.J., 2012. A study to assess the ability of radiographers to apply pattern recognition criteria and interpret radiographs (Doctoral dissertation, University of Johannesburg) 2012.
59. Williams I, Baird M, Pearce B, Schneider M. Improvement of radiographer commenting accuracy of the appendicular skeleton following a short course in plain radiography image interpretation: A pilot study. Journal of medical radiation sciences. 2018 Oct 9.
60. Tay YX, Wright C. Image interpretation: Experiences from a Singapore in-house education program. Radiography. 2018 May 4.
61. Derksen RJ, Bakker FC, Heilbron EA, Geervliet PC, Spaans IM, de Lange-de Klerk ESM, Veenings B, Patka P, Henk JTM. Diagnostic accuracy of lower extremity X-ray interpretation by ‘specialized’ emergency nurses. Eur J Emerg Med, 13 (2006), pp. 3-8.
62. Snaith B, Hardy M. Emergency department image interpretation accuracy: The influence of immediate reporting by radiology. International emergency nursing. 2014 Apr 1;22(2):63-8.
63. Khiroya T, Taylor S, Remedios D, de Lacey G.Trauma radiology: the accuracy of radiographers' assessment of appendicular trauma Br J Radiol, 70 (Suppl) (1997), p. 123.
64. Orames C. Emergency department X-ray diagnosis-how do radiographers compare?. Radiographer: The Official Journal of the Australian Institute of Radiography, The. 1997 Apr;44(1):52-55.
65. Hall R, Egan I, Kleeman SJ. The red dot system: the outback experience. Radiographer: The Official Journal of the Australian Institute of Radiography, The. 1999 Aug;46(2):83.
66. Remedios D, Ridley N, Taylor S, de Lacey G. Trauma radiology: extending the Red Dot system. Br J Radiol 1998; 71(S):60.
67. Smith T, Younger C. Accident and emergency radiological interpretation using the radiographer opinion form (ROF). Radiographer 2002; 49: 27–31.
68. Rolph H. Red dot radiographers: assessment of their ability to detect abnormalities and write a short quality description. BSc Thesis. Glasgow Caledonian University, 2003.
69. Mc Entee MF, Dunnion S. A FROC analysis of radiographers performance in identification of distal radial fractures. European Journal of Radiography. 2009 Sep 1;1(3):90-4.
70. Rutledge A, McEntee MF, Rainford L, O'Grady M, McCarthy K, Butler ML. The impact of clinical indications on visual search behaviour in skeletal radiographs. InMedical Imaging 2011: Image Perception, Observer Performance, and Technology Assessment 2011 Mar 3 (Vol. 7966, p. 79660Y). International Society for Optics and Photonics.
71. Brown N, Leschke P. Evaluating the true clinical utility of the red dot system in radiograph interpretation. Journal of medical imaging and radiation oncology. 2012 Oct;56(5):510-3.
72. Hlongwane ST, Pitcher RD. Accuracy of after-hour 'red dot' trauma radiograph triage by radiographers in a South African regional hospital. SAMJ: South African Medical Journal. 2013 Jan;103(9):638-40.
73. Wright C. RadBench: benchmarking image interpretation performance. Liverpool: UKRC; 2012. June, http://shura.shu.ac.uk/id/eprint/8432.
74. Yallappa S, Clarke D, Minford JE. Accuracy of red dot in clinical practice. Clinical Radiology. 2012 Sep 1;67:S20.
75. du Plessis J, Pitcher R. Towards task shifting? A comparison of the accuracy of acute trauma-radiograph reporting by medical officers and senior radiographers in an African hospital. Pan African Medical Journal. 2015;21(1).
76. Kranz R, Cosson P. Anatomical and/or pathological predictors for the “incorrect” classification of red dot markers on wrist radiographs taken following trauma. The British journal of radiology. 2015 Jan 19;88(1046):20140503.
77. Higgins S, Wright C. Traffic light: an alternative approach to abnormality signalling. Liverpool: UKRC; 2016. https://core.ac.uk/display/77055324.
78. Neep, M. Pozzias, E. To alert or not to alert: the rise of the frontline radiographer. The Australian Society of Medical Imaging and Radiation Therapy's 11th National Conference, Canberra, Australia, 22-24 March 2016, Journal of Medical Radiation Sciences, March 2016 Volume 63, Issue S1.
79. Wright C, Reeves P. RadBench: Benchmarking image interpretation skills. Radiography. 2016 May 1;22(2):e131-6.
80. Akimoto T, Wright C, Reeves P. Image interpretation performance: a multicentre undergraduate study. Manchester: UKRC; 2017. https://doi.org/10.13140/RG.2.2.36820.40328.
81. Neep, M. Brown, C. Reaching for new heights: Logan Hospital radiographer commenting. The Australian Society of Medical Imaging and Radiation Therapy's 12th National Conference, Canberra, Australia, 24-26 March 2017, Journal of Medical Radiation Sciences, March 2017 Volume 64, Issue S1.
82. Rodriguez CM. A student radiographer’s accuracy in ‘red-dotting’ appendicular skeletal radiographs: Clinical Audit. 2017
83. Tay YX, Wright C. Development of Image Interpretation skills - a follow up study of Singapore diagnostic radiographers. 2017.
84. Wright C, Reeves P. Image interpretation performance: a longitudinal study from novice to professional. Radiography 2017;23(1):e1e7.
85. Davis, B. Development of a reliable and sustainable radiographer commenting audit. The Australian Society of Medical Imaging and Radiation Therapy's 13th National Conference, Canberra, Australia, 15–18 March 2018, Journal of Medical Radiation Sciences, March 2018 Volume 65, Issue S1.
86. Freij RM, Duffy T, Hackett D, Cunningham D, Fothergill J. Radiographic interpretation by nurse practitioners in a minor injuries unit. Emergency Medicine Journal. 1996 Jan 1;13(1):41-43.
87. Benger JR. Can nurses working in remote units accurately request and interpret radiographs? Emerg Med J 2002;19: 68–70.
88. Lee GA, Chou K, Jennings N, O’Reilly G, McKeown E, Bystrzycki A, Varma D. The accuracy of adult limb radiograph interpretation by emergency nurse practitioners: A prospective comparative study. International journal of nursing studies. 2014 Apr 1;51(4):549-54.
89. Brealey S, Scally A, Hahn S, Thomas N, Godfrey C, Crane S. Accuracy of radiographers red dot or triage of accident and emergency radiographs in clinical practice: a systematic review. Clinical radiology. 2006 Jul 1;61(7):604-15.
90. Swaby-Larsen D. X-ray interpretation by emergency nurse practitioners: Dorthe Swaby-Larsen and colleagues discuss an audit of the ability of emergency nurses to interpret X-rays accurately. Emergency Nurse. 2009 Oct 1;17(6):24-9.
91. Tachakra S, Mukherjee P, Smith C, Dutton D. Are accident and emergency consultants as accurate as consultant radiologists in interpreting plain skeletal radiographs taken at a minor injury unit? Eur J Emerg Med, 9 (2) (2002), pp. 131-134.
92. Hardy ML, Barrett C. Requesting and interpreting trauma radiographs: a role extension for accident & emergency nurses. Accident and emergency nursing. 2003 Oct 1;11(4):202-13.
93. Summers A. Can nurses interpret X-rays safely without formal tuition?. Accident and emergency nursing. 2005 Jul 1;13(3):162-6.
94. McConnell J, Baird M. The accuracy of adult limb radiograph interpretation by emergency nurse practitioners: A prospective comparative study. International journal of nursing studies. 2015 Jan 1;52(1):495-7.
95. Free B, Lee GA, Bystrzycki A. Literature review of studies on the effectiveness of nurses ability to order and interpret X-rays. Australasian Emergency Nursing Journal. 2009 Feb 1;12(1):8-15.
96. Sakr M, Angus J, Perrin J, Nixon C, Nichol J, Wardrope J. Care of minor injuries by emergency nurse practitioners or junior doctors: a randomised controlled trial. The Lancet. 1999 Oct 16;354(9187):1321-6.
97. de Gruchy A, Granger C, Gorelik A. Physical therapists as primary practitioners in the emergency department: six-month prospective practice analysis. Physical therapy. 2015 Sep 1;95(9):1207-16.
98. Gill SD, Stella J. Implementation and performance evaluation of an emergency department primary practitioner physiotherapy service for patients with musculoskeletal conditions. Emergency Medicine Australasia. 2013 Dec;25(6):558-64.
99. Sutton M, Govier A, Prince S, Morphett M. Primary-contact physiotherapists manage a minor trauma caseload in the emergency department without misdiagnoses or adverse events: an observational study. Journal of physiotherapy. 2015 Apr 1;61(2):77-80.
100. Kinsella R, Collins T, Shaw B, Sayer J, Cary B, Walby A, Cowan S. Management of patients brought in by ambulance to the emergency department: role of the Advanced Musculoskeletal Physiotherapist. Australian Health Review. 2018 Jun 26;42(3):309-15.
101. Stanhope J, Grimmer-Somers K, Milanese S, Kumar S, Morris J. Extended scope physiotherapy for orthopedic outpatients: an update systematic review of the literature. J Multidiscip Healthc . 2012;5:37–45.
102. Ball ST, Walton K, Hawes S. Do emergency department physiotherapy Practitioner’s, emergency nurse practitioners and doctors investigate, treat and refer patients with closed musculoskeletal injuries differently?. Emergency Medicine Journal. 2007 Mar 1;24(3):185-8.
103. Daker-White G, Carr AJ, Harvey I, Woolhead G, Bannister G, Nelson I, Kammerling M. A randomised controlled trial. Shifting boundaries of doctors and physiotherapists in orthopaedic outpatient departments. Journal of epidemiology & community health. 1999 Oct 1;53(10):643-50.
104. Fennelly O, Blake C, FitzGerald O, Breen R, O'Sullivan C, O'Mir M, Desmeules F, Cunningham C. Advanced musculoskeletal physiotherapy practice in Ireland: A National Survey. Musculoskeletal care. 2018 Jun 21.
105. Desmeules F, Roy JS, MacDermid JC, Champagne F, Hinse O, Woodhouse LJ. Advanced practice physiotherapy in patients with musculoskeletal disorders: a systematic review. BMC Musculoskelet Disord . 2012;13:107.
106. Kersten P, McPherson K, Lattimer V, George S, Breton A, Ellis B. Physiotherapy extended scope of practice–who is doing what and why?. Physiotherapy. 2007 Dec 1;93(4):235-42.
107. Woollard M. The role of the paramedic practitioner in the UK. Australasian Journal of Paramedicine. 2015 Jul 15;4(1).
108. College of paramedics. Paramedic post-graduate curriculum guidance 2017. College of paramedics. The College of Paramedics 2017, Bridgwater.