Utilising transformers for American Sign Language fingerspelling recognition

Book chapter

Pinnington, J., Souag, A. and Azhar, H. 2024. Utilising transformers for American Sign Language fingerspelling recognition. in: 24th International Symposium on Computational Intelligence and Informatics (CINTI 2024)
AuthorsPinnington, J., Souag, A. and Azhar, H.
Abstract

This paper presents a machine learning approach to American Sign Language (ASL) fingerspelling recognition using Transformer models. Addressing the challenges of high variability in hand shapes, movement, and signing speed, the study utilises the new ASL Fingerspelling Recognition Corpus and the CRISP-DM methodology to develop and evaluate a proof-of-concept model. The model achieved a mean Levenshtein distance of 4.7, corresponding to an error rate of 16.37%. This research demonstrates the feasibility of using advanced AI techniques to enhance accessibility for the deaf and hard of hearing communities by translating ASL fingerspelling into text.

KeywordsAmerican Sign Language (ASL); Fingerspelling recognition; Transformer models; Machine learning; CRISP-DM; AI accessibility; Deaf and hard of hearing; Levenshtein distance; PyTorch; MediaPipe
Year2024
Book title24th International Symposium on Computational Intelligence and Informatics (CINTI 2024)
Output statusIn press
File
License
File Access Level
Restricted
Publication process dates
Deposited27 Nov 2024
Related URLhttps://conf.uni-obuda.hu/cinti2024/
EventIEEE 24th International Symposium on Computational Intelligence and Informatics (CINTI 2024)
Permalink -

https://repository.canterbury.ac.uk/item/99v5z/utilising-transformers-for-american-sign-language-fingerspelling-recognition

  • 10
    total views
  • 1
    total downloads
  • 8
    views this month
  • 0
    downloads this month

Export as

Related outputs

Machine learning in ASL fingerspelling recognition: A literature review
Pinnington, J., Souag, A. and Azhar, H. 2024. Machine learning in ASL fingerspelling recognition: A literature review. in: IEEE Xplore.
Enhancing cyberbullying detection with RoBERTa: A transformer-based approach
Azhar, H. and Runa, A. 2024. Enhancing cyberbullying detection with RoBERTa: A transformer-based approach. in: 16th International Conference on Global Security, Safety & Sustainability, ICGS3-24 - Cybersecurity and Human Capabilities through Symbiotic Artificial Intelligence Springer.
‘Parenting the Internet’ app: a Web guardian for protecting neurodiverse users online
Jones, R. and Azhar, H. 2024. ‘Parenting the Internet’ app: a Web guardian for protecting neurodiverse users online. in: IEEE 24th International Symposium on Computational Intelligence and Informatics (CINTI 2024) IEEE Xplore.
A novel dataset of annotated oyster mushroom images with environmental context for machine learning applications
Duman, S., Elewi, A., Hajhamed, A., Khankan, R., Souag, A. and Ahmed, A. 2024. A novel dataset of annotated oyster mushroom images with environmental context for machine learning applications. Data in Brief. https://doi.org/10.1016/j.dib.2024.111074
Unveiling pollution peaks: Comparing swarm intelligence with Drone Hill Climber
Prior, Oliver J., Hannan Bin Azhar, M. A., Sahota, Vijay and Turner, Scott 2024. Unveiling pollution peaks: Comparing swarm intelligence with Drone Hill Climber. in: 2024 IEEE 22nd Jubilee International Symposium on Intelligent Systems and Informatics (SISY) IEEE. pp. 399-404
A qualitative review of educational robots for STEM: Technical features and impacts
Manna, Soumya Kanti, Azhar, M. A. Hannan Bin and Greace, Ann 2024. A qualitative review of educational robots for STEM: Technical features and impacts. in: Proceedings of the International Convention MIPRO IEEE.
Innovative assistive device to enhance wrist drop treatment in patients
Trainer, C., Manna, S. and Azhar, H. 2024. Innovative assistive device to enhance wrist drop treatment in patients. in: Costin, H-N., Magjarević, R. and Petroiu, G. G. (ed.) Advances in Digital Health and Medical Bioengineering: Proceedings of the 11th International Conference on E-Health and Bioengineering, EHB-2023, November 9–10, 2023, Bucharest, Romania – Volume 2: Health Technology Assessment, Biomedical Signal Processing, Medicine and Informatics Cham Springer. pp. 489-497
Tele-controlled upper arm exoskeleton for post-stroke recovery
Manna, S., Khan, A., Dilley, O. and Azhar, H. 2024. Tele-controlled upper arm exoskeleton for post-stroke recovery. in: Costin, H-N, Magjarević, R. and Petroiu, G. G. (ed.) Advances in Digital Health and Medical Bioengineering Cham Springer. pp. 478-488
Design and implementation of a cost-aware and smart oyster mushroom cultivation system
Souag, A., Elewi, A., Hajhamed, A., Khankan, R., Duman, S. and Ahmed, A. 2024. Design and implementation of a cost-aware and smart oyster mushroom cultivation system. Smart Agricultural Technology. Volume 8. https://doi.org/10.1016/j.atech.2024.100439
Cancer: Investigating the impact of the implementation platform on machine learning models
Olowolayemo, A. S., Souag, A. and Sirlantzis, K. 2024. Cancer: Investigating the impact of the implementation platform on machine learning models. in: Mengoni, M. and Souag, A. (ed.) AIHealth 2024, The First International Conference on AI-Health ThinkMind.
Metaverse application forensics: Unravelling the virtual truth
Azhar, H. and Rush-Gadsby, O. Metaverse application forensics: Unravelling the virtual truth. in: Cybersecurity Challenges in the Age of AI, Space Communications and Cyborgs Proceedings of the 15th International Conference on Global Security, Safety and Sustainability, London, October 2023 Cham Springer. pp. 399-414
Transformer-based Models for Enhanced Amur Tiger Re-Identification
Bai, Xufeng, Islam, Tasmina and Bin Azhar, M A Hannan 2024. Transformer-based Models for Enhanced Amur Tiger Re-Identification. in: 2024 IEEE 22nd World Symposium on Applied Machine Intelligence and Informatics (SAMI) IEEE.
Breaking barriers: A novel framework to evaluate usability of accessibility applications
Azhar, H., Islam, T. and Marczak, J. 2023. Breaking barriers: A novel framework to evaluate usability of accessibility applications. in: 36th International BCS Human-Computer Interaction Conference British Computer Society. pp. 23-33
An interactive web portal for customised telerehabilitation in neurological care
Hannan Bin Azhar, M A, Mészáros, Zoltán, Islam, Tasmina and Manna, Soumya K. 2023. An interactive web portal for customised telerehabilitation in neurological care. in: 2023 IEEE 22nd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) IEEE. pp. 1814-1821
Trustworthy insights: A novel multi-tier explainable framework for ambient assisted living
Kasirajan, Merlin, Bin Azhar, M A Hannan and Turner, Scott 2023. Trustworthy insights: A novel multi-tier explainable framework for ambient assisted living. in: 2023 IEEE 22nd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) IEEE. pp. 2554-2561
Assistive telehealth systems for neurorehabilitation
Azhar, H. 2023. Assistive telehealth systems for neurorehabilitation.
Optimal locations and computational frameworks of FSR and IMU sensors for measuring gait abnormalities
Manna, S., Azhar, H. and Greace, A. 2023. Optimal locations and computational frameworks of FSR and IMU sensors for measuring gait abnormalities. Heliyon. 9 (4), p. e15210. https://doi.org/10.1016/j.heliyon.2023.e15210
Spying on kids’ smart devices: Beware of security vulnerabilities!
Azhar, H., Smith, D. and Cain, A. 2023. Spying on kids’ smart devices: Beware of security vulnerabilities! in: Jahankhani, H. (ed.) Cybersecurity in the Age of Smart Societies Proceedings of the 14th International Conference on Global Security, Safety and Sustainability, London, September 2022 Springer. pp. 123-140
Why should everybody learn Artificial Intelligence?
Turner, S. and Souag, A. 2022. Why should everybody learn Artificial Intelligence? ETD blog, Canterbury Christ church University
Cyber threats and exploits during the pandemic
Lo, J. and Azhar, H. Cyber threats and exploits during the pandemic. ASEAN Tech and Security, Singapore .
Z is for Zoombombing
Azhar, H. 2022. Z is for Zoombombing. Medium.
Progressive web app for real-time doctor-patient communication and searchable health conditions
Hannan Bin Azhar, M A and Mohan, Joseph Thomas 2022. Progressive web app for real-time doctor-patient communication and searchable health conditions. 2022 E-Health and Bioengineering Conference (EHB). https://doi.org/10.1109/EHB55594.2022.9991288
Forensic investigations of Google Meet and Microsoft Teams – two popular conferencing tools in the Pandemic
Azhar, H., Timms, J. and Tilley, B. 2022. Forensic investigations of Google Meet and Microsoft Teams – two popular conferencing tools in the Pandemic. in: Digital Forensics and Cyber Crime Springer Nature. pp. 20-34
Tele-tDCS: A Novel Tele-neuromodulation Framework using Internet of Medical Things
Herring, Samuel, Azhar, M. A. Hannan Bin and Sakel, Mohamed 2022. Tele-tDCS: A Novel Tele-neuromodulation Framework using Internet of Medical Things. in: Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies - BIODEVICES Setúbal, Portugal SCITEPRESS - Science and Technology Publications. pp. 84-93
Automatic identification of non-biting midges (Chironomidae) using object detection and deep learning techniques
Hollister, Jack, Vega, Rodrigo and Azhar, M. A. Hannan Bin 2022. Automatic identification of non-biting midges (Chironomidae) using object detection and deep learning techniques. in: Marsico, Maria D., Sanniti de Baja, Gabriella and Fred, Ana (ed.) Proceedings of the 11 International Conference on Pattern Recognition Applications and Methods SCITEPRESS - Science and Technology Publications.
A smart and secure IoMT tele-neurorehabilitation framework for post-stroke patients
Manna, S., Azhar, H. and Sakel, M. 2022. A smart and secure IoMT tele-neurorehabilitation framework for post-stroke patients. in: Bhaumik, S., Chattopadhyay, S., Chattopadhyay, T. and Bhattacharya, S. (ed.) Proceedings of International Conference on Industrial Instrumentation and Control ICI2C 2021 Singapore Springer. pp. 11-20
An inclusive student-led online class test during the pandemic
Manna, S. and Azhar, H. 2021. An inclusive student-led online class test during the pandemic . Assessment and Feedback Symposium 2021.
A forensic tool to acquire radio signals using software defined radio
Azhar, H. and Abadia, G. 2021. A forensic tool to acquire radio signals using software defined radio. in: Security and Privacy in Communication Networks : 17th EAI International Conference, SecureComm 2021, Virtual Event, September 6-9, 2021, Proceedings, Part I Springer.
Post-pandemic digital education: Investigating smart workspaces within the higher education sector
Azhar, M A Hannan Bin, Lepore, Emily Louise and Islam, T. 2021. Post-pandemic digital education: Investigating smart workspaces within the higher education sector. Proceedings of the BCS 34th British HCI Conference 2021. 34, pp. 284-288. https://doi.org/10.14236/ewic/hci2021.30
A study of user experiences and network analysis on anonymity and traceability of bitcoin transactions
Azhar, M.A.H.B and Whitehead, R.V. 2021. A study of user experiences and network analysis on anonymity and traceability of bitcoin transactions. EAI Endorsed Transactions on Security and Safety. https://doi.org/10.4108/eai.30-4-2021.169577
BCI controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients
Azhar, H. 2021. BCI controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients.
Forensic investigations of popular ephemeral messaging applications on Android and iOS platforms
Azhar, H., Cox, R. and Chamberlain, A. 2020. Forensic investigations of popular ephemeral messaging applications on Android and iOS platforms. International Journal on Advances in Security. 13 (1 & 2), pp. 41 - 53.
How can the semantic web and ontologies help history and archeology
Souag, A. 2019. How can the semantic web and ontologies help history and archeology. in: Dans les dédales du web. Historiens en territoires numériques Paris Éditions de la Sorbonne.
Comparisons of forensic tools to recover ephemeral data from iOS apps used for cyberbullying
Chamberlain, A. and Azhar, H. 2019. Comparisons of forensic tools to recover ephemeral data from iOS apps used for cyberbullying. in: CYBER 2019, The Fourth International Conference on Cyber-Technologies and Cyber-Systems IARIA. pp. 88-93
Recovery of forensic artefacts from a smart home IoT ecosystem
Azhar, H. and Bate, S. 2019. Recovery of forensic artefacts from a smart home IoT ecosystem. in: CYBER 2019, The Fourth International Conference on Cyber-Technologies and Cyber-Systems IARIA. pp. 94-99
BCI controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients
Casey, A., Azhar, H., Grzes, M. and Sakel, M. 2019. BCI controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients. Disability and Rehabilitation: Assistive Technology. 16 (5), pp. 525-537. https://doi.org/10.1080/17483107.2019.1683239
Effects of students’ preferences in use of lighting and temperature on productivity in a university setting
Azhar, H., Islam, T. and Alfieri, M. 2019. Effects of students’ preferences in use of lighting and temperature on productivity in a university setting. in: Zheng, P., Callaghan, V., Crawford, D., Kymalainen, T. and Reyes-Munoz, A. (ed.) EAI International Conference on Technology, Innovation, Entrepreneurship and Education Springer.
Use of wearable technology to measure emotional responses amongst tennis players
Azhar, H., Nelson, T. and Casey, A. 2019. Use of wearable technology to measure emotional responses amongst tennis players. in: Zheng, P., Callaghan, V., Crawford, D., Kymalainen, T. and Reyes-Munoz, A. (ed.) EAI International Conference on Technology, Innovation, Entrepreneurship and Education Springer.
Drone forensic analysis using open source tools
Azhar, H., Barton, T. and Islam, T. 2018. Drone forensic analysis using open source tools. Journal of Digital Forensics, Security and Law. 13 (1), pp. 7-30.
Using the AMAN-DA method to generate security requirements: a case study in the maritime domain
Souag, A., Mazo, R., Salinesi, C. and Comyn-Wattiau, I. 2018. Using the AMAN-DA method to generate security requirements: a case study in the maritime domain. Requirements Engineering Journal. 23 (557–580). https://doi.org/10.1007/s00766-017-0279-5
A cost-effective BCI assisted technology framework for neurorehabilitation
Azhar, H., Casey, A. and Sakel, M. 2018. A cost-effective BCI assisted technology framework for neurorehabilitation.
An investigation on forensic opportunities to recover evidential data from mobile phones and personal computers
Naughton, P. and Azhar, H. 2017. An investigation on forensic opportunities to recover evidential data from mobile phones and personal computers.
BCI controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients
Azhar, H., Barton, T., Casey, A. and Sakel, M. 2017. BCI controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients. Research and Knowledge Exchange Conference 2017.
Open source forensics for a multi-platform drone system
Barton, T. and Azhar, H. 2018. Open source forensics for a multi-platform drone system. in: Matousek, P. and Schmiedecker, M. (ed.) 9th EAI International Conference on Digital Forensics & Cyber Crime Springer. pp. 83-96
Evaluation of the MPS Predictive Policing Trial (redacted)
Bryant, R., Azhar, H., Blackburn, B. and Falade, M. 2015. Evaluation of the MPS Predictive Policing Trial (redacted).
Forensic analysis of popular UAV systems
Barton, T. and Azhar, H. 2017. Forensic analysis of popular UAV systems. Emerging Security Technologies (EST), 2017 Seventh International Conference on. https://doi.org/10.1109/EST.2017.8090405
A wearable brain-computer interface controlled robot
Azhar, H., Badicioiu, A. and Barton, T. 2016. A wearable brain-computer interface controlled robot.
Forensic analysis of the recovery of Wickr’s ephemeral data on Android platforms
Barton, T. and Azhar, H. 2016. Forensic analysis of the recovery of Wickr’s ephemeral data on Android platforms. in: Klemas, T. and Falk, R. (ed.) CYBER 2016 : The First International Conference on Cyber-Technologies and Cyber-Systems IARIA. pp. 35-40
Forensic analysis of secure ephemeral messaging applications on Android platforms
Azhar, H. and Barton, T. 2017. Forensic analysis of secure ephemeral messaging applications on Android platforms. in: Global Security, Safety and Sustainability - The Security Challenges of the Connected World: 11th International Conference, ICGS3 2017, London, UK, January 18-20, 2017, Proceedings Springer.
Reusable knowledge in security requirements engineering: a systematic mapping study
Souag, A., Mazo, R., Salinesi, C. and Comyn-Wattiau, I. 2016. Reusable knowledge in security requirements engineering: a systematic mapping study. Requirements Engineering Journal. 21 (251–283). https://doi.org/10.1007/s00766-015-0220-8
Usability and performance measure of a consumer-grade brain computer interface system for environmental control by neurological patients
Deravi, F., Ang, C., Azhar, H., Al-Wabil, A., Philips, M. and Sakel, M. 2015. Usability and performance measure of a consumer-grade brain computer interface system for environmental control by neurological patients. International Journal of Engineering and Technology Innovation (IJETI). 5 (3), pp. 165-177.
Criticality dispersion in swarms to optimize n-tuples
Azhar, H., Deravi, F. and Dimond, K. 2008. Criticality dispersion in swarms to optimize n-tuples. in: GECCO '08: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation New York Association for Computing Machinery. pp. 1-8
Particle swarm intelligence to optimize the learning of n-tuples
Azhar, H., Deravi, F. and Dimond, K. 2008. Particle swarm intelligence to optimize the learning of n-tuples. Journal of Intelligent Systems. 17 (S), pp. 169-196. https://doi.org/10.1515/JISYS.2008.17.S1.169
Automatic identification of wildlife using local binary patterns
Azhar, H., Hoque, S. and Deravi, F. 2012. Automatic identification of wildlife using local binary patterns. in: IET Conference on Image Processing (IPR 2012) Institute of Engineering and Technology. pp. 5-11
Zoometrics - biometric identification of wildlife using natural body marks
Hoque, S., Azhar, H. and Deravi, F. 2011. Zoometrics - biometric identification of wildlife using natural body marks. International Journal of Bio-Science and Bio-Technology. 3 (3), pp. 45-53.
A security ontology for security requirements elicitation
Souag, A. and Salinesi C., Mazo R., Comyn-Wattiau I. 2015. A security ontology for security requirements elicitation. https://doi.org/10.1007/978-3-319-15618-7_13
AMAN-DA: A knowledge reuse based approach for domain specific security requirements engineering
Souag, A. 2015. AMAN-DA: A knowledge reuse based approach for domain specific security requirements engineering. PhD Thesis Université Paris 1 Panthéon-Sorbonne CRI - Centre de Recherche en Informatique de Paris 1
Forensic acquisitions of WhatsApp data on popular mobile platforms
Shortall, A. and Azhar, H. 2015. Forensic acquisitions of WhatsApp data on popular mobile platforms. in: Proceedings of the Sixth International Conference on Emerging Security Technologies IEEE. pp. 13-17
A methodology for defining security requirements using security and domain ontologies
Souag, A., Salinesi C. and Comyn-Wattiau I. 2013. A methodology for defining security requirements using security and domain ontologies. Insight. Volume 16 (4), pp. 14-16. https://doi.org/10.1002/inst.201316414
Ontologies for security requirements: a literature survey and classification’
Souag, A. and Salinesi C., Comyn-Wattiau I. 2012. Ontologies for security requirements: a literature survey and classification’. https://doi.org/10.1007/978-3-642-31069-0_5