GenAI in the hands of experts: A qualitative study of academics' experiences and future recommendations

Conference paper


Malik, M., Nortcliffe, A., Turner, S., Abdel-Maguid, M. and Shah, Rehan 2024. GenAI in the hands of experts: A qualitative study of academics' experiences and future recommendations .
AuthorsMalik, M., Nortcliffe, A., Turner, S., Abdel-Maguid, M. and Shah, Rehan
TypeConference paper
Description

Creating robust multiple-choice questions can be perceived as a challenging task by academic staff. The design process can also be time consuming and rely on the knowledge and experience of staff. Many engineering educators make use of these questions for formative and summative assessments. These questions can be a great way to provide formative feedback to students and diagnose what teaching interventions may be needed for different students based on their performance on the questions. This work-in-progress paper describes some initial results from a study whereby augmenting the skills educators bring with Generative Artificial Intelligence (GenAI) tools such as ChatGPT was trialled. A short one-to-one workshop was delivered to individual participants from various Science, Engineering, Technology and Mathematics (STEM) disciplines. This paper also shares the GenAI-mind-bending iterative and sequential prompting technique used to co-produce outputs. Pre- and post- workshop surveys with open text questions and interviews are being used to capture staff perceptions of using these tools in creation of the questions. The paper concludes with recommendations for future research and ethical considerations, particularly concerning copyright issues.

KeywordsMultiple-choice question design; ChatGPT; Generative AI; Human in the loop; AI mind-bending; Prompting
Year2024
ConferenceUK & Ireland EERN24 Education Research Network Annual Conference 2024
Official URLhttps://epc.ac.uk/event/uk-and-ireland-engineering-education-research-network-annual-symposium-17th-18th-june-2024/
References

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.

Chien, Y. T., Chang, Y. H., & Chang, C. Y. (2016). Do we click in the right way? A meta-analytic review of clicker-integrated instruction. Educational Research Review, 17, 1-18. https://doi.org/10.1016/j.edurev.2015.10.003
https://doi.org/10.1016/j.edurev.2015.10.003

Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American journal of physics, 69(9), 970-977. https://doi.org/10.1119/1.1374249
https://doi.org/10.1119/1.1374249

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38.
https://doi.org/10.1145/3571730

Little, J. L., & Bjork, E. L. (2015). Optimizing multiple-choice tests as tools for learning. Memory & Cognition, 43, 14-26.
https://doi.org/10.3758/s13421-014-0452-8
PMid:25123774

Nicol, D. (2007). E‐assessment by design: using multiple‐choice tests to good effect. Journal of Further and higher Education, 31(1), 53-64.
https://doi.org/10.1080/03098770601167922

Malik, M., & Sime, J. A. (2022). Orchestrating Learning Together and Development of Team-Trust in Neurologically Typical and Neurologically Atypical Students: A Multicase Study. IEEE Transactions on Education, 65(3), 320-330.
https://doi.org/10.1109/TE.2022.3195043

Siontis, K. C., Attia, Z. I., Asirvatham, S. J., & Friedman, P. A. (2024). ChatGPT hallucinating: can it get any more human like?. European Heat Journal, 45(5), 321-323.
https://doi.org/10.1093/eurheartj/ehad766
PMid:38088452

Publication process dates
Deposited04 Sep 2024
Permalink -

https://repository.canterbury.ac.uk/item/98vzq/genai-in-the-hands-of-experts-a-qualitative-study-of-academics-experiences-and-future-recommendations

  • 30
    total views
  • 0
    total downloads
  • 4
    views this month
  • 0
    downloads this month

Export as

Related outputs

Unveiling pollution peaks: Comparing swarm intelligence with drone hill climber
Prior, O., Bin Azhar, M A Hannan, Sahota, V. and Turner, S. 2024. Unveiling pollution peaks: Comparing swarm intelligence with drone hill climber. https://doi.org/10.1109/SISY62279.2024.10737602
Impact of CDIO framework pedagogical approach adoption on the student learning and experience
Nortcliffe, A., Matei, G., Malik, M., Manna, S., Sheikholeslami, G. and James, H. 2024. Impact of CDIO framework pedagogical approach adoption on the student learning and experience.
SocMedHE: More than a conference
Turner, S. and Honeychurch, S. 2024. SocMedHE: More than a conference. The Journal of Social Media for Learning. 4 (1), pp. 25-38. https://doi.org/10.24377/LJMU.jsml.article724
An EDI engineering employability toolkit to aid engineering student progression
Nortcliffe, A., Fanusie, C., Matei, G., Makinde, M., Odev, S., Martin, E. and Eyres, S. 2024. An EDI engineering employability toolkit to aid engineering student progression.
How do we engineer inclusive education technology?
Nortcliffe, A. 2024. How do we engineer inclusive education technology?
Allyship education is key to EDI graduate engineering employment
Nortcliffe, A., Fanusie, C., Matei, G., Makinde, M., Odev, S., Martin, E. and Eyres, S. 2024. Allyship education is key to EDI graduate engineering employment. Royal Academy of Engineering.
Conceiving, designing, implementing and operating an EDI engineering employability learning toolkit to aid graduate employment
Nortcliffe, A., Fanusie, C., Matei, G., Makinde, M., Odev, S., Martin, E. and Eyres, S. 2024. Conceiving, designing, implementing and operating an EDI engineering employability learning toolkit to aid graduate employment.
Integrating AI into engineering education: Leveraging CDIO for enhanced assessment strategies
Nortcliffe, A., Saeidlou, S., Ghadiminia, N. and Imam, A. 2024. Integrating AI into engineering education: Leveraging CDIO for enhanced assessment strategies.
Potential practices for establishing curriculum agility through industrial engagement
Lyng, R. and Nortcliffe, A. 2024. Potential practices for establishing curriculum agility through industrial engagement.
Tailoring Poisson's ratio of a fabric using auextic structures
Robinson, B., Imam, A., Nortcliffe, A., Ahmed, A. and Sabir, T. 2024. Tailoring Poisson's ratio of a fabric using auextic structures.
Utilization of ChatGPT in CDIO projects to enhance the literacy of international students
Manna, S., Williams, S., Richmond-Fuller, A. and Nortcliffe, A. 2024. Utilization of ChatGPT in CDIO projects to enhance the literacy of international students.
The role of use cases when adopting augmented reality into higher education pedagogy
Ward, G., Turner, S., Pitt, C., Qi, M., Richmond-Fuller, A. and Jackson, T. 2024. The role of use cases when adopting augmented reality into higher education pedagogy.
Engineering learning of sustainable product lifecycle through CDIO
Nortcliffe, A., Imam, A. and Joyce, N. 2023. Engineering learning of sustainable product lifecycle through CDIO.
Is the future pipeline of education technology developers in jeopardy?
Nortcliffe, A. 2023. Is the future pipeline of education technology developers in jeopardy? Education Technology Insights.
A discussion of possibilities for establishing curriculum agility practices for industry ready graduates
Nortcliffe, A. and Lyng, R. 2023. A discussion of possibilities for establishing curriculum agility practices for industry ready graduates.
Engineering learning of sustainable product lifecycle through CDIO
Imam, A., Joyce, N. and Nortcliffe, A. 2023. Engineering learning of sustainable product lifecycle through CDIO.
The National Teaching Repository and social media
Turner, S., Faulkner, S and Withnell, N 2023. The National Teaching Repository and social media. https://doi.org/10.25416/NTR.24942471.v1
Trustworthy insights: A novel multi-tier explainable framework for ambient assisted living
Kasirajan, Merlin, Bin Azhar, M A Hannan and Turner, Scott 2023. Trustworthy insights: A novel multi-tier explainable framework for ambient assisted living. in: 2023 IEEE 22nd International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) IEEE. pp. 2554-2561
Practice-based engineering design for next-generation of engineers: A CDIO-based approach
Saeidlou, S., Ghadiminia, N., Nortcliffe, A. and Lambert, S. 2023. Practice-based engineering design for next-generation of engineers: A CDIO-based approach. in: The 19th CDIO International Conference: Proceedings - Full Papers
Integration of graduate employability skills through industry outsourced CDIO project
Manna, S., Joyce, N. and Nortcliffe, A. 2023. Integration of graduate employability skills through industry outsourced CDIO project. in: Lyng, R., Bennedsen, J., Bettaied, L., Bodsberg, N. R., Edstrom, K., Guojonsdottir, M. S., Roslof, J., Solbjord, O. K. and Oien, G. (ed.) The 19th CDIO International Conference: Proceedings - Full Papers NTNU SEED. pp. 425-435
How inclusive is VR to support EDI engineering curriculum learning?
Harrison, J., Forester, F., Ward, G., Tubby, M., Lithgow, P. and Nortcliffe, A. 2023. How inclusive is VR to support EDI engineering curriculum learning?
The National Teaching Repository − Sharing effective interventions: Learning from each other so that we can continue to enhance and improve what we do
Turner, S., Beckingham, S, Bullingham, L, Hartley, P, Cuthbert, K, Irving-Bell, D, Wooff, D, Tasler, N, Stinson, L and Withnell, N 2023. The National Teaching Repository − Sharing effective interventions: Learning from each other so that we can continue to enhance and improve what we do. Educational Developments. 24 (2), pp. 5-7.
An intelligent routing approach for multimedia traffic transmission over SDN
Jameel, Mohammed Al, Kanakis, Triantafyllos, Turner, Scott, Al-Sherbaz, Ali, Bhaya, Wesam S. and Al-khafajiy, Mohammed 2023. An intelligent routing approach for multimedia traffic transmission over SDN. in: IEEE.
Why should everybody learn Artificial Intelligence?
Turner, S. and Souag, A. 2022. Why should everybody learn Artificial Intelligence? ETD blog, Canterbury Christ church University
Optimizing artificial neural networks using LevyChaotic mapping on Wolf Pack optimization algorithm for detect driving sleepiness
Turner, S., Jassin, S.S. and Hassan, A.K.A 2022. Optimizing artificial neural networks using LevyChaotic mapping on Wolf Pack optimization algorithm for detect driving sleepiness. Iraqi Journal of Computers, Communications, Control & Systems Engineering (IJCCCE). 22 (3), pp. 128-136. https://doi.org/10.33103/uot.ijccce.22.3.12
Driver drowsiness detection using Gray Wolf Optimizer based on voice recognition
Sasim, S. S., Hassan, A. K. A. and Turner, S. 2022. Driver drowsiness detection using Gray Wolf Optimizer based on voice recognition. Aro - The Scientific Journal of Koya University. 10 (2), pp. 142-151. https://doi.org/10.14500/aro.11000
Practical ways to analyse Twitter data (quantitative and qualitative)
Turner, S. and Kelly, O. 2022. Practical ways to analyse Twitter data (quantitative and qualitative).
#LTHEchat 243: Self exclusion – through digital inequalities
Turner, S., Ward, G. and Elliott, C. 2022. #LTHEchat 243: Self exclusion – through digital inequalities. LTHEchat.
A reinforcement learning-based routing for real-time multimedia traffic transmission over software-defined networking
Al Jameel, M., Kanakis, T., Turner, S., Al-Sherbaz, A. and Bhaya, W. 2022. A reinforcement learning-based routing for real-time multimedia traffic transmission over software-defined networking. Electronics. 11 (15), p. 2441. https://doi.org/10.3390/electronics11152441
How are do you create an inclusive engineering graduate pipeline?
Nortcliffe, A. 2022. How are do you create an inclusive engineering graduate pipeline? Precision. June (24), pp. 8-11.
Evaluation of students’ performance in CDIO projects through blended learning
Manna, S., Battikh, N., Nortcliffe, A. and Camm, J. 2022. Evaluation of students’ performance in CDIO projects through blended learning.
Driver drowsiness detection using Gray Wolf Optimizer based on face and eye tracking
Jasim, S., Abdul Hassan, AK and Turner, S. 2022. Driver drowsiness detection using Gray Wolf Optimizer based on face and eye tracking. Aro - The Scientific Journal of Koya University. 10 (1), pp. 49-56. https://doi.org/10.14500/aro.10928
Deep learning approach for real-time video streaming traffic classification
Jameel, Mohammed Al, Turner, Scott, Kanakis, Triantafyllos, Al-Sherbaz, Ali and Bhaya, Wesam S. 2022. Deep learning approach for real-time video streaming traffic classification. in: 2022 International Conference on Computer Science and Software Engineering (CSASE) IEEE.
Digital reality in supporting students' learning, the challenges and opportunities
Nortcliffe, A. 2021. Digital reality in supporting students' learning, the challenges and opportunities. Education Technology Insights.
#SocMedHE more than a conference
Turner, S. 2021. #SocMedHE more than a conference.
Detecting a stroke-affected region in the brain by scanning with low-intensity electromagnetic waves in the radio frequency/microwave band
El rube’, I., Heatley, David and Abdel-Maguid, Mohamed 2021. Detecting a stroke-affected region in the brain by scanning with low-intensity electromagnetic waves in the radio frequency/microwave band. Healthcare. 9 (9). https://doi.org/10.3390/healthcare9091170
The legacy of Verena Holmes: inspiring next generation of engineers
Saeidlou, S., Ishaq, R., Nortcliffe, A. and Ghadiminia, N. 2021. The legacy of Verena Holmes: inspiring next generation of engineers.
Referencing within code in software engineering education
Turner, S. and Hill, G 2021. Referencing within code in software engineering education. National Repository of Teaching and Learning. https://doi.org/10.25416/NTR.14907891.v1
Free augmented reality
Turner, S. 2021. Free augmented reality. Edge Hill University. https://doi.org/10.25416/NTR.13622918.v1
Adaptive and flexible online learning during Covid19 lockdown
Manna, S., Nortcliffe, A., Sheikholeslami, G. and Richmond-Fuller, A. 2021. Adaptive and flexible online learning during Covid19 lockdown.
Why everyone should learn a bit about Machine Learning
Turner, S. 2020. Why everyone should learn a bit about Machine Learning.
Developing engineering growth mindset through CDIO outreach activities
Manna, S., Nortcliffe, A. and Sheikholeslami, G. 2020. Developing engineering growth mindset through CDIO outreach activities. in: Proceedings of the 16th International CDIO Conference Gothenburg, Sweden CDIO.
New approaches to engineering higher education: case studies
Nortcliffe, A., McIntosh, G. and Hernandez, E. 2019. New approaches to engineering higher education: case studies.
CDIO Open day learning activity to inspire the next generation of engineering applicants
Nortcliffe, A., Nortcliffe, A. and Sheikholeslami, G. 2019. CDIO Open day learning activity to inspire the next generation of engineering applicants.
Developing good practices for industrial engagement in co-creation of CDIO curriculum
Fortin, C, Nortcliffe, A. and Serreau, Y. 2019. Developing good practices for industrial engagement in co-creation of CDIO curriculum.
How accessible is the STEM post 16 education provision, the pipeline to computing and engineering programmes?
Nortcliffe, A., Stallard, J.A. and Barley, R. 2019. How accessible is the STEM post 16 education provision, the pipeline to computing and engineering programmes?
Statistically, does peer assisted learning make a difference on a UK engineering degree programme? HETL Scotland 2017
Nortcliffe, A., Parveen, S. and Keech-Pink, C. 2018. Statistically, does peer assisted learning make a difference on a UK engineering degree programme? HETL Scotland 2017. Journal of Applied Research in Higher Education. 14 (1), pp. 489-506. https://doi.org/10.1108/JARHE-04-2017-0047