GenAI in the hands of experts: A qualitative study of academics' experiences and future recommendations

Creating robust multiple-choice questions can be perceived as a challenging task by academic staff. The design process can also be time consuming and rely on the knowledge and experience of staff. Many engineering educators make use of these questions for formative and summative assessments. These questions can be a great way to provide formative feedback to students and diagnose what teaching interventions may be needed for different students based on their performance on the questions. This work-in-progress paper describes some initial results from a study whereby augmenting the skills educators bring with Generative Artificial Intelligence (GenAI) tools such as ChatGPT was trialled. A short one-to-one workshop was delivered to individual participants from various Science, Engineering, Technology and Mathematics (STEM) disciplines. This paper also shares the GenAI-mind-bending iterative and sequential prompting technique used to co-produce outputs. Pre- and post- workshop surveys with open text questions and interviews are being used to capture staff perceptions of using these tools in creation of the questions. The paper concludes with recommendations for future research and ethical considerations, particularly concerning copyright issues.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.

Chien, Y. T., Chang, Y. H., & Chang, C. Y. (2016). Do we click in the right way? A meta-analytic review of clicker-integrated instruction. Educational Research Review, 17, 1-18. https://doi.org/10.1016/j.edurev.2015.10.003
https://doi.org/10.1016/j.edurev.2015.10.003

Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American journal of physics, 69(9), 970-977. https://doi.org/10.1119/1.1374249
https://doi.org/10.1119/1.1374249

Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38.
https://doi.org/10.1145/3571730

Little, J. L., & Bjork, E. L. (2015). Optimizing multiple-choice tests as tools for learning. Memory & Cognition, 43, 14-26.
https://doi.org/10.3758/s13421-014-0452-8
PMid:25123774

Nicol, D. (2007). E‐assessment by design: using multiple‐choice tests to good effect. Journal of Further and higher Education, 31(1), 53-64.
https://doi.org/10.1080/03098770601167922

Malik, M., & Sime, J. A. (2022). Orchestrating Learning Together and Development of Team-Trust in Neurologically Typical and Neurologically Atypical Students: A Multicase Study. IEEE Transactions on Education, 65(3), 320-330.
https://doi.org/10.1109/TE.2022.3195043

Siontis, K. C., Attia, Z. I., Asirvatham, S. J., & Friedman, P. A. (2024). ChatGPT hallucinating: can it get any more human like?. European Heat Journal, 45(5), 321-323.
https://doi.org/10.1093/eurheartj/ehad766
PMid:38088452