Artificial Intelligence in Medical Education: Exploring ChatGPT’s Potencial as a Learning Tool in Ophthalmology

Abstract

INTRODUCTION: The recent introduction of large language models (LLMs) based on artificial intelligence (AI), the most popular of which is ChatGPT, has sparked interest in their application in Ophthalmology. The aim of this study is to investigate the contribution of ChatGPT as a learning tool during Ophthalmology residency.
MATERIAL AND METHODS: ChatGPT 3.5 (OpenAI, United States) was used to simulate an Ophthalmology exam, consisting of 260 multiple-choice questions, distributed among the 13 knowledge areas of the American Academy of Ophthalmology’s Basic and Clinical Science Course 2022-2023. Specialists from Centro Hospitalar de Lisboa Ocidental rated the justification provided by ChatGPT: 1 (very poor), 2 (poor), 3 (satisfactory), 4 (good) and 5 (very good). We tried to complement the model’s accuracy (number of correct questions in the direct experiment E1 and/or prompted E2) with precision (inter-response consistency over 3 repetitions), and to examine the quality of the justifications, as innovative elements.
RESULTS: ChatGPT accuracy was 63.1%, rising to 64.2% in the prompted test. In both tests, the best performance was recorded in fundamentals of ophthalmology (75.0% and 76.7%) and the worst in oculoplastics and orbit (46.7% and 55.0%), optics and rehabilitation and uveitis and inflammation (55.0%% and 53.3%; 55.0% and 53.3%). There was no statistically significant difference between the format of the question or the domain assessed and the model’s performance. ChatGPT gave the correct answers all three times in 69.3% of cases, twice in 17.2% and only once in 13.5% of situations. In 94.7% of the cases, the justifications were considered at least acceptable (≥3), of which 47.4% achieved the maximum score.
CONCLUSION: Our results confirm those described in the literature on LLMs and ophthalmic question banks. The probabilistic nature, the lack of specific training in Ophthalmology, and the inability to ensure the state of the art and to process images were the main limitations identified. Albeit recognizing ChatGPT’s competence to provide the right answers on different topics in Ophthalmology, we believe it is too early to recommend it as suitable learning tool for residents. However, it already demonstrates the accuracy, precision and scientific quality needed to possibly become a tool, especially if specifically trained in Ophthalmology in the future.