The research team includes UI Hydroinformatics Lab members Ibrahim Demir (CEE associate professor), Vinay Pursnani, and Yusuf Sermet.
Friday, April 28, 2023

The controversial artificial intelligence chatbot called ChatGPT could pass an exam that's part of the requirements to become a licensed professional engineer in the United States, according to the findings of a University of Iowa College of Engineering study. 

The research team consisting of a professor, a research scientist, and a graduate student gave a 134-question practice test comparable to the highly competitive Fundamentals of Engineering (FE) Environmental Exam to ChatGPT. The exam is typically a prerequisite to getting a professional engineering (PE) license, which is important to getting a job in engineering.

The FE exam is designed to test knowledge of a broad range of environmental engineering topics through multiple-choice questions, point-and-click options, drag-and-drop options, and fill-in-the-blank. 

Two versions of the test were given to three iterations of ChatGPT (GPT-3.5-Legacy, GPT-3.5-Turbo, and GPT-4). The GPT-4 base model with no training or customization scored 66.42%. With modest prompt modifications, the GPT-4 base model scored 75.37%, which can be reasonably considered a passing grade although there is no official passing score, according to the study. 

“The findings reflect remarkable improvements in mathematical capabilities across successive iterations of ChatGPT models, showcasing their potential in solving complex engineering problems,” the study states.  

ChatGPT, or GPT (Generative Pre-trained Transformer) for Chat, is an AI model that has been trained on large amounts of text data using unsupervised learning techniques, according to OpenAI, a San Francisco-based AI research lab that introduced ChatGPT in 2022. 

This is the latest example of ChatGPT passing challenging tests, including bar exams, the U.S. medical licensing exam, and an MBA exam at the University of Pennsylvania Wharton School of Business.  

The UI study is still in the process of being peer-reviewed for journal publication.  

Co-authors of the study include Ibrahim Demir, associate professor of civil and environmental engineering and director of UI Hydroinformatics Lab (UIHILab); Vinay Pursnani, a graduate research assistant at UIHILab who is pursuing a master’s degree in computer science; and Yusuf Sermet, a research scientist at UIHILab. 

The team noted potential and risks with the ChatGPT technology. Notably, cheating is a significant concern. However, ChatGPT could also be used to improve test preparation and assessment. As a tutor, ChatGPT could offer personalized instruction, make the test taker’s preparation process more efficient, expand access through tailored instruction, and serve as an unbiased grader for subjective questions.  

The study provided several recommendations for future work:  

  • Integrating computational tools and engineering software with AI. 

  • Developing and integrating AI study partners into the classroom 

  • Addressing AI challenges in education 

  • Investigating the role of AI in learning.

  • Preventing cheating with AI-resistant exam questions. 

  • Preparing next-generation professionals for the age of AI. 

  • Enhancing accessibility and inclusion. 

The study is available at https://arxiv.org/pdf/2304.12198.pdf.