AI students take on real candidates in UK university exams - and fool the examiners

• A new study has found almost all AI generated exam answers went unnoticed by markers
• On top of that, AI chatbots were scoring higher grades on average than real students
• Researchers say this is a sign that educators need to evolve in a changing world
• But one university leader says students also need to learn to work with AI

A UK university has turned the microscope on itself, conducting a real-world Turing test which saw AI infiltrate university exams - and the results could have some big implications.

In a new study from the University of Reading, published in science journal PLOS One this week, 33 alias student profiles were set up by the university’s tech team. The ‘students’ were all completing Bachelor of Science degrees in psychology, across different years of study. Researchers helped them to complete an array of online assignments and exams, using unedited answers written by OpenAI’s artificial intelligence chatbot GPT-4.

In what is though to be the largest study of its kind, researchers found that 94% of their AI submissions slipped by examiners unnoticed. On top of that, the AI students did better than their peers. Their grades were on average about half a grade boundary higher than real students, with an 83.4% chance that they would outperform a random selection of real student submissions.

Associate Professor Peter Scarfe, one of the study’s authors, said that their findings should be a “wake up call” for educators, with a recent UNESCO survey finding that less than 10% of schools worldwide had policies or guidance in place on the use of generative AI. “Many institutions have moved away from traditional exams to make assessment more inclusive. Our research shows it is of international importance to understand how AI will affect the integrity of educational assessments,” he said. “We won’t necessarily go back fully to hand-written exams, but global education sector will need to evolve in the face of AI.”

Co-author professor Etienne Roesch added: “As a sector, we need to agree how we expect students to use and acknowledge the role of AI in their work. The same is true of the wider use of AI in other areas of life to prevent a crisis of trust across society.

“Our study highlights the responsibility we have as producers and consumers of information. We need to double down on our commitment to academic and research integrity,” she continued.

What can universities actually do about AI?

With constant advancements in AI technology available to almost anyone, it has proven difficult for education providers trying to maintain academic integrity from their students to keep up. But this didn’t mean schools or universities had to go back to pen and paper exams just yet.

Some tech companies have effectively entered arms races with generative AIs, like assignment submission company Turnitin. The company initially claimed its AI detection tool could pick up “97 percent of ChatGPT and GPT-3 authored writing, with a very low less than 1/100 false positive rate”, the authors of the new study said. They later conceded that real-world use yielded different results from the lab, while other research indicated some of these tools were more likely to flag work by students for whom English was not their native language.

But educators say with AI generators most likely here to stay - and a valuable workplace tool students need to get used to - it’s important to learn to work with them. The University of Reading’s pro-vice chancellor of education, Professor Elizabeth McCrum, said of the study: “It is clear that AI will have a transformative effect in many aspects of our lives, including how we teach students and assess their learning.

“At Reading, we have undertaken a huge programme of work to consider all aspects of our teaching, including making greater use of technology to enhance student experience and boost graduate employability skills,” she continued. “Solutions include moving away from outmoded ideas of assessment and towards those that are more aligned with the skills that students will need in the workplace, including making use of AI. Sharing alternative approaches that enable students to demonstrate their knowledge and skills, with colleagues across disciplines, is vitally important.”

She believed this left the university in a “strong position” to help current and future students both learn about and benefit from the rapid developments in AI.

To learn more about using AI chatbots and some of the other risks they can pose, check out this piece by Boost’s technology specialist Matthew Mohan-Hickson - on what top cybersecurity experts have to say.