Assessment in an AI world

Read guidance on assessment design to better support student digital literacy and mitigate academic integrity challenges associated with generative AI.

The rise of generative AI provides a catalyst for us to question why and how we assess students, both to ensure our assessments are valid and robust, and that they, and the teaching that supports them, adequately prepare students for life after graduation (QAA, 7th Sept 2023).

Evaluate your existing assessments

Generative AI tools expose a wide range of current assessment modes to intentional or accidental academic misconduct. Listed below are some common assessment types with an overview of their potential vulnerability to academic misconduct using AI. 

Suggestions for adapting more vulnerable assessments to make them more AI resilient are included. Convenors should consider moving towards using lower risk assessment types that are intrinsically more resilient to generative AI.  

Very high risk 

Assessment types that are highly vulnerable to academic misconduct using AI include:

• Non invigilated quizzes and tests

  Quizzes and tests are vulnerable where questions ask students to define or reproduce basic disciplinary knowledge. Generative AI tools can respond very effectively to multiple choice (closed response) questions based around factual recall or basic knowledge application and can also provide supporting options to explain why each option is correct or incorrect.

  MCQs in particular are best used for low stakes coursework assessments designed to help students and staff test understanding. 

  If using, consider:

  • presenting questions that require the student to apply a concept or principle to an up-to date scenario or case study 

  • using distractors that are all plausible, consistent in content and structure, and share important information with the correct option thereby increasing the need for the evaluation of all options to identify the correct answer 

  • how to ensure questions remain clear and understandable to students. 

   

  Do not use images solely to trick AI. Accessibility requirements mean images must be accompanied with a text description. Generative AI tools can easily interpret images and diagrams. 

• Non-invigilated short answer questions

  Online non-invigilated examinations, which typically contain a significant proportion of short answer (open response) questions involving the recall of knowledge or basic knowledge application are also vulnerable. The ability of generative AI tools to provide responses in real-time negates the effect of reducing the period within which the assessment is completed.  

  If using, consider: 

  • posing hypothetical questions with no simple answer 

  • posing hypothetical scenario-based questions 

  • asking students to interpret information from a range of sources 

  • requiring students to draw on specific sources, e.g. chapter in course text book 

  • designing the assessment to be ‘open book’. 

   

  Students will need to be given the opportunity to practice responding to novel types of exam questions. 

High risk

Assessment types that are vulnerable to academic misconduct using AI include: 

• Essays and other long form writing

  Non-invigilated essays on broad, general, and well-known concepts are especially vulnerable to the impacts of generative AI where it can excel at presenting information and mimicking writing styles. Iterative prompting can dramatically increase the output quality, even mimicking the student's writing style. Generative AI can be used to redraft existing text to bypass plagiarism checkers, with many tools now capable of inserting genuine research citations.  

  Whilst generative AI can develop essays that may appear consistent and follow a logical structure, they can fall short in key areas like developing strong independent arguments, analysing and evaluating evidence, establishing connections between ideas, and demonstrating original thought.  

  If using, consider: 

  • reviewing marking criteria and relative weighting given to assessment requirements 

  • ensuring your students understand the purpose of the essay, how it aligns with the module learning outcomes, and what they will learn from engaging with the process of research and writing 

  • embedding opportunities for review and feedback at different stages in the development of essays 

  • incorporating reflective components, e.g. invite students to reflect on their personal experiences, opinions or observations on the topic or on the process they went through in developing their response 

  • our related guidance on developing writing assignments. 
• Literature reviews or paper critiques

  Generative AI tools are capable of literature searches and interrogating sources based on given research questions.  It's possible to upload papers to ChatGPT 4, Claude and many other generative AI tools and then ask questions of the sources. DocAnalyser has the same functionality but won’t hallucinate and will provide links to sources.

  Many can rephrase the contents in a different manner, for example producing summaries for a lay audience and include suggestions for future work, although the suggestions may not always be practical.

  Tools like Elicit, Consensus and SciSpace can use semantic searches to find papers. Tools like Litmaps can create dynamic maps of references and citations. Tools like Perplexity.ai and Consensus are designed to synthesize information from multiple sources and provide a broad overview of a topic while also allowing users to deep dive into specific areas.

  Watch a video on how AI makes literature reviews trivial.

  If using, consider: 

  • the purpose of your paper critique or literature review and how guiding students in the use of such tools might help them develop their academic skills, engage in higher order thinking and meet the relevant learning outcomes. How might such tools speed up the initial stages of locating and determining the content and relevance of sources so students can focus more time on focused reading? 

  • breaking down the task and adding a reflective element which asks students to consider the advantages and draw backs of using such tools at different stages of the research process. 

   

• Technical Reports

  The risks associated with reports are similar to essays, particularly if focused upon well-known topics, examples or issues.  

  Generative AI is capable of fabricating data that fit defined research trends. Some AI tools allow CSV or PDF files to be uploaded, which opens the scope for automated data analysis and interpretation, bypassing the critical thinking and understanding typically required. Many tools, including ChatGPT, Claude and Bard can write or debug code in common programming languages like Python or R to visualise the analysed data. Data analytics can then be performed using natural language prompts to integrate the data. 

  The ability of generative AI tools to process large amounts of data can lead to reports that appear factually accurate. However, they often lack depth. They might plagiarise existing work by combining information from various sources without truly understanding the underlying concepts or properly citing them.  Also, an AI-generated report might present data but lack a clear explanation of its significance, the reasoning behind the methodologies used, or the limitations of the study and may struggle with the crucial aspects of analysis and interpretation.  

  If using, consider: 

  • providing clear guidance on structure 

  • asking students to explain their process of interpreting the research methodology that generated the data, the implications of the data and how they reached their conclusions  

  • allowing students to use AI, with acknowledgment, for specific stages or processes 

  • reviewing marking criteria and reduce the weighting given to assessment requirements that AI can more easily complete. 
• Projects and dissertations

  The potential risks of generative AI tools for projects and dissertations are similar to those for essays and technical reports. Most at risk are projects and dissertations that form literature reviews or summaries of well-known topic areas. 

  If using, consider:

  • a process whereby project supervisors discuss and agree with students the appropriateness of using AI for the project (see, for example, guidelines for the use of generative AI in dissertation projects produced by Queens University Belfast) 

  • requiring students include a novel component, either disciplinary or localised, to the work 

  • incorporating an in-person oral assessment 

  • requiring students submit their projects or dissertations in stages, enabling you to observe their thought development and writing style - see scaffolded assessments.

Medium Risk

Assessment types within the mid-range of vulnerability to academic misconduct using AI include: 

• Reflective accounts

  Reflective accounts or commentaries, whilst focusing on a student's personal learning journey, can be susceptible to manipulation by generative AI. One area of vulnerability is the potential for generative AI to mimic surface-level reflection. They can be trained on student reflection examples and may be able to generate text that uses appropriate vocabulary and references specific course content. However, genuine reflection requires introspection, self-evaluation and a critical analysis of the learning process: aspects that current generative AI finds difficult.

  Generative AI tools can fabricate experiences or learning outcomes. While they can process course materials, they can't replicate the actual experience of grappling with concepts, participating in discussions or overcoming challenges. An inauthentic commentary might present a sanitised version of a student’s learning journey, lacking the genuine struggles and growth that they (the student) would typically need to describe. 

  If using, consider:

  • how reflective elements are assessed and aligned to the learning outcomes, i.e. how they provide a means for students to demonstrate their learning 

  • providing students with guidance on the purpose, process and practice of reflection as well as practice opportunities in class 

  • embedding reflective elements within iterative assessments so that the reflection, and feedback on it, can be applied. 

Low risk

Assessments with the least vulnerability to academic misconduct using AI include: 

• In-person invigilated exams

  In-person invigilated examinations, where access to third-party materials and online materials is typically restricted, are more resilient to generative AI tools than online non-invigilated assessments.
  
  Where they include questions requiring the recall of knowledge or basic knowledge application, students might pre-generate and memorise responses to commonly used question types. Review and refresh the questions used in examinations on an annual basis.

  If using, consider:

  • (particularly for high weighted examinations) how your approach to assessment across a course mitigates against the pedagogic limitations of such examinations.
• Academic portfolios

  Unlike a single exam or essay, academic porfolios showcase a collection of student work that has been developed over time. This cumulative aspect makes it difficult for AI to develop a portfolio that is cohesive and reflects an individual student's learning journey and development.
  
  Porfolios often contain diverse materials like creative projects, drafts with revisions, and reflections. This variety challenges current generative AI tools which can struggle to adapt to different formats and content types in a coherent manner. Portfolios also often emphasise critical thinking skills like selection, curation, and self-reflection. These skills are not easily replicated by generative AI, which can struggle to explain the rationale behind the chosen materials or articulate genuine personal growth.

• Scaffolded assessments

  Scaffolded assessment breaks down a complex learning objective into smaller, more manageable steps. A scaffolded assessment provides students with a staged series of tasks geared towards achieving an overall outcome. Each task is accompanied by instructions, support and measures to help check progress and enable the development of knowledge, understanding and skills, thereby improving student confidence in their own work and understanding of expectations of each task. 

  Unlike a single test where generative AI tools might mimic successfully the final answer, scaffolded assessments track progress over time. Drafts, revisions, reflections, and feedback from staff are all part of the evaluative process. These elements are challenging for AI to produce as they require genuine understanding and adaptation throughout the learning process. 

  Scaffolded assessments emphasise critical thinking and problem-solving alongside the acquisition of knowledge and skills. It's difficult for generative AI to replicate the thought process behind a solution or the ability to learn from mistakes, skills that become evident through scaffolded tasks and the interactions with staff members and fellow students.  

  If using, consider: 

  • how to design in reasonable adjustments and support students with exceptional circumstances  

  • asking students to submit assignment cover sheets, identifying areas they were least confident in/would like feedback on, or aspects they have tried to improve based on previous feedback. Provide pro-formas or prompts. 

   

  Many of the assessment types mentioned can be broken down and scaffolded to lower their risk. 

• Oral assessments

  Interviews and oral presentations rely on dynamic interaction and human judgment. This makes them more challenging for generative AI to exploit. Their strength is in their ability to enable follow-up questions and to engage the student in meaningful dialogue about their discipline area. Generative AI struggles to adapt to these dynamic exchanges and demonstrate genuine understanding.
  
  Oral assessments directly evaluate communication skills and critical thinking in real-time. These are areas where AI tools struggle, making it difficult to convincingly replicate natural human communication or thought processes. 

  If using, consider:

  • Our related guidance on oral assessment, providing advice on best practice when designing and implementing a wide range of inclusive oral assessments.
• Practical assessments

  Practical assessments require students to apply their skills and knowledge in real-world settings. Generative AI struggles with tasks that demand physical manipulation, creativity, and real-time adaptation. Building a prototype, conducting an experiment, or performing a complex procedure all fall into this category. These hands-on activities require problem-solving, critical thinking, and on-the-go adjustments.
  
  Practical assessments often encourage originality and showcase a student's unique approach. Generative AI, whose responses are based upon the datasets it has been trained on, finds it difficult to mimic this level of individual creativity and initiative.

  If using, consider:

  • the resource commitment to ensure the task is manageable for both students and instructors. 
• Capstone assessments

  Capstone assessments require students to demonstrate their ability to combine, understand and apply their knowledge and skills from across a discipline or a range of modules within their programme.

  Generative AI typically excels at specific tasks within a single domain, and the emphasis on connecting ideas across different areas and from perhaps diverse sources makes capstone assessments much more challenging for the tools to successfully respond. They can incorporate open-ended questions or tasks that also require skills in critical thinking and analysis.
  
  These areas challenge AI, which may struggle to grapple with nuanced problems or demonstrate genuine understanding beyond rote memorisation.
  

Try AI for yourself

The list above is not exhaustive. If in doubt about the reliability of your assessment, you can try out generative AI tools for yourself.  

Use your Sussex credentials to sign into Microsoft Copilot, which is data-protected. Other tools, such as ChatGPT or Claude are also free to use but will require you to sign up. Paid-for versions of these tools can be more powerful but many don't protect the data you enter.

Try entering a sample past exam question, problem set or writing task and see what it comes up with:

• experiment with additional prompts to see how much you can refine the output.  
• if few prompts are needed to achieve a good quality response, you will need to review your assessment 
• if a good quality response can be achieved, but only via effective prompting, consider how much knowledge of the subject is required to achieve such an outcome and whether generating a submission in this way sufficiently demonstrates the knowledge, understanding and skills you are trying to assess.  

Consider your current assessments critically:

• does the current design really measure what you want it to measure?  
• will a few minor changes to your existing assessment approach suffice?  
• if not, how else might you assess students’ learning?

Making changes to your assessment

Before making changes, discuss any proposed changes with your course convenor or Board of Study to ensure they align with and support a scaffolded approach to assessment design across the course.

• Minor changes

  If implementing minor changes, ensure all adaptions:

  • align with module learning outcomes 

  • are compliant with Competition and Marketing Authority (CMA) requirements. 

   

  Minor changes may also include giving students permission to use AI in an assistive role in the preparation of elements of their assessments. See the guidance on AI and academic integrity.  

  It's important to to talk with your students about the acceptable uses of generative AI and what would constitute academic misconduct. This can vary from module to module.

• Major changes

  Generative AI is a catalyst for redesigning assessment at the programme level. When considering major changes to assessments, module and course convenors and schools could consider updating learning outcomes and assessment criteria to reflect capabilities that graduates will need in an AI-enabled world. This can be supported via:  

  • curriculum design which focuses on assessment for (rather than ‘of’) learning, e.g. applying Authentic Assessment principles 

  • designing in opportunities for students to develop their feedback literacy, confidence in their own abilities and understanding of assessment expectations 

  • assessing the process by which a final product or submission was generated, rather than the product itself  

  • making assessments more flexible.  

   

  This includes creating space to develop students and staff digital literacy (specifically AI in this case).

Learn more

For an overview of assessment strategies, and their strengths and weaknesses, see the QAA advice on developing sustainable assessment strategies in the era of ChatGPT.

For options to consider when making changes to assessment, see UCL’s assessment menu of fifty learning activities that require higher order thinking and/or engage students critically with AI. 

Find out how to access help and support, and find additional links and examples of practice, on the main AI in Teaching and Assessment page.

See the University of Sussex Teaching with AI Communnity of Practice padlet for a wealth of up-to-date resources.

Explore our website for further guidance on curriculum and assessment design, check dates for workshops and contact your Academic Developer or Learning Technologist if you would like more support. 

See more from Artificial Intelligence in teaching and assessment