Working Paper · Assessment Design · Human-Centered AI

Human-AI Augmented Instructional Design

A practitioner design study showing how AI-generated pretest responses can help instructors redesign case-study assessment questions so they assess course-specific reasoning, judgment, justification, and alignment rather than generic application.

Read overview Back to archive

Overview

From generic application to course-specific judgment

Generative artificial intelligence creates new challenges for case-study assessment because it can produce plausible responses when the question provides the framework, evidence, and expected response structure. This does not mean case-study assessment should be abandoned, but it does require instructors to reconsider what kind of learning evidence their questions produce.

This paper introduces Human-AI Augmented instructional design as a human-led process in which AI-generated pretest responses are used as diagnostic evidence to support assessment question redesign. In this approach, AI is not the author of the assessment and not an objective judge of quality. Instead, AI functions as a stress test that helps instructors see whether a question invites generic application or requires course-specific reasoning.

The paper uses a Peloton marketing case to examine how original exam questions could be answered by generative AI using only student-available materials. The responses were then analyzed to determine whether each question required students to select tools, compare alternatives, justify decisions, and connect their work to course-specific examples and processes.

Problem of practice

The generic application problem

Generic application occurs when a case-study question gives students, or AI, enough of the framework, evidence, and response pathway to produce a plausible answer without demonstrating the deeper reasoning the instructor intended to assess.

The framework is supplied

Questions often name the exact tool students should use, such as SWOT, buyer decision process, positioning, or consumer behaviour categories. This can reduce the need for tool selection or justification.

The evidence is visible

Case-study prompts often provide enough visible evidence for AI to match case details to familiar disciplinary concepts and generate a coherent answer.

The response path is predictable

When categories, stages, or evaluation criteria are obvious, the answer may appear strong without showing whether the learner can reason with course-specific tools, examples, or judgment.

Human-AI Augmented process

Using AI as an instructional design stress test

Human-AI Augmented instructional design is a human-led process. The instructor submits original case-study questions to generative AI using only student-available materials. The AI response is then reviewed to identify whether the question can be answered through general disciplinary knowledge and visible case evidence alone.

The goal is not to make questions artificially difficult or impossible for AI to answer. The goal is to determine whether the question is actually assessing the intended learning: reasoning, judgment, tool selection, comparison, justification, and connection to course-specific instruction.

AI is used as diagnostic evidence, but the instructor remains responsible for interpretation, fairness, constructive alignment, question redesign, and final pedagogical judgment.

Design moves

How questions are redesigned

The redesigned case-study questions preserve the broad learning goals but change the structure of the task so students must show course-aligned reasoning.

From framework application to framework selection

Instead of naming the exact framework in the question, students may be asked to rank qualified tools learned in class and justify the most relevant option for the case.

From description to evaluative judgment

Instead of describing every stage of a model, students may be asked to identify the weakest, strongest, most defensible, or most vulnerable point and justify their judgment.

From supplied categories to student-selected categories

Instead of giving students the categories to use, questions can require students to recall, select, compare, and apply the most relevant categories from course instruction.

From broad disciplinary language to course-specific reasoning

Instead of accepting general marketing language, redesigned questions require students to connect case evidence to processes, examples, tools, and criteria discussed in class.

From answer production to reasoning visibility

Ranking, comparison, justification, and recommendation prompts make student reasoning more visible and more assessable.

From AI-proofing to assessment alignment

The purpose is not to defeat AI. The purpose is to design assessments that more clearly produce evidence of intended learning outcomes.

Design principles

Principles for AI-era case-study assessment

AI answerability can reveal assessment-design vulnerabilities.

A strong AI response may show that the question provides too much of the framework, category structure, evidence, or response pathway.

AI failure can also be diagnostically useful.

A weak or misaligned AI response may show that the question depends on unstated classroom context, course-specific tools, or examples not visible in the prompt.

Case questions should assess reasoning, not only fluent output.

A polished answer is not always evidence of learning. Stronger assessment design makes interpretation, justification, and decision-making visible.

Course-specific reasoning should remain central.

Students should demonstrate the tools, examples, processes, and evaluative criteria emphasized during instruction.

AI should support instructor judgment, not replace it.

AI-generated responses are diagnostic artifacts. The instructor remains responsible for interpretation, fairness, alignment, and assessment quality.

The goal is not to make assessment obscure.

A better AI-era question is not simply harder. It is clearer, fairer, more aligned, and better able to assess the reasoning the course is designed to develop.

Application

Where this process can be used

This approach is especially useful for instructors who use case studies, applied assessments, business cases, scenario-based exams, professional judgment tasks, or written analysis questions. It can help instructors examine whether a question is producing evidence of the intended learning outcomes or simply inviting generic application.

The process can also support faculty development, assessment redesign workshops, course-level AI policy discussions, and institutional conversations about academic integrity and authentic assessment in an AI-saturated learning environment.

Research status

Limitations and future research

This paper is presented as a practitioner design study rather than an empirical study of student learning outcomes. It does not report student grades, student submissions, academic integrity data, classroom outcomes, interviews, or surveys.

The evidence consists of instructor-created case-study questions, AI-generated pretest responses, and the subsequent redesign of those questions. The paper therefore does not claim that the redesigned questions improved learning, reduced misconduct, or produced deeper student outcomes.

Future research could examine this Human-AI Augmented question pretesting process through prospective, ethics-approved classroom studies. Such studies could compare student responses to original and redesigned questions and examine whether redesigned questions produce richer evidence of student reasoning, stronger use of case evidence, more accurate application of course-specific concepts, or improved rubric reliability.

Collaboration

Interested in AI-era assessment redesign?

I welcome conversations about case-study assessment, faculty development, assessment redesign, academic integrity, AI pretesting, and human-centered instructional design.

Start a conversation View more ideas