The Fairer Exam: AI in Assessment and Learning Analytics

Learning analytics can show that a student paused on a problem, revised an answer three times, or stopped submitting work. It cannot, on its own, establish why. The student may be confused, bored, ill, working offline, using assistive technology, sharing a device, or solving the problem in a way the platform does not record. Converting a pattern into a high-stakes judgment without context turns an educational signal into an administrative shortcut.

AI can strengthen formative feedback, help educators notice patterns, and reduce mechanical scoring work. It should not silently determine grades, placement, discipline, disability support, graduation, or a student’s potential. Those decisions require valid assessment design, professional judgment, relevant evidence, an accessible process, and a route for the learner to understand and challenge the outcome.

Define the educational decision first

Start with the learning claim. What knowledge, skill, reasoning process, or transfer should the assessment reveal? Specify the population, curriculum, language, testing conditions, accommodations, consequences, and acceptable uncertainty. A short practice quiz and a graduation examination should not share the same evidence standard.

Separate three uses:

• Formative use helps a learner or teacher decide what to do next.
• Summative use supports a judgment about achievement after instruction.
• Program analytics examines patterns across courses, cohorts, or services.

A model suitable for low-stakes hints is not automatically valid for summative scoring. A cohort trend should not be used to label an individual. Document the intended and prohibited uses before collecting data or training a model.

Map the human decision workflow

Trace how an observation becomes an action. A student produces work; the system captures events; an algorithm scores or classifies them; a teacher sees a recommendation; an institution may change instruction, support, grade, or status. At each transition record the evidence, uncertainty, authority, and appeal route.

Define which outputs are suggestions, which require educator confirmation, and which can trigger only reversible low-impact actions. A practice recommendation may be automated within a teacher-approved curriculum. A misconduct allegation, grade change, placement decision, or denial of service needs qualified human review and often additional evidence.

The U.S. Department of Education report Artificial Intelligence and the Future of Teaching and Learning recommends keeping humans in the loop and making systems inspectable, explainable, and overridable. It is policy guidance, not a validation of any product or a rule that “human in the loop” automatically makes a decision fair.

Build a defensible data lineage

Learning data can include responses, drafts, time stamps, clicks, hints, speech, video, attendance, accommodations, device state, and teacher notes. For each field record its source, educational purpose, time, transformation, access, retention, and whether it becomes part of an education record. Preserve the original work and rubric version behind a derived score.

Event logs are incomplete representations of learning. Offline reading, peer discussion, handwritten planning, anxiety, network failure, and accessibility tools may be invisible or misread. Do not infer motivation or character from platform activity. Use missingness as a condition to investigate, not evidence of disengagement.

Maintain version links among item, rubric, curriculum objective, model, prompt, scorer, and final decision. If a score is challenged months later, the institution should be able to reconstruct what evidence and rules were used.

Protect student privacy and agency

Collect the least data needed for the stated educational purpose. Avoid continuous webcam, emotion, location, or behavioral surveillance when less intrusive evidence can answer the question. Separate classroom support data from discipline, advertising, model training, and unrelated institutional analytics.

In the United States, FERPA regulations apply within their defined institutional scope and give parents or eligible students rights concerning education records, amendment, and disclosure. The Department’s student privacy resources also address education technology and third-party services. Other jurisdictions and education levels have different laws; a FERPA claim is not a global privacy assessment.

Publish plain-language notices explaining inputs, purpose, recipients, retention, automated processing, teacher role, and challenge process. Provide a workable alternative when a tool is optional. Contracts should control vendor use, re-disclosure, security, deletion, incident response, and model training.

Design assessment before adding AI scoring

Validity begins with tasks and rubrics. Sample the intended knowledge broadly enough, avoid construct-irrelevant barriers, and define acceptable responses. If writing quality is not the target, unnecessary language complexity may distort a science score. If reasoning is the target, final-answer exact match may miss genuine understanding.

Use AI to propose feedback or apply a rubric only after expert scoring guidance exists. Structured rubrics should identify required evidence, partial credit, misconceptions, and cases needing review. Keep model-generated explanations separate from the official score unless an educator verifies them.

Adaptive assessment needs a calibrated item bank, exposure control, content balancing, stopping rules, and monitoring. Selecting easier material after a noisy early error can trap a learner in a low-expectation path. Educators need a way to reset or override the sequence.

Evaluate scoring like a measurement instrument

Create a representative, securely governed evaluation set with expert labels and adjudicated disagreements. Include languages, grade levels, subject domains, response lengths, assistive technologies, nonstandard but valid methods, blank or corrupted work, and adversarial inputs. Keep students or classrooms from leaking across train and test where related work could inflate performance.

Report agreement with qualified human scorers, exact and adjacent-score agreement, error distribution, calibration, consistency across repeated conditions, and performance by relevant subgroup. Examine consequential errors separately: a one-point difference near a placement threshold matters more than the same difference in ungraded practice.

Human labels are not perfect ground truth. Measure inter-rater agreement, rubric ambiguity, and changes after adjudication. If experts disagree often, the automation target may be underspecified.

Evaluate learning impact, not only score matching

A feedback system can match teacher comments and still harm learning by giving answers too early, increasing dependence, narrowing strategies, or overwhelming students. Run studies that compare meaningful outcomes: revision quality, retention, transfer, help-seeking, self-explanation, teacher workload, and longer-term achievement.

Separate product engagement from learning. More clicks, time in an app, or completed hints do not prove understanding. Use appropriate comparison groups and analyze who benefits, who disengages, and whether gains persist outside the tool.

For tutoring applications, the operational boundaries described in AI education and tutoring agents are relevant: grounded curriculum, bounded tools, teacher visibility, and safe escalation.

Make accessibility part of validity

If the interface prevents a learner from perceiving, navigating, responding, or reviewing, the assessment may be measuring access barriers rather than the intended construct. Test keyboard use, screen readers, magnification, reflow, captions, timing adjustments, speech input, reduced motion, color independence, and error recovery.

WCAG 2.2 is a W3C Recommendation for web content accessibility, not a complete assessment-validity standard and not a substitute for accommodations or user research. Combine conformance checks with disabled learners and accessibility professionals. Track whether accommodations change model behavior or confidence.

Our overview of AI and assistive technology explains why an assistive signal should not be treated as suspicious behavior. Alternate input patterns must be represented in fraud and integrity evaluations.

Treat academic integrity as an evidence problem

AI-writing detectors and remote-proctoring flags are probabilistic signals, not proof of misconduct. False accusations can carry severe educational and psychological consequences. Do not use a single detector score, gaze pattern, typing rhythm, or network anomaly as the basis for discipline.

Define an investigation procedure with independent evidence, trained review, student notice, opportunity to respond, and appeal. Preserve the submitted work and tool version. Consider assessment redesign: oral explanation, staged drafts, source notes, in-class checkpoints, authentic projects, and reflection can provide richer evidence of learning than surveillance.

Generative AI policy should distinguish permitted assistance, required disclosure, prohibited substitution, and accessibility use. UNESCO’s guidance advocates a human-centered and age-appropriate approach; it does not create one universal classroom rule.

Give educators usable control

Teachers need more than a risk color. Show the underlying work, rubric criterion, relevant history, uncertainty, and alternative interpretations. Let educators correct a score, suppress a recommendation, change the learning path, or request more evidence. Capture a reason without turning override rates into a performance target against teachers.

Provide training on intended use, known limitations, privacy, accessibility, and escalation. Avoid automation that adds monitoring duties without reducing workload. Consult students, families, educators, support staff, and administrators during procurement and evaluation.

The broader principles in AI in education apply: technology should strengthen the relationship between learner and educator, not replace it with an opaque ranking.

Monitor KPIs with educational meaning

Useful measures include:

• scorer agreement and calibration by task and subgroup;
• rate and outcome of educator overrides and student appeals;
• time to feedback and feedback usefulness;
• revision, retention, and transfer outcomes;
• false integrity flags and investigation resolution;
• accessibility defects and accommodation parity;
• percentage of decisions with reconstructable lineage;
• privacy incidents, deletion completion, and unauthorized access;
• teacher time saved or added;
• differential assignment to remedial or advanced material;
• model abstention and manual-review backlog.

Set stop conditions for missing evidence, unsupported language, corrupted input, out-of-scope accommodations, excessive uncertainty, or broken audit logging. High-stakes decisions need stricter thresholds and smaller automation scope.

Rehearse failure modes

Test situations such as:

• a dialect or multilingual response receives systematically lower scores;
• a screen reader changes timing and triggers an integrity flag;
• missing homework caused by connectivity is labeled disengagement;
• an early adaptive error limits later access to advanced material;
• a model gives polished feedback that contradicts the curriculum;
• a vendor silently changes the scoring model mid-term;
• teacher overrides do not update reports shown to families;
• a generated comment reveals sensitive accommodation information;
• old data continues affecting recommendations after correction;
• an institution uses formative analytics for discipline.

Each failure needs an owner, detection signal, correction path, and communication plan.

Roll out from feedback to higher stakes

Begin with a low-stakes formative task where teachers can compare suggestions with student work. Establish baseline learning and workload measures. Run the model in shadow, then allow opt-in use with visible educator approval. Collect appeals and accessibility evidence before expanding.

Next, pilot at course level with a defined curriculum and diverse evaluation set. Keep official grades independent until reliability, validity, subgroup impact, privacy, and operational controls are demonstrated. Any move toward summative use should receive fresh assessment expertise, governance approval, and a stronger evidence standard.

Retain manual assessment, exportable records, model and rubric rollback, vendor exit, and a way to correct downstream reports. Review the use when curriculum, population, law, accommodations, or model changes.

AI can help educators see work sooner and offer more timely support. It becomes educationally responsible only when analytics remain evidence for professional judgment rather than a shortcut around it.

Source notes

Source status was checked on 2026-07-30. The U.S. Department of Education report Artificial Intelligence and the Future of Teaching and Learning is nonbinding 2023 policy guidance that emphasizes human-centered, inspectable, and overridable systems. The Department’s privacy and education technology resources and FERPA regulations portal are authoritative for their U.S. legal scope, not for every jurisdiction. UNESCO’s guidance for generative AI in education and research was last updated in January 2026 and presents global human-centered policy guidance. WCAG 2.2 is a W3C Recommendation for web accessibility; it does not validate an assessment or AI scorer.