Can the person affected actually challenge the decision?

Human oversight is not meaningful unless the person affected can understand the decision, introduce missing context and obtain a different outcome.

In today’s governance note, I want you to consider what happens after an AI-supported decision has been made.

A student has been flagged as at risk. An application has been deprioritised. An assessment has been questioned. Access to a course, support service or opportunity has been affected. Can the person challenge the decision?

Most organisations will say that a human remains involved. But the existence of a human reviewer does not by itself create meaningful oversight. If the reviewer sees the same system output, works from the same incomplete data and is expected to follow the recommendation, the appeal may simply reproduce the original decision. I would design the appeal route before introducing the system.

An effective appeal route should move through five distinct stages:

Begin by deciding what the affected person will be told. They need to know that AI influenced the decision and understand its role clearly enough to decide whether there is something to challenge. A generic statement that your institution “uses AI” does not explain what happened in their particular case.

The next question is who conducts the review. This person should have the authority, knowledge and institutional permission to question the system’s recommendation. They should not be asked merely to confirm that the process was followed. A meaningful review must also admit new information. Educational data will always be partial. Attendance, submission patterns, engagement metrics and previous performance may reveal something, but they do not contain the whole person. Illness, disability, caring responsibilities, inaccessible systems or a sudden change in circumstances may never appear in the data available to the model.

If the appeal process cannot accommodate that context, the institution is allowing the boundaries of its dataset to become the boundaries of its judgment. Finally, the outcome must be changeable. An appeal route that can explain a decision but cannot correct, reverse or remake it is a communications process, not an accountability mechanism.

The EU AI Act treats certain educational uses of AI as high-risk, including systems used to determine access or admission and systems used to evaluate learning outcomes. Its guidance also states that where a high-risk system assists decisions about a person, the affected person must be informed. In relevant circumstances, they may request a clear and meaningful explanation of the AI system’s role. European Commission guidance on navigating the AI Act, high-risk system guidance. These requirements provide a regulatory baseline. I would go further and treat contestability as part of institutional legitimacy.

Education routinely makes decisions that shape access, progression and future opportunity. Introducing AI does not reduce the institution’s responsibility for those decisions. It increases the need to show where judgment remains, how context can re-enter the process and who has the authority to say that the system was wrong. Do not wait for the first complaint to discover whether your appeal route works. Test it before deployment, including with cases in which the data is technically accurate but institutionally incomplete. Human oversight becomes meaningful only when a person can understand the decision, challenge its basis and obtain a different result.