We spent a decade asking whether automated decisions are accurate and fair. We skipped the simpler question: when one goes against you, can you actually do anything about it?
In 2013, Michigan introduced an automated system called MiDAS to detect unemployment-benefit fraud. Over the next two years it falsely accused roughly 40,000 people, with its error rate later estimated near 93%. Some people had wages garnished, faced financial hardship, or spent years trying to reverse decisions that had received little meaningful human review.
The case exposes a problem that extends far beyond one government system. We increasingly allow software to screen applications, flag transactions, rank candidates, moderate accounts, assess risk, and influence access to services. Yet the mechanism for challenging those decisions is often much weaker than the mechanism for making them. That distance between an automated decision and a practical way to contest it is the recourse gap.
Accuracy, Fairness and Recourse
Debates about automated decision-making often mix three different questions together, even though each describes a separate property of the system.
▪ Accuracy asks whether the system reaches the correct result. A fraud model should identify actual fraud, while a hiring system should distinguish suitable candidates from unsuitable ones with reasonable reliability.
▪ Fairness asks whether errors or outcomes fall disproportionately on particular groups. A system can achieve strong average accuracy while still performing worse for certain populations or relying on variables that act as proxies for protected characteristics.
▪ Recourse asks what happens after the system gets an individual case wrong. Can the affected person learn that automation influenced the decision, understand the reason, correct inaccurate information, reach someone with authority to reconsider it, and supply evidence supporting an appeal?
The distinction matters because accuracy and fairness are normally evaluated across many decisions. Recourse becomes important at the individual level. A system can be 99% accurate and still create a serious problem for the remaining 1% if those people have no realistic way to challenge the result.
How Automated Decisions Become Actions
An automated decision is rarely a single piece of software saying yes or no. It is usually a pipeline. A system begins with data. It may look at income, transaction history, location, account activity, words in a résumé, previous purchases, or hundreds of other variables. A model processes those inputs and produces a score or classification.
The consequence usually appears at the next stage: a decision threshold. A fraud system might generate a risk score of 0.82. Someone has already decided that scores above a certain level should trigger a review, block a payment, or suspend an account. That threshold converts a prediction into an action.
This distinction matters because the model is not always the source of the problem. Bad input data can distort the score. A poorly chosen threshold can turn minor uncertainty into an automatic rejection. A downstream workflow can give a model far more authority than its reliability justifies.
Automated systems also change over time. Models are retrained, thresholds adjusted, data sources replaced, and software updated. If the original version is not recorded, the system that rejected someone last month may no longer exist in exactly the same form when that person finally appeals.
Why AI Explanations Get Difficult
Some automated systems are straightforward to explain. A basic decision tree or points-based scorecard can show which factors were considered and how they influenced the result.
More complex machine-learning systems are different. Large ensembles and neural networks can base their outputs on interactions across huge numbers of parameters. They may produce a reliable prediction without containing a simple human-readable explanation for how that particular prediction emerged.
Explainable AI, commonly called XAI, attempts to close that gap. Techniques such as SHAP can estimate which features contributed most strongly to an output. A lending system, for example, might indicate that debt level and a short credit history had the greatest effect on a rejection.
For people challenging decisions, counterfactual explanations can be even more useful. Instead of explaining every internal calculation, the system identifies what would have changed the result. A message such as “the application would have met the threshold if verified income were $4,000 higher” gives the person something specific to inspect or correct.
The limitation is that these explanations are often additional tools placed around the model rather than direct windows into its reasoning. Companies must still choose to generate them, preserve them, and make them available when a decision is disputed.
Where Recourse Usually Breaks
A person rarely loses the ability to challenge an automated decision at one obvious moment. Recourse usually breaks gradually across several stages.
MiDAS demonstrates the pattern particularly well. People were not always given a clear explanation of how their cases had been assessed, human review had been heavily reduced, and contesting an accusation could require individuals to reconstruct old employment information after the system had already treated them as responsible.
| Stage | What useful recourse requires | What often happens instead |
| Notice | Clear disclosure that automation influenced the decision | A generic rejection or suspension message |
| Explanation | A specific reason the person can examine and dispute | A score, category, or vague reference to internal criteria |
| Human review | Someone with enough information and authority to change the result | Support staff who can only repeat what the system says |
| Evidence | Access to the records behind the decision while they still exist | Data controlled by the organization and retained for limited periods |
A failure at any one of these stages can make an appeal ineffective. When several fail together, the affected person ends up arguing against an outcome rather than dealing with someone capable of reconsidering the original decision.
When Human Review Is Mostly Cosmetic
Companies often respond to concerns about automation by saying that a human remains involved. That description can mean very different things.
In a genuine human-in-the-loop system, a person reviews an important decision before it takes effect and has authority to change it. In a human-on-the-loop model, people monitor the system and intervene when unusual cases appear. At the other end, automation acts independently and humans become involved only after a complaint.
The presence of a person does not automatically make the process meaningful. Automation bias makes the problem harder. People have a tendency to trust computer-generated recommendations, particularly when those recommendations arrive with confidence scores or authoritative-looking labels. A reviewer who sees “high risk” before examining the evidence may be more likely to confirm the machine’s judgment.
Useful human oversight therefore requires more than adding an employee to the workflow. Reviewers need access to the relevant information, an understanding of the system’s limitations, and genuine authority to disagree with it.
The EU AI Act recognizes this problem by requiring human oversight for certain high-risk systems and specifically addressing the danger of over-reliance on automated outputs. The practical challenge is making sure “human review” means real reconsideration rather than a procedural rubber stamp.
Why the Recourse Gap Keeps Growing
Automation scales far more easily than appeals. A software system can process thousands or millions of cases at very low marginal cost. Every serious appeal, however, may require a trained employee to inspect records, understand context, communicate with the affected person, and potentially reverse an earlier action.
This produces a structural imbalance. Organizations can automate decisions faster than they can build systems for reconsidering them. Responsibility can also become fragmented. One company may build the model, another supply the data, a third integrate the software, and a fourth use the final score to make a decision. When something goes wrong, the affected person may struggle to identify which organization can actually fix it.
Evidence creates another problem. Models change, logs expire, data is overwritten, and configurations are updated. A decision can happen in milliseconds while an appeal begins weeks later, by which time parts of the original technical record may already have disappeared.
Several major public-sector automation failures have shown what happens when those weaknesses combine:
| System | What went wrong | Main recourse failure |
| Michigan MiDAS | Tens of thousands were falsely accused of unemployment fraud | Automated decisions operated with weak human review |
| Dutch childcare benefits system | Thousands of families were wrongly treated as fraud risks | Families struggled to understand and reverse risk-based decisions |
| Australian Robodebt | Automated income matching produced large numbers of questionable debts | Citizens carried much of the burden of disproving automated claims |
The common problem was not simply that software produced mistakes. High-volume automation was deployed without equally strong mechanisms for identifying, reviewing, and reversing those mistakes.
The Law Is Starting to Demand Answers
Regulators have increasingly focused on notice, explanations, and human review because people cannot challenge decisions they do not know were automated.
Europe’s GDPR established some of the strongest early protections. Article 22 addresses decisions based solely on automated processing when they create legal or similarly significant effects. Related provisions give individuals rights connected to information about automated processing, while safeguards can include human intervention and an opportunity to contest an outcome.
European courts have also pushed against explanations that amount to little more than “an algorithm decided.” In the SCHUFA case, the EU’s highest court strengthened scrutiny of automated credit scoring and the role such scores play in consequential decisions.
The EU AI Act extends the framework for certain high-risk AI uses, including systems deployed in employment, credit, and essential services. It places additional emphasis on meaningful explanations and human oversight.
Regulation, however, cannot solve the problem by terminology alone. A company can technically insert a human into an automated workflow while giving that person little information or authority. The effectiveness of recourse ultimately depends on what the review process allows someone to do.
Recourse Depends on Keeping the Record
Every appeal assumes that the evidence behind the original decision still exists. Automated systems generate logs, timestamps, scores, sensor readings, model outputs, and other technical records. Those records can be surprisingly temporary. Logs may be automatically deleted, models retrained, settings changed, and data replaced as systems evolve.
Without a reliable record of the original decision, even an organization willing to investigate may be unable to recreate exactly what happened.
The issue becomes especially clear when software interacts with the physical world. Modern vehicles can record speed, braking, steering inputs, warnings, and information from driver-assistance systems. After a collision, those records may help reconstruct what occurred, but only if the relevant data and vehicle are preserved before information is overwritten, altered, or lost.
When the Machine Leaves a Trail
A modern collision can leave behind far more than skid marks and photographs. Vehicles may record speed, braking, steering inputs, warning signals, and activity from driver-assistance systems, while nearby cameras, mobile devices, and connected services can add their own timestamps to the sequence of events. The challenge is often not finding data, but working out which records are reliable and how they fit with the physical evidence.
That technical trail can become especially important when responsibility is disputed. The type of evidence a Louisville car accident attorney may examine can now extend beyond witness accounts and vehicle damage to electronically recorded information that helps establish what happened in the seconds before a crash. It is a useful example of a much broader problem with automated systems: once the original digital record is lost, altered, or overwritten, challenging what the technology did becomes considerably harder.
Building Systems People Can Challenge
The recourse gap is not an unavoidable consequence of AI. Much of it comes from product and workflow decisions, which means systems can be designed differently.
▪ Make contesting a decision part of the product. A visible option to report an incorrect automated outcome should lead to an actual review rather than a support maze that eventually returns the same machine-generated answer.
▪ Give reviewers authority as well as access. Human oversight works only when reviewers can inspect relevant information, understand why the system reached its conclusion, and reverse the result when the evidence supports doing so.
▪ Preserve decision provenance from the beginning. Systems should record the data used, model or software version involved, important settings, and resulting action at the time the decision occurs. Reconstructing this months later should not depend on memory or incomplete logs.
▪ Explain decisions in language people can use. A useful explanation should identify the specific factors that mattered and, where possible, indicate what could have changed the outcome. Generic statements about “internal criteria” provide little basis for an appeal.
▪ Put a real timetable around review. An appeal process loses much of its value if the affected person can wait indefinitely while evidence expires or consequences continue. High-stakes systems need defined response periods and procedures for preserving disputed records.
These measures add friction to automated systems, but that friction serves a purpose. Automation is powerful partly because it can make decisions extremely quickly. Recourse requires deliberately preserving enough human judgment and technical evidence to slow the process down when a particular decision deserves another look.
Verdict : An Unappealable Decision Is a Verdict
The biggest weakness in automated decision-making is not simply that machines can be wrong. Human decision-makers are wrong too. The difference is what happens after the mistake.
As AI becomes embedded in hiring, finance, public services, online platforms, and physical systems, organizations need to treat the ability to challenge a decision as part of the technology itself. Notice, useful explanations, real human review, and preserved records should be designed alongside the model rather than added after complaints begin.
A system can process millions of cases efficiently and still fail badly if the small percentage of people it misjudges have nowhere meaningful to go. The goal is not to stop automated decision-making. It is to prevent automation from quietly turning a fallible prediction into an unquestionable final answer. When software can say “no,” there should still be a practical way for a person to ask why, show that it is wrong, and get the decision changed.
