Why do teams give up on AI automation after one mistake?


People forgive a colleague’s mistake much faster than a machine’s, so one error is often enough for a team to drop AI automation that works. What helps is keeping a person close and letting the team adjust how the automation works.
I recently gave a webinar about AI automation and the people who work next to it. Most of the hour went to one question: why do teams give up on automation that works?
People forgive a colleague’s mistake. When a machine makes the same mistake, they decide it doesn’t work. Researchers at Wharton showed this in five studies: people who saw an algorithm make a mistake lost confidence in it faster than in a human who made the same one. They avoided the algorithm even after seeing it do better than the human [3].
…once people see an algorithm make a mistake, they expect it to keep making mistakes.
Berkeley Dietvorst, who led the Wharton studies, explains that once people see an algorithm make a mistake, they expect it to keep making mistakes. With a person, we assume they’ll learn. His co-author Cade Massey points out that this second assumption is often wrong: in many jobs, people keep making the same mistakes too [8].
No automation is perfect, and plenty of systems you used this year still keep a person close. Three of them show how it works.
Passport gates compare your face with the photo on your passport chip. In the UK, three kinds of Border Force officers work around them: one watches the queue for people who need help or a closer look, one supervises the gates on a screen and can step in, and one checks everyone the gates send to a desk. An inspection report describes these roles as a way to make up for the automation [6]. The gates are quick at matching a face to a photo. Spotting an imposter or a child travelling alone still takes a person.
The inspection report is clear that the officer watching the screens isn’t there to mind the machines. They still look out for safeguarding, customs and security problems that the gates aren’t built to catch [6].
Spam filters catch most junk, but they also catch real mail. Roughly one in ten marketing emails never reaches the inbox, and more than three in four people check their junk folder at least once a week. Almost half check it at least once a day [5]. We let the filter do the sorting and still check its work.
Autopilot has flown planes for decades, and there are still pilots in the cockpit. Most passengers want it that way. In a 2017 UBS survey of 8,000 people in the US, Europe and Australia, only 17% said they would fly on a pilotless plane, and about half said they wouldn’t buy a ticket for one even if it were cheaper [2].
17% said they would fly on a pilotless plane, and about half said they wouldn’t buy a ticket for one even if it were cheaper.
In all three, the automation does the routine work and a person stays close for the cases it can’t handle. It’s the same rule I use for deciding what to automate: if a task is the same every time, let technology remember it. If it needs judgment or care, keep it human.
Your team probably doesn’t have a border officer or a pilot watching over its automation. So how do you stop people from giving up on it after the first mistake? The same Wharton researchers tested one answer: let people adjust it.
We love what we help build, so give people a way to change how the automation works. In their follow-up study, people could adjust an algorithm’s forecasts by a few points. In the first experiment, use of the algorithm went from 32% to 73%. How much people were allowed to adjust it hardly mattered [4]. A little control was enough.
How much people were allowed to adjust it hardly mattered. A little control was enough.
The IKEA effect points the same way: people value things they helped build, but only when the effort leads somewhere [9]. So if your team suggests a change, tell them what happened with it.
This is the answer to the forgiveness problem. People who could adjust the algorithm were more satisfied with it, and they lost less confidence in it after seeing it make a mistake [4].
Why does a little control help so much? People tend to see an algorithm as rigid: a person can learn from a mistake, but an algorithm will just make it again [4]. Being able to correct it puts the learning back in a person’s hands. A 2021 study came at it from the other side: when people saw that an algorithm could learn, its mistakes stopped costing it as much trust [1]. So if your automation gets better over time, show your team.
Most of these studies were lab experiments in the US, with forecasts, storage boxes and Lego sets. The Wharton studies used simple forecasting models, not AI. Newer research on AI chatbots shows the same thing, though: trust dropped right after an error, and people kept doubting the tool even after it was accurate again [7]. The passport and email figures are from the UK, and the pilot survey is from 2017. The results line up, but every team is different, so treat this as a starting point.
Sources
Berger, B., Adam, M., Rühr, A. and Benlian, A. (2021). Watch me improve: algorithm aversion and demonstrating the ability to learn
CNBC (2017). Who would be willing to fly in a pilotless plane? Hardly anyone (UBS survey)
Dietvorst, B., Simmons, J. and Massey, C. (2015). Algorithm aversion: people erroneously avoid algorithms after seeing them err
Dietvorst, B., Simmons, J. and Massey, C. (2018). Overcoming algorithm aversion: people will use imperfect algorithms if they can (even slightly) modify them
DMA (2023). Consumer email tracker series: what’s in your junk folder?
Independent Chief Inspector of Borders and Immigration (2023). A re-inspection of ePassport gates
Journal of Cognitive Engineering and Decision Making (2025). Mitigative strategies for recovering from large language model trust violations
Knowledge at Wharton (2015). Confidence games: why people don’t trust machines to be right
Norton, M., Mochon, D. and Ariely, D. (2012). The IKEA effect: when labor leads to love

Edita Dermontaité
Hands-on CPO and product leader who still builds (with AI)
I’ve often been the first: the first product hire at Leadcamp and Bizzy, where I led product as CPO for five years, and the first user researcher at Lighthouse, where I spent two years building the research function.
For the past year I’ve worked as a fractional CPO and hands-on product leader, helping B2B SaaS teams between traction and scale connect what engineering builds to what brings in revenue.
I started out as a graphic designer, so I still design and test things myself, these days mostly with AI: in Claude and Lovable, or straight in code with Cursor.
Book a free 30-minute call, or a coffee in Ghent. We’ll look at what’s slowing your product down and what to do about it. I work with teams fractionally or hands-on as an embedded product leader, 2–5 days a week, depending on your needs and my availability.
Product leadership for B2B SaaS between traction and scale, from a hands-on CPO and product leader who still builds (with AI). I take product decisions off the founder’s shoulders and get product, engineering and sales moving in one direction: revenue.

New North BV
BE 1025.547.940
9050 Ghent
Belgium