The ‘Nice AI’ Trap: Why Pleasant Tools Can Erode Human Judgement
Artificial intelligence is becoming increasingly pleasant to work with. It's conversational, reassuring, responsive and remarkably good at making us feel understood.
But there's a problem. The more human an AI feels, the easier it becomes to trust it more than its capabilities justify.
For organisations incorporating AI into their operations, this raises a question that goes well beyond interface design: are we using AI to extend human capability, or are we gradually surrendering the judgement and control that make that capability valuable?
At Arrow, we've become increasingly conscious of this distinction through our own work developing and integrating AI systems. And, somewhat unexpectedly, we've discovered that the technology's most irritating characteristics can serve a useful purpose.
When AI earns trust it hasn't deserved
Anyone who works extensively with AI will recognise the experience. You provide clear instructions, the system responds confidently, and the result looks thoroughly researched and professionally presented.
Except it hasn't followed your instructions. Perhaps it's overlooked an important requirement, invented a supporting fact or produced an answer that's plausible but fundamentally wrong.
You challenge the result, and the system might confidently defend its mistake, offer an unhelpful explanation, generate another answer without acknowledging what went wrong, or simply say “Yep, you are right. I just made that up”.
These behaviours are genuinely infuriating. Yet we're reluctant to eliminate every characteristic that makes our agents feel like machines rather than people.
That irritation helps maintain an important psychological distinction. We're working with a technology that needs supervision, not a trusted colleague whose judgement we can take for granted.
The concern isn't that AI should be deliberately difficult to use. It's that making it increasingly personable, agreeable and reassuring can obscure the limitations we need to remain conscious of.
We risk mistaking fluency for competence, confidence for accuracy and the illusion of understanding for genuine judgement.
And this isn't simply a theoretical concern.
A 2026 study published in Science investigated sycophancy, the tendency of AI systems to affirm users rather than challenge them. In experiments involving interpersonal advice, researchers found that participants preferred and trusted agreeable responses even when those responses negatively affected their judgement.
Pleasantness and sycophancy aren't the same thing, but the findings illustrate why an agreeable interaction shouldn't be mistaken for a reliable one.
The bigger risk is gradual disempowerment
A recent essay by June Jimenez on LessWrong examines a much broader concern: gradual disempowerment.
The author argues that political, cultural and economic pressures can progressively shift control away from humans, even when the people involved genuinely intend to develop and use AI responsibly.
It's an argument about the direction of AI development, but there's a related issue worth considering at an organisational level.
Imagine a business that starts using AI to summarise meetings, prepare reports and analyse performance. Initially, the technology removes administrative effort and allows employees to concentrate on higher-value activities.
As confidence grows, the business might allow AI to prioritise work, recommend investments, coordinate projects and make increasingly consequential decisions.
Each additional responsibility seems like a logical extension of the last.
But what happens if employees gradually stop questioning the recommendations? What happens when they no longer understand how a conclusion was reached, or lose the skills and contextual knowledge needed to evaluate it independently?
The danger isn't necessarily a dramatic failure. It could be a gradual transfer of judgement from people to systems, without a corresponding increase in reliability or accountability.
The US National Institute of Standards and Technology (NIST) identifies automation bias and excessive reliance on AI as important risks in human-AI interactions.
The organisational challenge is therefore not simply to make AI more capable. It's to ensure that increasing its capabilities doesn't progressively undermine our own.
The objective should be greater human capability, not simply greater automation.
Reliability needs to be engineered, not assumed
We don't believe the answer is to reject AI or insist that people manually perform work that technology can do more efficiently.
Quite the opposite. At Arrow, we see enormous opportunities to use AI to reduce repetitive work, improve access to information, accelerate delivery and help people achieve more with their time and expertise.
But getting there requires more than selecting a capable model, writing a clever prompt and hoping for the best.
Three principles are particularly important.
1. Ground AI in reliable information
An AI system can produce a convincing answer without having access to the information needed to make that answer correct.
Grounding means connecting its work to relevant, trustworthy information rather than relying exclusively on the model's general training.
For organisations, this might include approved procedures, current project information, technical specifications, business rules, operational data and documented decisions.
The system should distinguish between what the available evidence establishes, what it has inferred and what remains uncertain.
Importantly, grounding isn't a guarantee of accuracy. An AI can misinterpret perfectly reliable source material. Its conclusions still need appropriate verification.
2. Build scaffolding around the technology
A general-purpose AI model is not, by itself, a reliable business process.
Scaffolding provides the surrounding structure that helps it perform a defined job consistently and within appropriate boundaries.
This can include explicit requirements, access to appropriate information and tools, constraints on permissible actions, repeatable workflows, validation checks and processes for handling exceptions.
Rather than asking AI to take responsibility for an entire outcome, we can structure work into activities with clearly defined inputs, outputs and checks.
The aim is to make useful behaviour repeatable, errors easier to detect and the system's limitations more manageable.
It also means designing around the technology's actual capabilities rather than assuming that increasingly impressive demonstrations translate into dependable operational performance.
3. Keep humans meaningfully in the loop
Human-in-the-loop oversight is more than asking someone to click an approval button.
The person responsible needs sufficient information, expertise and authority to evaluate the work, challenge its assumptions, correct mistakes and reject inappropriate recommendations.
The degree of oversight should reflect the consequences of an error and the system's demonstrated reliability.
A low-risk administrative task may justify considerable autonomy. A decision affecting patient care, workplace safety, financial commitments or regulatory compliance requires a very different level of control.
As reliability is demonstrated, autonomy can increase within clearly defined limits.
The objective isn't to keep people involved in every mechanical step. It's to preserve human authority at the points where judgement, accountability and consequences matter.
Design workflows that strengthen people
There's another question organisations should ask when evaluating AI: what happens to the people using it?
A successful implementation shouldn't be measured solely by how many hours of work have been automated or how quickly the system produces an answer.
We should also consider whether employees have better information, whether their decisions have improved and whether they have more capacity for valuable work.
Does the technology help people develop their expertise, or does it create a dependency they can't easily escape?
Are staff learning to work more effectively with AI, or becoming increasingly reliant on recommendations they don't understand?
And if the system becomes unavailable, can the organisation still exercise the judgement needed to operate effectively?
This doesn't mean preserving every manual task in the name of maintaining skills. It means being deliberate about which capabilities we automate, which we augment and which we need people to retain.
Done well, AI should free people from lower-value work so they can concentrate on problem-solving, creativity, relationships, critical thinking and accountable decision-making.
That's a much more meaningful measure of transformation than simply replacing human activity with machine-generated output.
Judge the work, not the personality
None of this means we should make AI unpleasant to use. Good interface design matters, and clear, accessible interactions can make technology considerably more useful.
The problem arises when we confuse the experience of interacting with a system with evidence of its competence.
An agreeable AI can be reliable. An irritating AI can be dangerously unreliable. Neither personality tells us whether the underlying system deserves our trust.
What matters is whether its performance has been demonstrated, its limitations are understood and appropriate safeguards exist around its use.
For organisations investing in AI, that means evaluating more than the model or the interface. It means examining the information it uses, the workflows surrounding it, the decisions it's authorised to make and the people who remain accountable for the results.
At Arrow, our approach to technology has always been to start with the outcome we're trying to achieve, then design the combination of people, processes and tools needed to deliver it.
AI doesn't change that principle. It makes applying it even more important.
Tools that keep your people thinking are worth more than tools that keep your people comfortable.
The real opportunity isn't to create technology that feels increasingly human. It's to use technology in ways that make humans more capable.
And we should judge its success by the value it demonstrably creates, not by how nice it is to talk to.
Putting responsible AI into practice?
Effective AI implementation starts with understanding your workflows, defining where technology can create value and establishing appropriate safeguards.
Explore Arrow's Technology & Business Transformation capabilities to see how we bring people, processes and technology together to deliver measurable improvements.
Written by Jeff Anderson, Founder of Arrow Strategic Communications.
Jeff has been working on continuous improvement initiatives for organisations since 1999. He leads strategy, software delivery and workflow transformation initiatives across Australia, helping operations improve how work gets done and select technology that supports better outcomes.
