Stop the trAIn, I want to get off!

The artificial intelligence revolution has arrived with the subtlety of a freight train at full throttle. Across boardrooms and IT departments worldwide, there's a palpable sense of urgency to "do something with AI.” The fear of being left behind is real, the competitive pressures are mounting, and the promise of efficiency gains is tantalising. But in this headlong rush toward an AI-enabled future, are we moving too fast for our own good?

The evidence suggests we are. An increasing number of companies are making high-profile AI powered mistakes. Both Taco Bell and McDonald's ended their AI drive-through programmes after viral videos showed their systems either struggling with genuine orders or being forced into meltdowns by pranksters. Meanwhile, Air Canada was ordered to pay damages after its chatbot gave false information about bereavement fares to a grieving customer – it learned the hard way that when an AI assistant speaks in your name, you own its words. These examples aren't edge cases; they're warnings.

The pressure on boards and executives is understandable. Shareholders demand innovation, competitors announce AI initiatives weekly, and technology vendors promise transformative results. IT leaders face a particularly acute dilemma: appear visionary by embracing AI quickly, or risk seeming out of touch with technological progress. This creates a perfect storm where the question shifts from "should we adopt AI?" to "how fast can we deploy it?"

Yet this velocity carries significant risks. The gap between experimentation and production use has become dangerously narrow, with many organisations treating AI implementation as simply another software deployment rather than the fundamental business transformation it represents.

The Case for Strategic Gates

What's needed urgently is a structured approach with clear gates between experimentation and production deployment. These aren't bureaucratic hurdles designed to slow innovation, they're essential checkpoints that separate successful AI adoption from expensive failures.

The first gate must ensure the business case is not just written but genuinely understood. Too many AI projects begin with a solution searching for a problem, driven by technology enthusiasm rather than business need. A robust business case should articulate specific outcomes, quantifiable benefits, and realistic timeframes. More importantly, it should identify what success looks like and how it will be measured. Without this foundation, organisations risk investing heavily in technology that delivers marginal or unmeasurable value.

The second critical gate is the deliberate design of “humans in the loop.” Too many AI rollouts treat human oversight as a comforting phrase rather than an engineered control. The right question is not “will a human be involved?” but “where must a human be involved, and why?” There are moments in a workflow where AI is a helpful assistant - drafting, summarising, classifying, suggesting. There are other moments where AI becomes a decision-maker in all but name: prioritising who gets attention, which customers are flagged, which candidates are filtered out, which claims are denied. Those moments demand explicit review points, clear accountability, and an operating model for overrides, otherwise humans drift into becoming passive validators of machine output.

The Ethics Imperative

Perhaps most crucial is the ethical review gate. This examination should be undertaken with rigor and honesty, addressing fundamental questions that many organisations would prefer to avoid. What decisions are we actually devolving to AI? When we automate a hiring screening process or a loan approval decision or a customer service interaction, we're not just improving efficiency, we're delegating judgment that carries real consequences for real people.

The issue of bias in AI systems is well-documented but insufficiently addressed in practice. Amazon discovered this when its AI recruiting tool systematically downgraded resumes from women, effectively automating gender discrimination in hiring before the project was scrapped. Apple Card faced public backlash when its credit algorithm gave women lower credit limits than their spouses with similar financial profiles. IBM Watson for Oncology made unsafe treatment recommendations due to flawed training data that reflected narrow patient populations. Every AI model reflects the biases present in its training data, whether those biases are obvious or subtle – and in very large data sets they can be very, very subtle.

An ethical review must scrutinise data sources, interrogate assumptions, and seek out potential discriminatory outcomes. If an AI system is making or shaping decisions that affect people’s livelihoods, access to services, or legal rights, then the organisation deploying that system must be able to explain its reasoning, measure its error modes, and demonstrate that it has actively tried to prevent predictable harms.

This isn't a one-time exercise but an ongoing commitment and forms a fourth critical gate. AI models aren’t “set and forget.” They can degrade as data changes, usage evolves, or adversarial behaviour emerges. Generative systems can produce fluent nonsense, and non-generative systems can make brittle decisions when the world shifts. A production AI capability therefore requires monitoring that is as operationally real as uptime monitoring for critical infrastructure.

The Chicago Sun-Times and Philadelphia Inquirer learned this lesson when their AI-generated summer reading list recommended books that didn't exist, an embarrassing failure that required public apologies and damaged their credibility. Deloitte Australia had to return $290,000 in fees after one of its reports was found to contain fabricated citations and court quotes that were never spoken. Sports Illustrated faced a major scandal when it was revealed the storied magazine had published articles written by AI-generated authors with AI-created headshots, fundamentally undermining its journalistic credibility.

Organisations need clear protocols for identifying when AI has strayed from reliability, and these protocols must be tested before deployment, not discovered through customer complaints or business failures.

The Human Cost

Beyond technical considerations lies perhaps the most important question: what happens to the workforce? The impact on human workers cannot be treated as an afterthought or an unfortunate externality. When AI is introduced, jobs inevitably change. The optimistic scenario involves employees freed from repetitive tasks to focus on higher-value work. But is this what actually happens?

The Commonwealth Bank of Australia provides a cautionary tale. In 2024, the bank replaced a 45-person call centre team with AI voicebots, believing the technology could reduce call volume by 2,000 calls weekly. Instead, call volume remained unchanged, and the bank scrambled to offer overtime to remaining workers while pulling managers from other roles to answer phones. Just one month later, facing union pressure and public criticism, CBA issued an apology and offered to rehire the displaced workers, admitting they had "not adequately considered all relevant business considerations."

In many cases, the reality is more troubling than CBA's swift reversal. Workers find themselves relegated to verifying AI outputs—spending their days checking the machine's homework rather than applying their expertise directly. This creates a peculiar paradox: employees need deep expertise to effectively evaluate AI decisions, but they're no longer using that expertise in primary work, potentially leading to skill degradation over time.

When workforce reduction is part of the business case, and let's be honest, it often is, organisations have an ethical obligation that extends beyond the legal minimum. Are measures in place to reskill affected workers? Training programs sound good on paper, but are they genuinely designed to give displaced employees competitive skills in the current job market? Or are they box-ticking exercises that create the appearance of responsibility without the substance?

The Long-Term Talent Crisis

Looking further ahead, there's a brewing crisis that few organisations are addressing: the erosion of foundational skills in their future leaders. This phenomenon is already visible in professional services firms, where junior staff who once spent years developing expertise through hands-on work now oversee AI tools instead of doing the work themselves.

The problem is insidious. Today's junior and middle managers are tomorrow's senior leaders. If they've never deeply engaged with the fundamental work of their function—if they've managed AI outputs rather than developed core competencies—what happens when they reach positions requiring strategic judgment built on years of practical experience? Earlier this year LexisNexis released a report entitled “The Mentorship Gap”, which found that more than 70% of senior law partners are worried that junior lawyers are losing the ability to develop “legal reasoning.”

We risk creating a leadership class that understands processes abstractly but has never wrestled with their complexities directly. They'll know how to prompt an AI system but not how to think critically when that system fails or when it encounters situations beyond its training – like pilots who don’t know how to fly the plan when the autopilot breaks. This isn't a hypothetical concern—it's an emerging reality in fields from accounting to legal services to software development.

Building the Right Foundation

None of this is an argument against AI adoption. The technology offers genuine benefits and will undoubtedly transform how we work. But transformation requires foundation and guardrails, not just velocity.

The financial and reputational stakes are substantial. Binance was fined $1 billion in 2023 after its AI systems misclassified over 1.6 million restricted transactions, allowing trades with sanctioned regions. Bittrex paid $29 million for similar compliance failures. These aren't just technology failures, they're governance failures. According to research from Harvard Law School, 72% of S&P 500 companies now disclose at least one AI-related reputational risk in their SEC filings.

Organisations need comprehensive AI governance frameworks before scaling deployment. These frameworks should address technical standards, ethical guidelines, human oversight requirements, and workforce impact considerations. They should establish clear decision rights: who can approve AI deployments, what review processes are mandatory, and how concerns are escalated and resolved.

Investment in these foundations isn't a cost, it is insurance against far more expensive failures. A poorly designed AI system that discriminates against customers, makes erroneous decisions at scale, or creates workforce crises will cost far more than the time and resources required for proper governance upfront.

Moreover, companies that get this right will gain competitive advantages. They'll build trust with customers who increasingly scrutinise how businesses use AI. They'll retain talent in a market where workers are wary of AI-driven job insecurity. They'll develop institutional knowledge about effective AI deployment that becomes a strategic asset.

Conclusion

The AI train is indeed moving fast, and no organisation can afford to ignore it. But the answer isn't to leap aboard without looking. It's to approach AI adoption strategically, with eyes open to both opportunities and risks.

By establishing clear gates between experimentation and production, ensuring robust business cases, building in human oversight, conducting thorough ethical reviews, and addressing workforce impacts honestly, organizations can harness AI's power while avoiding its pitfalls.

The companies that will ultimately succeed aren't those that deployed AI fastest, but those that deployed it most thoughtfully—with proper foundations, appropriate guardrails, and genuine consideration for the technology's impact on customers, employees, and society. That's not slowing down the train. That's making sure it's on the right track, heading toward a destination worth reaching, with all passengers accounted for.

The urgency is real, but so is the responsibility. It's time to ensure we're building AI systems that serve our values, not just our velocity.