AI Governance in Regulated Business: What Leaders Must Do Now

Artificial intelligence has moved from a technology discussion to a compliance discussion. If your regulated business is using AI tools — for customer monitoring, risk assessment, EDD, fraud detection, or any other regulatory function — and you cannot demonstrate meaningful governance over how those tools operate, you are sitting on an increasingly visible risk.

Regulators haven't issued fully prescriptive AI rules in most sectors yet. But across financial services, gaming, and healthcare, the direction is consistent: regulators are signalling clearly that they intend to act, and the businesses that wait for detailed guidance before building governance will be the ones facing the hardest questions first.

What "AI Governance" Actually Means for a Regulated Business

AI governance is not a technology exercise. It is a risk management discipline that answers a straightforward set of questions: What decisions is AI making or influencing? Who is accountable for those decisions? How do we know the AI is performing as intended? And what happens when it gets it wrong?

In regulated environments, the decisions AI touches are often directly regulatory in nature. In gaming, this means flagging a player for responsible gambling intervention or escalating a customer for EDD review. In financial services, it means credit decisions, transaction monitoring alerts, and fraud scoring. In healthcare, it means triage recommendations and patient risk stratification. When an algorithm drives these decisions and something goes wrong — a vulnerable individual not identified, a false positive triggering a discriminatory outcome, a SAR filed on incorrect grounds — the regulator will ask: what governance did you have in place?

"The vendor told us it worked" is not a governance answer in any of these contexts.

The Regulatory Risk of Unaudited AI

The FCA's AI Framework, published in 2024, set out detailed expectations around accountability, explainability, and fairness that apply across every FCA-regulated firm — from banks to payment institutions to insurance providers. The UKGC has been explicit that gaming operators cannot outsource regulatory accountability to technology. The CQC has begun engaging with how AI-assisted decision-making in healthcare settings must be governed and audited.

The core concern is universal: AI tools that operate as black boxes — producing outputs without auditable reasoning — are incompatible with regulated environments where organisations must be able to evidence their decisions. If an AI system flags a customer and you act on that flag, you need to be able to explain the basis for that decision to the customer, to an internal auditor, and to a regulator. If the answer is "the AI said so," that is not sufficient.

Beyond explainability, there are bias risks that apply across all regulated populations. AI models trained on historical data can encode historical discrimination — against certain demographics, payment methods, or behavioural patterns that correlate with protected characteristics. In any jurisdiction where anti-discrimination obligations apply alongside sector-specific regulation, an unvalidated AI producing systematically biased outputs is a dual liability.

Key Governance Requirements

A defensible AI governance framework for a regulated business should cover at minimum:

Inventory and classification. You need to know what AI tools you are using, which decisions they influence, and what the risk level of each application is. A product recommendation engine is not the same risk as a money laundering detection model or a patient triage tool.

Explainability requirements. For any AI tool influencing a regulatory decision, you should be able to produce a plain-language explanation of how the decision was reached. This means working with vendors to understand their models — and being willing to exclude tools that cannot provide this.

Bias and fairness testing. Before deploying an AI model in a regulated function, you should conduct and document testing for bias across protected characteristics. This should be repeated whenever the model is updated or your customer population changes materially.

Human oversight mechanisms. AI should augment human judgement, not replace it, in regulatory decision-making. Your governance framework should specify the points at which a human must review an AI-generated decision before it is acted upon — particularly for customer-facing interventions.

Audit trails. Every AI-influenced decision in a regulated context should generate a log: what input data was used, what the output was, what action was taken, and who reviewed it. These logs need to be retained in line with your record-keeping obligations.

Vendor due diligence. If you are using third-party AI tools — which most organisations are — you need documented assurance from vendors that their models are regularly validated, that you will be notified of material changes, and that they can support your governance obligations. A vendor without governance documentation is a vendor you should think carefully about using for regulated functions.

What Regulators Are Signalling Across Sectors

The FCA's AI Framework sets the direction for financial services. The UKGC has been clear that gaming operators cannot outsource regulatory accountability to technology. Across healthcare, the CQC is developing expectations around AI-assisted clinical decisions. The common thread is accountability: regulators expect to be able to identify who is responsible for an AI-influenced decision and what governance sat behind it.

The direction of travel across all these sectors is toward organisations being required to demonstrate not just that they use technology, but that they govern it. "We use an AI system" will need to be followed by "and here is how we know it works as intended, here is how we test it, and here is who is accountable for it."

Building a Proportionate Framework

AI governance doesn't need to be bureaucratic to be effective. A proportionate framework for most regulated businesses would comprise:

- A register of AI tools and their regulatory risk classification - Documented explainability requirements per risk tier - A validation schedule (at minimum annual for high-risk applications) - A named accountable owner for each high-risk AI application — typically the MLRO for financial crime tools, the compliance lead for customer risk tools - A vendor questionnaire that covers governance, explainability, and change notification - Staff training so that people using AI-assisted tools understand their oversight role

The businesses that build this now will find it straightforward to demonstrate compliance when prescriptive rules arrive. The ones who wait will be retrofitting governance into deployed systems under regulatory pressure — which is always harder, and always more expensive.

The Risk of Vendor Inertia

One of the most common challenges I encounter across sectors is organisations that want to govern AI properly but are blocked by vendors who cannot or will not provide the information needed to support governance. This is itself a risk signal. If a vendor building tools for use in a regulated environment cannot explain how their models work, cannot provide validation evidence, and cannot support your audit obligations — that vendor is creating regulatory exposure for you.

When you negotiate AI tool contracts, governance obligations should be in the contract. Not as an afterthought, but as a core commercial condition. Vendors who want regulated business clients need to meet regulated business standards.

The technology is not going away. The regulation is coming. The governance gap is closeable — but only if you start now.