Governing Agentic AI: Practical Steps for Managing AI Agents Safely

Unlike generative AI, which typically supports individual tasks, agentic AI can coordinate activities across multiple functions.  If we consider AI agents as the different members of a football team, then the agents are the players, working together but with different roles (e.g., goalkeeper, defence, midfield attack), governed by the manager, to achieve the goal of winning the match.

Consider the use of AI in the onboarding of new employees in an organisation:

When used correctly, AI agents and agentic AI can be a business enabler.  Unlike general purpose AI tools, AI agents can be tailored to specific business objectives and workflows.  They can introduce significant benefits, such as increasing operational efficiency, aiding in decision making, improving customer experience and identifying trends.  

However, this comes at a trade-off.  As we have seen with organisations adopting increasingly advanced AI capabilities, greater autonomy is often accompanied by a greater potential for unintended outcomes.  Improper usage can significantly increase your organisation’s risk, as the complex interactions of the different agents can be difficult to predict, test and manage.  Without appropriate governance, these risks can lead to a range of adverse outcomes, such as unintended information disclosure, service disruption, reputational damage, and financial impacts, to name but a few.

AI governance is already crucial for organisations using generative AI seeking to manage AI-related risks effectively.  If you are looking to further increase your organisation’s AI capability through agents, strong governance becomes even more essential due to their greater autonomy and complexity, as well as their increased capacity to create widespread impacts if they fail or behave unexpectedly.

Often, organisations build agents using existing AI models instead of creating their own models from scratch.  However, taking this approach does not relieve your organisation of its responsibilities.  Although it would be easy to assume that governance of the models used is solely the responsibility of the model creators, your organisation is responsible for how it uses the model, i.e., how it behaves, what it can access, and what actions it can perform.

Critically, you need to govern the agent, not the model.  Whilst the same model can be used for both generative AI purposes (chat, drafting) and agentic systems (autonomous actions, decisions), these very different uses require separate governance.

In our earlier onboarding example, we had agents for onboarding, IT support, facilities, HR as well as the generative AI itself.  Each of these agents, even if using the same underlying model, have different access, instructions and capabilities, which in turn necessitate subtly different governance.

The first step is to understand your AI system landscape.  Start by identifying every agent involved in the agentic AI system and defining the scope of each one.  For each agent, establish what it is intended to do, who will use it or have access to it, which systems and information it needs access to, and which actions it can perform.  Then compare the access requested with the access genuinely required.  In our experience, this simple exercise often reveals unclear ownership, duplicated capabilities and permissions that are broader than the agent’s role requires.  As with an employee, an agent should only be permitted to access information and perform actions necessary for its role.

A useful lesson from governance reviews is not to rely solely on a central list of approved AI tools.  Speak to the teams designing workflows, ask which systems their agents can access and verify the permissions in practice.  The gap between an agent’s intended role and its actual access is often where the most significant risks emerge.

The table below draws on the issues we most frequently encounter when discussing or reviewing AI agent deployments with organisations.  It sets out why each issue matters and the practical measures that can help reduce the risk.

In practical terms, validation and/or verification should include routine tasks, unusual requests and situations where the agent should refuse, pause or escalate.  A technically successful response is not enough: reviewers should also confirm that the action was authorised, traceable and consistent with the organisation’s policies.  Starting with a limited user group makes it easier to observe behaviour and adjust controls before wider deployment.

Human oversight should be designed around decisions and consequences, rather than treated as a generic approval step.  Organisations should identify where a person must review an action before it occurs, where retrospective monitoring is sufficient and who has the authority and practical ability to stop the agent.  If intervention is too slow, too complex or assigned to someone without the necessary context, the control may exist on paper but fail in practice.

Before an agent is deployed, its accountable owner should be able to answer five questions: What is the agent permitted to do?  Which data and systems can it access?  When must it involve a person?  How will its actions be monitored?  What happens if it behaves unexpectedly or becomes unavailable?  If these questions cannot be answered clearly, the agent is unlikely to be ready for operational use.

The controls outlined above provide a practical start point.  However, based on our experience helping organisations implement AI governance programmes, aligning with established frameworks can provide valuable structure and consistency, as well as providing stakeholders with increased assurance of your governance arrangements’ effectiveness and maturity.

ISO 38507 adds an important governance perspective by providing guidance to governing bodies on how to oversee and govern the use of AI within their organisations.  For agentic AI, this is particularly relevant because greater autonomy and interconnected decision-making can create consequences that extend beyond individual systems or technical teams.  Governing bodies should therefore ensure that AI use remains aligned with organisational objectives, risk appetite, legal obligations and corporate values, while management establishes the processes and controls required to deliver that direction.

At the management-system level, ISO 42001 and the NIST AI Risk Management Framework (RMF) provide useful structures for organisations looking to govern AI and AI agents responsibly.*  Of particular note is that much of the information and control that needs to be established would be brought out in an AI system impact assessment (AIIA), the completion of which is a key requirement of ISO 42001.  These assessments can be completed in accordance with your organisation’s preferred approach, such as that defined within ISO/IEC 42005:2025.

*To learn more about these frameworks, their purpose, and the similarities and differences between them, read our blog on Artificial Intelligence Frameworks and Regulations: ISO 42001, the NIST AI RMF and the EU AI Act.

For organisations at an early stage in their AI journey, a proportionate first month could focus on four activities:

• Identify agent use cases and owners
• Record the systems, information and actions available to each agent
• Agree where human review or intervention is required
• Test one limited use case before wider deployment.

This will not replace a complete governance programme, but it will provide the visibility needed to prioritise the next steps.

AI agents and agentic AI have the potential to deliver significant business benefits, but their autonomy and interconnected nature introduce risks that organisations must understand and manage.  Our practical experience points to a simple lesson: governance is most effective when it develops alongside the use case, not after the technology has already become embedded.  Organisations that define ownership, boundaries, oversight and intervention arrangements at the outset are better positioned to realise the benefits of AI agents safely and sustainably.