A joint follow-up by Lawrence Advocaten and Hunt & Hackett to the three-part series on shadow AI.
The first three parts of this series focused on how organizations can manage shadow AI from legal and organizational perspectives. Policy, approved applications, clear responsibilities, and careful procurement are essential for this. However, governance primarily determines what an organization wants to allow. It does not establish what is actually happening from a technical standpoint.
An approved application may reach more information than intended. A policy rule prescribing what AI may or may not do does not constitute a technical security boundary. This is precisely where governance and security converge: the technical configuration must make visible what is truly happening, enforce where the boundaries lie, and demonstrate that those boundaries work in practice.
This becomes more urgent now that AI is shifting from standalone chatbots to applications embedded in SaaS platforms, browsers, development environments, and business applications, and are linked to internal data, APIs, and other tools. At the same time, AI agents are emerging: applications that can independently plan or execute tasks and access data, tools, or other systems to do so. In this context, NIST emphasizes that access to diverse data, tools, and applications imposes new requirements on identification and authorization(1).
As a result, the question is shifting from what information employees provide to AI to a broader question: where is the AI located, under what identity does it operate, what data can it reach, what is it connected to, and what can it do?
From input to access, connection, and action
With shadow AI, an employee personally provides information to an application. Integrated AI can also retrieve information from corporate sources to which the employee or application has access. Consequently, existing rights become relevant in a new way. Before AI, an employee with access to thousands of documents could already reach that information, but could not easily search through, combine, and interpret everything. An AI application can.
Many underlying security issues are therefore not new. Overly shared documents, broad SaaS rights, poorly managed service accounts, and deficient logging already existed. However, AI can make it much easier to exploit existing access and combine information from different sources. When AI is subsequently connected to systems via APIs, connectors, or other links, the step toward action is taken. An AI agent can, depending on its privileges, not only retrieve and analyze information but, for example, also send an email, modify a record, or execute code.
Therefore, an organization must look beyond just the model or the application: under what identity does AI operate, what information can it reach, what is it connected to, and what actions can it perform?
The actual attack surface surrounding AI: six questions
Before measures are chosen, it must first be clear how AI actually runs throughout one's own environment. Six questions help to clarify this.
1. Where is the AI actually located?
This sounds simple but is becoming increasingly difficult to answer. Which public AI services, coding assistants, local models, AI features within SaaS platforms, API connections, and AI agents are actually in use?
A formal application register does not necessarily provide the full answer. AI is appearing more and more as a functionality within existing software, while developers and other technically savvy employees can relatively easily link new models, tools, and agents.
The Cloud Security Alliance now explicitly mentions detection of shadow AI as an additional measure. Technical monitoring can make unauthorized use of AI applications and external models visible. Data classification and Data Loss Prevention (DLP) help to signal and limit unwanted data transfer. NIST is also developing its Cyber AI Profile based on the idea that existing cybersecurity practices must be expanded now that AI is becoming increasingly intertwined with systems and business processes(2).
Inventory cannot, therefore, be solely administrative. Technical visibility is needed to determine what is actually being used and to be able to consciously assess and control that use.
2. Under what identity does AI operate?
This question becomes especially important once AI starts acting on behalf of people or systems. Does an application use the employee's identity, delegated user rights, its own identity, or an existing service account? And who can revoke that access? The identity used largely determines the scope the application receives.
Microsoft(3) advises treating an AI agent as an independent digital identity, with a clear purpose, an owner, and only the rights necessary for the task. NIST(5) is investigating along the same lines how existing principles for identification, authentication, and authorization should be applied to AI agents.
Many of these principles come from identity and access management. What is new is primarily that software can independently combine rights across multiple systems. As a result, the effective reach of an AI agent can be much larger than any individual permission implies(5).
The question, therefore, is not only what rights an identity has, but what effective reach is created when an AI agent can combine those rights.
3. What data can AI actually reach?
An organization may use a carefully selected enterprise solution and have made sound contractual agreements with the vendor. This does not automatically eliminate problems in its own data access.
Suppose an application uses an employee's existing rights. Then that application can also find information that the employee formally has access to, but of which they might barely have been aware without AI. Old SharePoint rights, overly shared folders, and historically grown access within SaaS platforms can suddenly become much more relevant.
In such a case, AI does not necessarily change who has formal access. It does, however, change what can practically be achieved with that access. Therefore, two questions must be placed side by side:
- What data does the application need for its task?
- What data can it technically actually reach?
When those answers diverge, there is a problem that cannot be solved with better instructions to the model. Data classification, minimal rights, correct access within SaaS platforms, DLP, controlled data sources, and restrictions on outbound data traffic thereby become part of the security surrounding AI. The Cloud Security Alliance emphasizes that existing principles of data security remain relevant when models, data sources, infrastructure, and external services become part of the same technical chain(6).
Architecture also plays a role. A controlled route to AI, linked to existing enterprise identities, authorization, and logging, offers more possibilities to technically enforce policy than an environment where every user, application, or AI agent can independently create new connections to models and data sources.
4. What is AI connected to and what sources are treated as reliable?
This is where an important difference with traditional software arises. Text in a document, website, or email is, in principle, data there. A language model interprets that text for meaning. As a result, information processed by the model can also attempt to influence the behavior of the system.
That forms the basis for indirect prompt injection. In this scenario, an attacker does not place a malicious instruction directly into the query to the model, but into information that the application later processes itself, such as a website, document, or email.
Suppose an AI agent is instructed to assess incoming email. A message contains text that attempts to make the AI agent ignore the original instruction, search for internal information, and subsequently use an available tool to send that information externally.
For the organization, the distinction is clear. The instruction from the authorized user is a command. The content of an external email is information from a source that cannot automatically be trusted. For the model, however, both end up in the context it must interpret. Google explicitly points out the risk with AI agents that manipulation of tools, memory, and retrieved data can lead to undesirable actions. Defense must therefore consist of multiple layers, including restrictions on tools and privileges, technical policy rules, user control, and oversight of the agent's actions(7).
This is not limited to email or websites. RAG sources, API results, tool descriptions, MCP servers, and other AI agents can also provide context that influences behavior. MCP makes connections with tools and data sources simpler and more standardized. Thus, the offered functions, used rights, and reliability of such a connection also become part of the security issue. OWASP mentions, among other things, tool poisoning, prompt injection, memory manipulation, and risks surrounding authentication and authorization(8).
The same applies to memory. Manipulated or incorrect information that is stored and later reused can have long-lasting effects. Not only the model, but also the origin of context, linked sources, available tools, and connections therefore determine the risk profile.
5. What is AI actually allowed to do?
As AI agents are allowed to act more independently, not only access but also functional authority becomes important. An agent that can read information has a different risk profile than an agent that can modify it or independently execute an action. An agent summarizing email therefore does not need to be able to send email, and an agent preparing a configuration change does not need to implement it independently.
OWASP describes excessive functionality, overly broad privileges, and too much autonomy as "excessive agency"(9). The answer resembles minimal rights but goes further: the independent operational scope must also match the task and potential impact. In practice, this may mean that read and write rights are separated, only pre-approved tools are available, elevated privileges are granted temporarily, and human approval is required for sensitive or irreversible actions(10).
Here lies an important distinction between an instruction to the model and a technical security boundary. An instruction attempts to guide the behavior of the model. A technical security boundary determines what is actually possible. A system instruction could, for example, indicate that confidential information should never be sent externally. That is useful. But when an AI agent technically has access to a tool with which arbitrary information can be sent to any external destination, that possibility still exists.
A hard security boundary works differently. The AI agent is not given that function. Only predetermined destinations are reachable. Certain data cannot technically leave that route. Or a separate authorization check blocks the action until someone approves it. The behavior of generative AI is not entirely predictable. Privileges must not be. Those must be technically hard-bounded.
That premise also changes how prompt injection should be viewed. The goal is, of course, to recognize and prevent manipulation as well as possible. But the architecture should not depend on the model always recognizing every wrong or malicious instruction. The environment must be structured such that potential damage remains limited if that first layer of defense fails. Google and Microsoft therefore advise a layered approach in which the model is only one part of the defense and rights, tools, authorization, and control are technically bounded outside of it(11).
6. Can the organization see, reconstruct, and stop what AI is doing?
The first question was about visibility: where is the AI located? Here it is about the next step: what is an application or AI agent actually doing? Suppose an AI agent first retrieves information from an internal document, then calls an API, subsequently consults another system, and ultimately executes an action.
Can it be determined after the fact which AI agent was active, under what identity, what information was used, which tools were invoked, what rights applied in that process, and what was ultimately modified or sent?
Storing only the conversation with the model is insufficient for this. Microsoft warns that logging only the final answer has little investigative value when information about tool invocations, effective rights, and authorization decisions is missing. Therefore, identity, used role, scope, touched source, executed action, and the coherence between steps must also be recorded(12).
Visibility, moreover, is only valuable if an organization can also intervene. Can an AI agent be disabled immediately? Can tokens and other access rights be revoked? Can a risky connection be blocked? Can an executed action be reversed where possible? And is it clear who is authorized to make those decisions?
NIST also covers auditing, non-repudiation, and the revocation of credentials in this regard. Microsoft goes further and explicitly advises a rapid shutdown capability with which credentials and tokens are effectively invalidated(13).
This touches on a broader security principle. Without sufficient visibility, detection becomes more difficult. Without good reconstruction, the cause of an incident remains unclear and there is a risk that only the symptom is solved. With AI agents, this is extra relevant because one command can lead to a series of actions across multiple systems.
Look at attack paths, not just individual measures
The six questions deliberately do not provide a universal list of measures. Not every application has the same risk profile. A search function over a carefully delineated knowledge base requires different security than an AI agent that can access email, CRM, cloud resources, and external websites.
Therefore, it is useful to look at attack paths. By an attack path, we mean the route along which an attacker can combine various systems, rights, and weaknesses to ultimately achieve a goal. AI can change existing attack paths and create new connections between components.
Take an approved enterprise application. The model used may be carefully configured, and the provider may have been thoroughly assessed. But when a compromised user account can find sensitive information much faster via that application due to historically overly broad rights, a significant part of the risk lies in identity and data access.
Take an AI agent that reads external documents and can request internal information. A possible attack path then begins with manipulated external information. That influences the AI agent, which subsequently calls a linked internal source and possibly uses an available tool to send information externally.
When securing AI, an organization must therefore not only look at vulnerabilities specific to AI. Equally important is how AI changes existing attack paths or enables new ones. Which measures deserve priority depends on the function of the application, the accessible data, connected systems, assigned rights, and potential impact.
A measure that exists is not yet a measure that works
The next step is validation. Suppose an organization has configured that an AI agent may not send confidential information externally. How certain is it that this restriction actually holds under various circumstances?
What happens when the AI agent processes a manipulated document? Can individually allowed tools together create an undesirable route? And are sensitive actions actually blocked while the complete sequence of actions can be reconstructed after the fact? That should not just be evident from an architecture diagram or configuration. It must be tested.
This follows a principle that is central to how Hunt & Hackett views security: security must be demonstrably effective, not just well-intended or correctly designed. The real question is whether a measure holds up when someone actually tries to circumvent it. That is especially important with AI. Not every component of system behavior can be fully predicted in advance. The technical boundaries around it must therefore be visible, testable, and demonstrably effective.
Secure implementation is not the endpoint
A secure design is necessary, but the risk profile continues to change thereafter. New data sources, tools, permissions, and functions are added. An application that started as read-only may be allowed to make changes a few months later, without anyone consciously introducing an entirely new system.
AI security thus also becomes an operational issue. As soon as an application becomes part of a business process, the organization must not only know how it is designed, but also be able to determine if its actual behavior deviates from expectations. Suspicious actions must be detectable and investigatable. The information required for this may be scattered across identities, endpoints, SaaS platforms, cloud environments, APIs, network traffic, and the AI application itself. Looking only at the model's log files will not provide a complete picture.
If an AI agent becomes part of an attack path, the defense must also be able to follow that entire path. Which identity was used? Which source influenced the agent? Which data was consulted? Which tools were invoked? What follow-up action took place? And where could intervention have occurred before the ultimate impact happened?
Incident preparedness must also change accordingly. An organization should not wait until an incident occurs to figure out who can shut down an AI agent, which data sources might have been consulted, and what technical information is needed to determine the scope of the incident. Those capabilities and responsibilities must be established in advance and practiced where necessary.
Security, therefore, does not stop at design and assessment. It continues into continuous detection, investigation, response, and incident preparation. As soon as AI becomes part of the operational environment, it must also become part of the broader view of attack paths and the capacity to detect, contain, and reconstruct incidents.
AI security is not a new silo
AI security should not become a standalone security domain. AI uses existing identities, corporate data, SaaS platforms, cloud environments, APIs, endpoints, and networks.
New technology surrounding AI may require additional measures, but many relevant disciplines are already known: identity and access management, data security, least privilege, segmentation, monitoring, validation, and incident preparedness. For the same reason, NIST applies the existing Cybersecurity Framework to the changing risks surrounding AI, rather than introducing an entirely separate model(14).
The question, therefore, is not how to build a separate program for AI alongside the existing security program, but how AI changes the existing attack surface and which parts of the security program need to evolve along with it.
Conclusion: from governance to technical reality
This brings us back to the starting point of this series. Governance determines which AI applications an organization accepts, which data may be used for them, and under what legal and organizational conditions AI can be deployed. Technical security then determines what AI can actually achieve and do. Monitoring and validation must demonstrate whether those boundaries hold up in practice.
Shadow AI has made it visible what happens when AI emerges outside of governance. The next challenge is broader: AI that is consciously allowed must also remain manageable as applications reach more data, connect systems, and execute actions autonomously via AI agents.
The core question thus becomes: can an organization demonstrate where AI is present, what reach that AI has, what technical boundaries have been enforced, and what happens when an application does not behave as expected? Good governance and technical security do not hinder AI adoption. On the contrary, they make it possible to apply AI on a larger scale and with confidence.
That is where governance and security come together in practice.
Share:
Shadow AI, the GDPR and data breaches (3): policy, procurement, trade secrets, enforcement and the role of the DPO