AI chatobot security: how to audit one already in production
- 1 day ago
- 9 min read
Written by: Juan Holgado
Deploying an AI assistant takes an afternoon these days. An API, a reasonable system prompt, a few tests with friendly questions, and it is already handling queries on your website.
Across the projects we review, the path repeats itself with little variation. The project starts in marketing or in customer support, it ships quickly under pressure to have some AI in production, and it never goes through the security review applied to every other internet-facing system.
That leads to a question worth asking before a client or an auditor asks it. If someone requested the security report for that assistant tomorrow, would it exist? In most of the organizations we work with, the answer is no: no risk assessment, no security testing and, in a share of cases, no logging of conversations. The assistant is still there, available 24 hours a day, answering strangers and connected to internal systems.
In a chatbot, instructions and data share the same channel
In a conventional web application, program instructions and user data live in separate spaces. In a chatbot they reach the model as text, through the same route. The model has no reliable way to tell what the developer wrote from what someone trying to manipulate it is writing. Almost every other problem follows from that characteristic.
The system is not deterministic either. The same question can receive different answers today and tomorrow, so a passing test guarantees little over the medium term. Tuning the prompt lowers the probability of failure and leaves open the route that allows it.
There is no CVE to warn you either. Nobody publishes an advisory when an assistant starts disclosing other customers' information.
From an assistant that answers to an agent that takes action
The risk has grown over the past year, because these assistants now execute actions as well as hold conversations. They query document repositories, call internal APIs, open tickets and process returns. Every connection turns a conversation into an action with consequences.
Those connections are no longer programmed one by one. With agents and protocols such as MCP, an assistant discovers tools at runtime, chains several calls to meet an objective and carries memory across sessions. It has moved from a component that responds to an actor that makes decisions.
OWASP has reflected that shift by separating the Top 10 for LLM Applications from the Top 10 for Agentic Applications, because they describe two distinct attack surfaces. In the enterprise deployments we have reviewed, both tend to be present at the same time.
Four public incidents and what each one demonstrated
In 2023 it emerged that Samsung employees had pasted internal code into a public AI assistant to get help with their work. Confidential information leaving through a channel nobody had classified as an exit channel (CNBC; AI Incident Database, 768).
In December of that year, screenshots circulated of a Chevrolet dealership's assistant agreeing to sell a car for 1 dollar (AI Incident Database, 622). A month later, in January 2024, DPD's customer service bot in the United Kingdom ended up swearing and criticizing its own company (ITV News; AI Incident Database, 631).
The fourth case is the one that changed the frame of reference. In February 2024, the Civil Resolution Tribunal of British Columbia ruled that Air Canada had to compensate a passenger for the incorrect information its chatbot had given him about bereavement fares. The airline argued that the chatbot was a separate entity from the company, and the tribunal rejected that argument (Moffatt v. Air Canada, 2024 BCCRT 149).
The four cases produced different consequences: data leakage, manipulation of business logic, reputational damage and legal liability. All of them were carried out by typing, without a single attack tool.
The two most common configuration failures
Those are the cases the industry cites most often and, for that very reason, the least representative: they are the ones somebody captured. What we usually find in audits is not spectacular.
Permissions that stop at the interface. A document search assistant built on top of the company's document management system needs to read the repository, so it is given a service account. The permissions of the user asking reach the chat and stop before the search: the person authenticates against the assistant, and the query is executed by the service account. The system works for months because nobody asks for what they are not entitled to. The day somebody does, and it tends to be a curious employee rather than an attacker, the assistant returns the contents of an HR folder or another client's documentation, well written and in a helpful tone. No exploit, no tooling, and with a log entry identical to that of a legitimate query, in the cases where logging exists.
Tools nobody has reviewed. The assistant has six or seven tools available because they shipped with the framework, and in production it uses two. The other five are still there, unreviewed and unmonitored, and sometimes without anyone on the team remembering they exist. They remain inert as long as nobody invokes them, although the model knows they are available: steering the conversation toward an objective that requires them is enough for it to use them on its own initiative. This is excessive agency in its purest form, and it is fixed by deleting lines of configuration.
These two patterns do not generate headlines, because they leave no screenshot to share. They surface when somebody goes looking for them.
Uneven responses depending on who is asking
The fastest-breaking risk is the assistant producing a racist, sexist or demeaning comment about a group of people. Current models ship with considerably more out-of-the-box filters than those of a few years ago, although those filters give way under pressure, and multi-turn conversation is the terrain where they give way first. An afternoon of attempts and a screen recording are enough to document it.
There is a quieter version of the same problem, and in our experience it is the more dangerous one. The assistant responds with a different tone, a different level of detail or a different willingness to help depending on how the person writes, what name they give or which language they ask in. In an isolated case nobody notices it. It becomes visible in the aggregate, and by then the organization has spent months serving one specific group of customers worse without ever having decided to.
The consequences follow two paths. The reputational one acts immediately: a screenshot circulates within hours, the context is lost along the way, and the brand stays associated with that sentence far longer than it takes to fix the prompt.
The legal one moves more slowly and costs more. In Spain, Law 15/2022, of 12 July, on comprehensive equal treatment and non-discrimination includes artificial intelligence within its scope of application and devotes its Article 23 to algorithms and automated decision-making mechanisms. Regulation (EU) 2024/1689, known as the EU AI Act, classifies as high-risk certain uses linked to employment or to access to essential services. And the Air Canada precedent sets a clear direction for any organization running a public assistant: courts treat the assistant as part of the company that deploys it.
This risk has an uncomfortable feature. No scanner detects it, so it has to be sought deliberately and methodically.
The end user's exposure
Almost the entire debate on this subject revolves around damage to the company, which is understandable, because the company pays for the audit. When the assistant serves patients, citizens, customers or employees, the greater exposure falls on the person at the other end.
That person sends their personal data to a chat without knowing where it is stored, for how long, or who consults it afterward. If context isolation fails, they may receive information belonging to another user. Without a record of the conversation, they have no way to prove what they were promised. And they receive answers written with impeccable confidence about serious matters, with no indication of their accuracy.
That last point is the one that concerns us most. A model that does not know something rarely admits it: it fills the gap, and does so in well-written prose. An industry professional identifies the error with relative ease. An elderly person asking about an administrative procedure, or somebody checking a health question at three in the morning, takes the answer at face value.
OWASP has reached a similar conclusion through the data. In the 2026 edition of the Top 10 for LLM Applications, misinformation climbs positions and shows the widest gap in the entire list between the perception of the professionals surveyed and the record of real-world incidents. The relevant change concerns the framing more than the ranking: a confident, wrong answer has come to be treated as a security problem and no longer only as a quality problem.
The limits of a traditional penetration test
Auditing the application that hosts the chatbot remains necessary: authentication, session management, API exposure, transport. All of that has to be done, and all of that leaves untouched the behavior of the model, which is where the new risk sits.
The object of the test changes. Here you verify whether the assistant respects its own limits, and the result is expressed as a success rate per category.
That format has a practical implication. A technique that works on a small fraction of attempts is still exploitable by anyone with patience, because the cost of retrying is zero. And the measurement expires quickly: a change in the model version or an adjustment to the prompt can reopen what had been closed.
The two halves of an AI assistant audit
The work has two halves, and the reference framework is public.
The adversarial half. The assistant is attacked under controlled conditions and a success rate is measured per category, following the OWASP Top 10 for LLM Applications: direct and indirect prompt injection (LLM01), sensitive information disclosure (LLM02), hidden context exposure (LLM08), which in the previous edition was called system prompt leakage, and improper output handling (LLM10). To that is added the abuse of each client's specific business logic, which appears on no list. Part of the work is automated with broad batteries of attacks. Another part requires manual execution, because multi-turn chains and the abuse of a specific business context escape the tooling.
The architecture half. It tends to yield the most serious findings, and it is organized around 4 questions:
Do the permissions of the user asking propagate all the way to the document search, or does the service query with service permissions?
Is what the model returns validated before an action is executed with that data?
What can the assistant invoke, and what would it actually need to invoke?
Is any record kept of the conversation?
With agents in play, the questions from the Agentic Top 10 are added: agent identity and privileges, runtime containment, and memory poisoning across sessions.
A good share of the findings is fixed by cutting back what the assistant is able to do, and guardrails barely touch them. Excessive agency (LLM03) has climbed positions in this year's OWASP list for that reason.
The inventory that comes first. Before all of the above there is a phase that looks bureaucratic and rarely is: listing the assistants that actually exist in the organization, with an owner, the data they can reach and the actions they are allowed to take. In several projects, that spreadsheet has been the first finding, because two or three assistants turn up that nobody had declared.
Legal obligations enforceable since August 2026
Since 2 August 2026, the transparency obligations of Article 50 of Regulation (EU) 2024/1689 have been enforceable. They include informing the person that they are interacting with a machine, with fines of up to 15 million euros or 3% of global annual turnover.
Regulation (EU) 2026/1744, known as the Digital Omnibus on AI and in force since 27 July 2026, has postponed much of the obligations for high-risk systems to 2 December 2027 for Annex III systems and to 2 August 2028 for those in Annex I. It has also relaxed the AI literacy obligation in Article 4 and introduced a transitional period of 4 months for the marking of systems placed on the market before 2 August 2026. The transparency requirement in Article 50 keeps its date: anyone operating a public chatbot already falls within its scope.
The GDPR never stopped applying to the personal data that comes in through the chat. And for organizations subject to Spain's National Security Framework, ENS, or to NIS2, an assistant connected to internal systems falls squarely within scope.
To organize the governance of all this, the certifiable reference today is ISO/IEC 42001:2023.
Three immediate checks
If the assistant is already in production and an audit is not immediate, there is room to reduce exposure without major work.
Verify that it discloses it is an AI. Since 2 August it has been mandatory to inform the person at the start of the interaction. The check takes 30 seconds and it is the only item on this list that may be in breach today.
Build the inventory. Which assistants exist, who maintains them, what data they can reach and what actions they can execute. It takes half a morning's work.
Assume the system prompt is public. If it contains credentials, pricing rules or sensitive business logic, that content is best removed sooner rather than later.
The assistant as a production system
Described precisely, an AI chatbot meets the conditions of a newly hired employee: no security training, access to internal systems, no assigned supervisor, and talking to strangers around the clock. Framed that way, the relevant question stops being whether it deserves an audit and becomes how we came to have several in production without having run one.
The useful part of the answer is that the work is well understood. The public OWASP framework exists, the criteria for separating what is fixed by writing from what requires architectural decisions exist, and the 4 questions from the previous section exist, which any team can answer this week without external help. Permissions that are applied at the interface and lost in the retrieval layer, tools exposed beyond need, and the absence of logging call for decisions about architecture and permissions, and fixing them rarely means rebuilding the project.
At Aktios we run red teaming exercises against AI assistants in production, and we leave the system documented for an ISO/IEC 42001 audit. If you have an assistant serving customers and could not say what it answers to a malicious instruction, it is better to find out in a controlled environment than in a screenshot.
_edited_edited.png){kind=logo}
