A public-facing AI assistant can appear polished, helpful and secure right up to the moment it exposes confidential information, follows a malicious instruction, or gives an employee a convincing but unsafe answer. AI penetration testing services help organisations examine those failure points before an attacker, disgruntled insider or accidental user does.
For Australian organisations, the question is not whether artificial intelligence will be used across the business. It is where it is already influencing customer interactions, staff decisions, software development, security operations and data handling. Testing must reflect that practical reality. A meaningful engagement examines both the AI system itself and the surrounding environment that makes it useful – identity controls, integrations, data stores, APIs, cloud configuration, monitoring and human workflows.
AI creates a different attack surface
Traditional penetration testing remains essential. An attacker still looks for exposed services, weak authentication, insecure code, misconfigured cloud resources and paths to sensitive data. AI changes the attack surface by introducing new ways to influence a system and new locations where sensitive information can travel.
A large language model connected to internal documents, for example, may be protected by single sign-on but still be susceptible to prompt injection. An attacker might craft content in a document, email or web page that attempts to influence how the model responds when it later processes that content. If the assistant has access to business data or can take actions through connected tools, the consequence can be more than an inaccurate answer.
The risks also extend beyond generative AI. Machine learning models can be affected by poor data handling, exposed model endpoints, weak access controls, insecure training pipelines and excessive permissions. AI-enabled applications can introduce vulnerabilities through third-party model providers, plugins, retrieval systems, orchestration frameworks and developer shortcuts made in the rush to deliver capability.
This does not mean every AI use case requires a specialist red-team exercise. The right level of testing depends on what the system can access, who can use it, whether it makes or informs material decisions, and the impact if it is manipulated. An internal writing assistant with no sensitive data has a different risk profile from a customer-facing chatbot connected to payment, health, legal or government information.
What AI penetration testing services should test
Effective AI penetration testing services do not simply ask a chatbot inappropriate questions and produce a list of entertaining responses. They apply a structured adversarial process to determine whether an AI-enabled system can be manipulated, whether it exposes information or performs unauthorised actions, and whether the organisation can detect and contain misuse.
The work normally begins with scoping. This identifies the system’s purpose, users, data classifications, integrations, privileges and business-critical processes. It also establishes clear rules of engagement. Production AI systems may interact with live customers, operational platforms or sensitive records, so testing needs to be rigorous without disrupting services or compromising privacy obligations.
Testing may then assess prompt injection and jailbreak resistance, including direct attempts to override system instructions and indirect attempts delivered through untrusted content. Consultants examine whether the model reveals system prompts, confidential context, credentials or information belonging to another user. Where a system uses retrieval-augmented generation, testing should validate document permissions, tenant segregation, citation handling and the potential for manipulated source material to affect responses.
Tool and API integrations deserve particular attention. If an AI assistant can search records, send messages, create tickets, modify files or initiate business processes, the test should determine whether it can be induced to act outside approved authority. This includes checking whether the system appropriately confirms high-impact actions, applies least privilege and preserves a reliable audit trail.
The engagement should also cover the familiar security fundamentals around the AI environment: authentication, session management, authorisation, API security, application logic, cloud configuration, secrets management and logging. AI-specific weaknesses rarely exist in isolation. A weak identity configuration or an overly broad service account can turn a model-level flaw into a significant business incident.
AI-assisted testing is not the same thing
There is an important distinction between testing an AI system and using AI to assist a penetration test. Security teams increasingly use AI-enabled tools to analyse code, identify patterns, draft test cases and help sort large volumes of reconnaissance data. Used carefully, these tools can improve efficiency and allow experienced testers to focus more time on validation, exploitation paths and remediation priorities.
However, AI-generated output is not evidence on its own. It can misunderstand the environment, overstate a finding, miss a business control or suggest unsafe testing actions. Automated scanners – whether they use AI or not – identify possibilities. Senior penetration testers confirm whether a weakness is exploitable, determine what an attacker could actually achieve and explain the risk in the organisation’s context.
For that reason, organisations should be cautious of services that promise fully automated penetration testing at a fraction of the usual effort. Automation has a valuable role in continuous exposure management and vulnerability discovery. It is not a replacement for independent assurance, careful manual testing or informed judgement when sensitive systems are involved.
Turning findings into risk reduction
A useful report should help leaders make defensible decisions, not leave them with a long technical backlog. Findings need clear evidence, realistic impact, affected assets and prioritised remediation actions. For AI systems, that may include tightening access boundaries, separating data sources, reducing tool permissions, adding approval steps, improving prompt and content filtering, revising logging, or preventing sensitive data from being supplied to unapproved models.
The best remediation is often layered. A guardrail at the model interface may reduce harmful outputs, but it should not be the only protection for confidential data or high-value actions. Strong identity and access management, data classification, network segmentation, secure development practices and monitoring remain essential. Controls should assume that prompts and filtering can fail under determined testing.
This is where penetration testing connects naturally with broader cyber security governance. Results can inform a security strategy, risk register, supplier assurance process, secure AI policy and staff awareness programme. They can also be mapped to controls already used across the organisation, including the Essential Eight, CIS Controls, NIST Cybersecurity Framework and NIST 800-53. The mapping should be practical rather than performative: it should show which controls need improvement, who owns the action and how effectiveness will be verified.
Selecting the right engagement
When procuring a provider, ask how the team distinguishes AI-specific testing from conventional web application testing, how it handles sensitive data and production systems, and who performs the work. The answer should involve experienced practitioners, a defined methodology and clear oversight – not a generic scan followed by a templated report.
It is also reasonable to ask what happens after testing. Some organisations need independent assurance only. Others need a partner that can help remediate findings, improve Microsoft 365 or cloud controls, test changes and provide continuing support. Both models can be appropriate, provided independence is maintained where formal assurance is required and responsibilities are clear.
At Sentaris, this work can be integrated across consulting, assurance and engineering: identifying the business risk, validating the controls through adversarial testing, then supporting the technical uplift. That approach is particularly useful where AI adoption is moving faster than internal security capacity.
AI capability will continue to be embedded in ordinary business systems, often before it is labelled as an AI project. The practical next step is to identify where it can access sensitive information or trigger meaningful action, then test those pathways with the same discipline applied to any other business-critical service.