AI makes reconnaissance faster and lures cleaner. It has not created a new class of attack. The defenses that fail are the ones built on spotting bad writing.
Key judgments
Observed means a named source reported it. Assessed means it is our reading of that evidence.
| # | Judgment | Confidence |
|---|---|---|
| 1 | Observed adversary use of AI is mostly productivity: research, drafting, translation and tailoring. Google found little that was genuinely new capability. (Observed) | High |
| 2 | Poor grammar and awkward phrasing are no longer dependable phishing signals, in any language the attacker needs. (Assessed) | High |
| 3 | Reconnaissance is the under-discussed change. Attackers use AI to research targets and unfamiliar technology and to build role-specific pretexts. (Observed, assessed) | Medium |
| 4 | The most damaging recent social engineering relied on human callers and legitimate workflows. The failures were process failures, not message-quality failures. (Observed) | Medium |
| 5 | No source gives a dependable measure of how much phishing overall is AI-generated. Prevalence figures come from provider and vendor visibility and should be read that way. | Low |
What has been observed
Google Threat Intelligence Group tracks adversary use of its Gemini model. A China-nexus actor used it for initial reconnaissance on targets of interest, then for phishing research, and also to work through cloud, vSphere and Kubernetes environments it was unfamiliar with. Iran’s APT42 used it to draft phishing material that impersonated people from reputable organizations such as think tanks, and for translation. A North Korea-nexus cluster, UNC1069, produced cryptocurrency-themed lures and Spanish-language messages, and used deepfake image and video lures to deliver a backdoor through a fake video-conferencing download.
Underground forums now advertise AI tooling for phishing kits and lure creation. Google noted that almost every notable tool it saw mentioned phishing support. Actors have also tried to talk around model safeguards by posing as researchers, students or capture-the-flag participants.
Verizon’s latest Data Breach Investigations Report page states that about 15% of attack techniques are now bolstered by generative AI.
The same reporting is careful about limits. Google saw guardrails hold in many cases, found the “novel” AI-enabled malware it examined to be experimental, and noted that a lot of the activity was productivity gain rather than new capability. Its visibility is also limited to its own platform.
What AI changes in the phishing chain
| Stage | Without AI | With AI assistance | What the defender should do |
|---|---|---|---|
| Reconnaissance | Manual research on staff, roles, vendors and technology. Slow and uneven. | Fast summaries of public information about people, roles and unfamiliar systems. Observed in a China-nexus actor’s cloud and Kubernetes research. | Assume the attacker knows your org chart and tooling. Reduce what is publicly exposed, and stop treating that knowledge as proof of legitimacy. |
| Lure writing | Spelling and tone errors, limited languages. | Fluent, localized, role-appropriate text that can impersonate a known organization or person. | Stop teaching staff to look for typos. Teach them to verify the request. |
| Live interaction | A human on the phone or in chat. | Deepfake audio or video lures observed in a North Korea-nexus campaign. | Verify out of band, through a channel the requester did not supply. |
| After the click | Credential capture, MFA relay, app consent. | Unchanged. AI does not alter what happens next. | Phishing-resistant authentication; control over who can approve connected apps. |
Why defenses miss it
Awareness programs often rely on error-spotting, and click rates reward that. The help desk is the weaker point. Reset procedures that rely on questions answerable from public or stolen data, such as a manager’s name or an employee ID format, are exactly what cheap reconnaissance defeats. Google’s analysis of the UNC6040 campaign shows the result: callers posing as IT support persuaded employees to enter credentials and MFA codes and approve a malicious connected app, with no Salesforce vulnerability involved. The technique was human, and it worked on process.
What defenders should change
-
Retire “look for errors” guidance. Replace it with a rule about the request: any ask to move money, reset a credential, approve an app or install software is confirmed through a known channel, whatever the message looks like.
-
Harden the help desk. Verification must not depend on facts an attacker can find or buy. Alert when a password or MFA reset is followed by a new device registration.
-
Use phishing-resistant authentication where it matters most. FIDO2 and passkeys bind the login to the real site, so a convincing fake page cannot relay them. Start with administrators and finance.
-
Control app consent. Restrict who can approve connected and OAuth apps, and review approvals made soon after a support call.
-
Look at your own exposure. Check which staff directories, role announcements, org charts and vendor lists are public. You cannot remove everything. You can stop publishing what a pretext needs.
-
Test with realistic pretexts. Include voice and chat, multi-step scenarios, and the help desk. Measure how many people report and how fast, not only how many click.
-
Watch the infrastructure. Monitor for lookalike domain registrations, unusual mailbox rules and new device registrations after resets.
What we do not know
Nobody outside the model providers can see the full picture, and the providers see only their own platforms. Prevalence claims vary by vendor and method. Some of the most convincing lure content is generated with tools no provider monitors. Deepfake use is documented but not yet shown to be widespread. Attackers’ reliance on AI will likely grow; today’s evidence does not tell us how fast.
What to do with this
Run one test that targets your help desk with a pretext built only from public information about a real employee. If it succeeds, that is your first fix, ahead of any change to email filtering or training content.
Sources
-
Google Cloud Threat Intelligence, Adversarial misuse of AI by threat actors (Google Threat Intelligence Group)
-
Google Cloud Threat Intelligence, UNC6040 voice phishing and data extortion analysis
How we label evidence
Facts carry a source. Analysis and recommendations are marked as ours. Examples that are hypothetical are labeled as hypothetical.
Part of the solutionProtect What MattersGuard the identities, data and systems the business depends on.See how we work →Related service
Phishing Operations
Use this to test and improve phishing resilience.