A phishing simulation is only worth running if the results are believable and the follow-up teaches something. Attack simulation training in Microsoft Defender for Office 365 handles delivery, landing pages, training and reporting, but the choices you make in the wizard decide whether you get a useful baseline or noise. In this post I'll cover licensing and roles, the planning that comes first, each wizard step, how to read the results, and where advanced delivery does and doesn't come in.
Prerequisites#
- Licence: Microsoft 365 E5 or Defender for Office 365 Plan 2 (included or as an add-on). E3 tenants get a limited trial: a Credential Harvest payload plus the ISA Phishing or Mass Market Phishing training experiences.
- Roles: the Microsoft Entra roles Attack Simulation Administrator (runs campaigns) and Attack Payload Author (builds payloads only), or Security Administrator; Security Reader and Security Operator can only view. Unified RBAC isn't supported, nor is adding Attack Simulation Administrators through Email & collaboration permissions, and there are no PowerShell cmdlets, so assign the Entra roles, ideally just-in-time with Privileged Identity Management.
- Mailboxes: Exchange Online gives full reporting; on-premises mailboxes work with reduced reporting.
- Reporting: users need an obvious way to report, so check the built-in Report button in Outlook is enabled under Settings › Email & collaboration › User reported settings.
You'll find the feature at Email & collaboration › Attack simulation training in the Microsoft Defender portal.
Plan before you launch#
- Decide what you're measuring. A first campaign is a baseline of clicks, credential entries and reports.
- Pilot it with IT and security first, check it renders in Outlook desktop, web and mobile, and confirm the landing page opens in your browsers. Microsoft warns that some simulation URLs can be flagged by Google Safe Browsing.
- Brief the service desk and SOC so calls and reports about the message are handled consistently, without announcing the test to everyone.
- Check third-party tools. If an API-based email security product, sandbox, proxy or browser isolation service opens links in mail, add exceptions for the simulation domains Microsoft lists on its getting-started page. Otherwise users can't reach the landing page, or you record clicks no person made.
Step-by-step: launch a simulation#
On the Simulations tab select Launch a simulation, then work through the wizard.
1. Technique#
| Technique | What it tests |
|---|---|
| Credential Harvest | A link to a fake sign-in page; the classic baseline |
| Malware Attachment | Opening an attachment |
| Link in Attachment | A link inside an attachment that leads to a sign-in page |
| Link to Malware | A link to a file on a file-sharing site |
| Drive-by URL | Visiting a site that runs background code (watering hole) |
| OAuth Consent Grant | Granting a malicious app access to data |
| How-to Guide | No test: a short guide, such as how to report phishing |
Credential Harvest, Link to Malware, Drive-by URL and OAuth Consent Grant can use a QR code instead of a URL.
2. Name and description#
Use a name that sorts well in reports, such as Baseline – Credential Harvest – Finance – Wave 1.
3. Payload and login page#
Pick from Global payloads (Microsoft's catalog, in many languages) or Tenant payloads (yours, including customised copies). The Predicted compromise rate uses Microsoft 365 history to estimate what share of recipients a payload will compromise, and the complexity filter (High, Medium, Low) reflects how many warning signs a payload contains. For a baseline, choose a lure your users really see, such as a document share or mailbox notice, rather than the hardest one available. Credential Harvest and Link in Attachment payloads also use a login page you can swap.
4. Target and exclude users#
Target all users or specific users and groups, then exclude anyone who shouldn't receive it: the pilot team, briefed staff and non-personal mailboxes.
5. Assign training#
Use the Microsoft training experience (let it choose modules or pick them yourself), redirect users to your own learning platform, or assign none. Training follows what each user did with the message, so set a realistic due date and let reminders do the chasing.
6. Landing page and notifications#
The landing page is the teaching moment users see after falling for the payload; built-in pages explain what happened and what to look for next time. The Microsoft default end-user notifications cover training assignment and reminders, and you can add a positive reinforcement message for users who report the simulation.
7. Launch details#
Launch now or schedule it, set when it ends, and consider region-aware time zone delivery so users receive it at the scheduled time in their own time zone. Review and submit.
Verify: read the results#
Open the running simulation from the Simulations tab. Its report answers three questions: did everyone receive it, what did each user do (delete, report, open the payload, enter credentials), and who has completed training. The Reports tab shows trends across simulations, such as training efficacy and repeat offenders, and compares actual results with the predicted compromise rate. Good looks like this:
- Delivery is complete for everyone targeted.
- The actual compromise rate is at or below the predicted rate and falls campaign by campaign.
- The report rate rises; reporting is what helps your SOC.
- Training is finished by the due date and the repeat offender list shrinks.
A burst of clicks within seconds of delivery across many users usually points to a security tool following links, not to people. Fix the exception and treat that campaign's numbers with caution.
Where advanced delivery fits#
Advanced delivery (Email & collaboration › Policies & rules › Threat policies › Advanced delivery) lets a third-party simulation vendor's messages through unfiltered: you list its sending domains, sending IPs and simulation URLs on the Phishing simulation tab. It also covers SecOps mailboxes. Microsoft's getting-started guidance for Attack simulation training doesn't ask you to configure it for Microsoft's own simulations. If you also run a third-party tool, use advanced delivery for it rather than mail flow rules or weakened anti-phishing policies, and keep its entries tight.
Tips & gotchas#
- Click tracking is independent of your Safe Links settings. Attack simulation training uses Safe Links to record clicks even when Track user clicks is off in your policies.
- Automate the programme. Simulation automations can combine several techniques and payloads on a recurring schedule, and training campaigns assign modules without any simulation.
- Teach reporting between simulations with the How-to Guide technique and the QR code training modules.
- Compare like with like. To see which departments are most susceptible, run identical simulations scoped by department.
- Check regional availability. Microsoft notes that reported-email telemetry isn't yet available in its newest regions (Norway, South Africa, the UAE and Germany).
