A Conditional Access policy that's wrong by one condition can lock out a department, or your own admins. Microsoft Entra ID gives you two safety nets: report-only mode, which evaluates a policy against real sign-ins without enforcing it, and the What If tool, which simulates a single sign-in on demand. In this post I'll combine them into a repeatable rollout routine.
Prerequisites#
- Microsoft Entra ID P1 for Conditional Access. You need Conditional Access Administrator to change policies; Security Reader is enough to review results.
- For the insights and reporting workbook: a Log Analytics workspace that receives your sign-in logs, plus Security Reader and Log Analytics workspace Contributor access.
- At least two emergency-access accounts, excluded before you build anything (see the tips at the end).
Step 1: Create the policy in report-only#
Build the policy as usual in Entra ID › Conditional Access › Policies, exclude your emergency-access accounts, and set Enable policy to Report-only. In Microsoft Graph this state is enabledForReportingButNotEnforced. The policy is now evaluated at every matching sign-in, but its grant and session controls aren't enforced: nobody is blocked or prompted for MFA because of it.
Watch out: report-only policies that require a compliant device can still prompt users on macOS, iOS and Android to select a device certificate, repeatedly, until the device is compliant. Exclude those platforms from report-only compliance policies. Policies that target user actions, such as registering security information, can't run in report-only at all.
Step 2: Read the results per sign-in#
In Entra ID › Monitoring & health › Sign-in logs, open a sign-in and select the Report-only tab. Each report-only policy shows one of four results:
| Result | What it tells you |
|---|---|
| Report-only: Success | Conditions matched and the controls were already satisfied, for example by an existing MFA claim or a compliant device. |
| Report-only: Failure | Conditions matched but a non-interactive control wasn't satisfied. When enforced, this sign-in would be blocked. |
| Report-only: User action required | The user would have been prompted (MFA, terms of use), so the final outcome is unknown. |
| Report-only: Not applied | Conditions didn't match, for example an excluded user or location. |
Review non-interactive sign-ins too, because desktop and mobile apps refresh their tokens there.
Step 3: Measure impact across the tenant#
- Policy impact (preview): for Security Reader and above, a snapshot of the potential impact on interactive sign-ins over the last 24 hours, 7 days or month, with sample sign-ins.
- Insights and reporting workbook: go to Entra ID › Conditional Access › Insights and reporting, select the report-only policy and a time range (up to 90 days of streamed data), and read the Success, Failure, User action required and Not applied tiles. Breakdowns cover device platform, client app, location, application and more.
For custom questions, query the workspace directly. This lists who would have been blocked by report-only policies in the past week:
SigninLogs
| where TimeGenerated > ago(7d)
| mv-expand CAPolicy = ConditionalAccessPolicies
| where tostring(CAPolicy.result) == "reportOnlyFailure"
| summarize SignIns = count() by Policy = tostring(CAPolicy.displayName), UserPrincipalName, AppDisplayName
| order by SignIns descChange the filter to reportOnlyInterrupted to list sign-ins that would have needed user action.
Step 4: Simulate edge cases with What If#
Report-only shows what happened; What If answers "what would happen if…". Open Entra ID › Conditional Access › Policies › What If. Identity, target resource, device platform and client app are required; add the IP address, country, risk levels and any other condition your policies use. The report lists the policies that would apply, with the grant and session controls each requires, and the ones that wouldn't, showing the first condition that didn't match.
- Only enabled and report-only policies are evaluated.
- Fill in every condition your policies depend on. The tool runs on the What If evaluation API, and a condition you leave empty can't be matched.
- Choose the specific app (its application ID) rather than a group such as Office 365.
- Service dependencies aren't simulated: testing Teams doesn't include policies that apply to Exchange Online.
Step 5: Roll out in stages#
- Leave the policy in report-only long enough to cover a full business cycle, including monthly or occasional processes.
- Investigate every Failure and User action required result, fix the conditions or exclusions, and re-check.
- Switch the policy to On for a pilot group and watch the sign-in logs for failures.
- Expand in waves until it covers the full scope.
- For changes to an enforced policy, create a report-only copy with the change, compare the two, then update the original and delete the copy.
Verify#
- After enforcement, the policy appears on the sign-in's Conditional Access tab with Success or Failure instead of a report-only result.
- A What If run for an emergency-access account lists the policy among those that don't apply.
- In the workbook, the policy now appears in the Enabled group rather than Report-only.
Tips & gotchas#
- Emergency access: keep two or more cloud-only accounts on the
*.onmicrosoft.comdomain, protected with a passkey (FIDO2) or certificate-based authentication, and exclude them from every enforced policy that blocks or restricts sign-in. Report-only policies don't need the exclusion, but adding it from day one means it's already there at enforcement. Monitor their sign-in and audit logs. - Don't create block or compliant-device policies for all users and all resources without exclusions; that's how admins lock themselves out.
- What If doesn't check whether a real device is compliant or which methods a user has registered. Confirm those in the sign-in logs.