Testing Conditional Access policies

Conditional Access policies are powerful and easy to misconfigure. A single overly-broad policy can lock thousands of users out of Microsoft 365. Treating CA policies like production code — testing before enforcement, rolling out gradually — is essential.

Report-only mode

Every Conditional Access policy supports Report-only mode — the policy evaluates but doesn't enforce. The logs show what would have happened if the policy were active.

For every new policy:

1. Configure it with the conditions and controls you want.
2. Set state to Report-only instead of On.
3. Wait at least a week for representative traffic.
4. Review sign-in logs to see which sign-ins would have been affected.
5. Adjust the policy if findings are unexpected.
6. Move to On when confident.

This single discipline prevents most CA disasters.

What-If tool

The Conditional Access What-If tool lets you simulate a specific sign-in scenario and see which policies would apply:

1. In Conditional Access, open What If.
2. Specify: user, app, IP / location, device, sign-in risk, user risk, client app type.
3. The tool evaluates all policies and shows which would grant or block, and what controls would be required.

Useful for validating specific edge cases — "what happens if our finance lead in Mexico signs in from her hotel during a conference?" or "what about a contractor's Mac without MDM enrolment?"

Sign-in log analysis

The Entra ID sign-in logs show every CA policy evaluation:

• Applied policies — which policies evaluated for this sign-in.
• Result per policy — grant, block, report-only.
• Reasons — why each control was applied.

For investigating "why can't this user sign in" or "what policy is blocking this app," sign-in logs are the authoritative source. Reading them carefully takes practice — start with one failed sign-in, walk through every applied policy.

Rolling out a new policy

A safe rollout pattern:

1. Design the policy with clear scope and intent. Document it.
2. Report-only for a week or two. Review logs.
3. Pilot on a small group — assign the policy to a test security group of 10–20 users. Move from report-only to enforce for that group.
4. Expand to a department — broader rollout, but still constrained.
5. Tenant-wide enforcement — only after confident at smaller scales.

For high-impact policies (block legacy auth, require compliant device), rolling out in waves over weeks is normal.

Common mistakes

• Skipping report-only because the policy "is obviously safe."
• Excluding the wrong test accounts from the policy during enforcement.
• Not including emergency-access (break-glass) accounts in exclusions.
• Mass-enforcing many policies at once — if multiple new policies fire on the same day, isolating which broke something is hard.
• Editing live policies in production — clone, edit the clone, swap when validated.

Break-glass exclusions

Every Conditional Access policy needs break-glass account exclusions. The two break-glass accounts (minimum) should be excluded from every CA policy so they can always sign in. Validate this regularly — easy to forget when adding new policies.

Combine with PIM and Identity Protection

For the most powerful policies:

• Combine with PIM so privileged role activation requires step-up authentication via auth contexts.
• Combine with Identity Protection so risky users / risky sign-ins trigger blocking or step-up.
• Combine with device compliance so non-compliant devices are blocked from sensitive apps.

These multi-layer policies are powerful but also more brittle. Test extensively.

For Microsoft 365 tenants with serious Conditional Access posture, this disciplined testing pattern is what separates "we have CA configured" from "we have CA working without locking out users monthly." The discipline pays back constantly.