Defender XDR custom detection rules: from advanced hunting query to alert and action

Advanced hunting answers a question once; a custom detection rule asks it on a schedule, raises an alert when there's a match and can respond automatically. Getting there takes more than pasting a query into the wizard: the query has to return the right columns, the frequency decides how far back each run looks, and actions only work when the matching identifiers are in the results. In this post I'll walk through building, testing and maintaining a rule in the Microsoft Defender portal.

Prerequisites#

• Data from Defender for Endpoint Plan 2 or other Defender XDR workloads, or Microsoft Sentinel data in the Defender portal.
• Permission to manage the data the rule queries: the Microsoft Defender permission Security settings (manage), or the Security Administrator role. Security Operator is only enough when Defender for Endpoint RBAC is off; with it on, you also need the Manage security settings permission. Sentinel data needs Microsoft Sentinel Contributor or higher.
• Permissions per workload: managing rules on Email* tables needs Defender for Office 365 permissions, and IdentityLogonEvents needs both Defender for Cloud Apps and Defender for Identity. You also need rights to every device group in the rule's scope.
• A working query. If you're new to KQL, start with advanced hunting for Intune admins.

Step-by-step#

1. Shape the query#

Run the query in Hunting › Advanced hunting first and make sure it returns:

• Timestamp (or TimeGenerated) for the alert time.
• For Defender for Endpoint tables, DeviceId and ReportId, so alerts get the right device group scope and a process tree. For other Defender tables, Timestamp and ReportId from the same event.
• A strong identifier for the impacted asset: DeviceId, DeviceName or RemoteDeviceName; RecipientEmailAddress, SenderFromAddress or a sender or recipient object ID; AccountObjectId, AccountSid or AccountUpn.

A simple filter query keeps these columns automatically:

DeviceProcessEvents
| where FileName in~ ("powershell.exe", "pwsh.exe")
| where ProcessCommandLine has_any ("-EncodedCommand", "-enc")
| where InitiatingProcessFileName in~ ("winword.exe", "excel.exe", "outlook.exe")
| project Timestamp, DeviceId, DeviceName, ReportId, AccountUpn,
          InitiatingProcessFileName, ProcessCommandLine

When you aggregate, carry the event identifiers through with arg_max, or the service can't tie the alert back to a real event:

DeviceLogonEvents
| where ActionType == "LogonFailed"
| summarize (Timestamp, ReportId) = arg_max(Timestamp, ReportId), Failures = count()
    by DeviceId, AccountName
| where Failures > 20

Don't filter on Timestamp or TimeGenerated to set the time window: the service already pre-filters each run by its lookback. If you want a time filter for performance, Microsoft's own sample uses ingestion_time() matched to the run frequency.

2. Create the rule and pick the frequency#

Select Create detection rule from the query, or + Create detection rule on the custom detection rules page. Give it a unique name and alert title (plain text only), then set severity, category, MITRE ATT&CK techniques, a description and recommended actions for responders.

Continuous (NRT) only works when the query uses a single supported table, only supported operators, no joins, unions or externaldata, and no comment lines. A Custom frequency (5 minutes to 14 days) is offered only when the rule uses Microsoft Sentinel data alone. Because runs evaluate ingestion time and the lookback is longer than the interval, the same event can be seen twice; custom detections deduplicate those into one alert.

3. Enrich the alert#

• Dynamic title and description: reference result columns in double curly brackets, for example {{AccountUpn}} on {{DeviceName}}, up to three columns per field.
• Custom details: up to 20 key-value pairs shown in the alert side panel, with a 4 KB limit per alert; above that, all custom details are dropped.
• Entity mapping: choose which column is the impacted device, user or mailbox. Entities drive how alerts group into incidents, so map them carefully.

4. Choose response actions#

Blocking files also needs Remediate permissions for files. If the identifier column is missing, the action has nothing to act on.

5. Set the scope and create#

Choose All devices or specific device groups. The rule only queries, and only acts on, devices in scope; scope doesn't affect rules that check only mailboxes or identities. Select Create: the rule runs immediately against the past 30 days of data, then on its schedule.

Verify#

1. Open the rule from Hunting › Custom detections and check its previous runs and status.
2. Reproduce the behavior on a test device in scope, wait for the next run, and confirm an alert under Triggered alerts. From the alert you can rerun the triggering query in advanced hunting.
3. Check Triggered actions to confirm each response ran against the intended device, file, user or email, and follow up in the Action center as you would for any response.

Tips and gotchas#

• Each rule can raise at most 150 alerts per run. Tune out normal activity before you enable it; a noisy rule hides real hits.
• Run new rules as alert-only for a while, and add isolation or user actions once false positives are near zero. Automatic actions on a bad query affect real users.
• A failed or partly failed run usually means a time-out, throttling, a result-size limit, or events that can't be matched back to their Timestamp, DeviceId or ReportId, often because an aggregation removed them.
• On the rule page, use Run to test after editing, Modify query to open it in advanced hunting, and Turn off rather than delete while you investigate.
• Rules that qualify for Continuous (NRT) can be moved with Migrate now on the custom detection rules page.
• Alerts from custom detections land in incidents like any other alert; see your first hour with a Defender XDR incident.

References#

• Create custom detection rules in Microsoft Defender XDR
• Manage custom detection rules in Microsoft Defender XDR