Patching servers in order with Configuration Manager orchestration groups

Deploying the monthly updates to one big server collection is simple until two nodes of the same cluster restart at the same time. Orchestration groups in Configuration Manager fix that: they decide how many members of a group, or which one next, may install updates, and they can run a PowerShell script before and after each device. In this post I'll build an orchestration group for a two-node cluster, choose timeouts that fit your maintenance windows, and show how to monitor and rerun it.

How it works#

An orchestration group is a console object that lists devices and a rule. The rule applies to every software update deployment that reaches a member, through any collection, so you keep deploying updates to your normal server collections. Deadlines, maintenance windows and deployment schedules still apply. Within those, the group lets members update in one of three ways:

• Allow a percentage of the machines to be updated at the same time: with 50 members and 10%, five devices update at once, and the number grows with the group.
• Allow a number of the machines to be updated at the same time: a fixed count, whatever the group size.
• Specify the maintenance sequence: an explicit order, one device at a time.

For each member, the flow is: pre-script, update installation, a restart if required, then the post-script. Orchestration groups are the evolution of the older Server Groups feature. Enabling orchestration groups turns server groups off, and sites that had server groups enabled had them moved to orchestration groups automatically when they installed version 2002 or later. Since version 2111 the feature is no longer pre-release.

Prerequisites and limits#

• Enable Orchestration Groups under Administration › Updates and Servicing › Features.
• Use a Full Administrator account: role-based administration isn't available for orchestration groups, and approving their scripts needs Full Administrator or Operations Administrator.
• Keep members on the latest client version and assigned to the same site. A device can belong to only one orchestration group, and a group can have up to 1,000 members.
• Orchestration groups don't work while the hierarchy is in interoperability mode.
• Two bypasses to remember: updates a user starts from Software Center skip orchestration, and so do updates in the Definition classification.

Step-by-step#

1. Write the pre and post scripts#

Scripts can't take parameters, are limited to 50,000 bytes, and must return 0 for success; anything else is a failure. Use $env:COMPUTERNAME instead of parameters. A pre-script that drains a failover cluster node:

try {
    Import-Module FailoverClusters -ErrorAction Stop
    Suspend-ClusterNode -Name $env:COMPUTERNAME -Drain -Wait -ErrorAction Stop | Out-Null
    exit 0
} catch {
    exit 1
}

And a post-script that brings the node back and checks it, so a failure stops the sequence before the next node goes down:

try {
    Import-Module FailoverClusters -ErrorAction Stop
    Resume-ClusterNode -Name $env:COMPUTERNAME -Failback Immediate -ErrorAction Stop | Out-Null
    if ((Get-ClusterNode -Name $env:COMPUTERNAME).State -ne 'Up') { exit 1 }
    exit 0
} catch {
    exit 1
}

2. Create the orchestration group#

1. Go to Assets and Compliance › Orchestration Group and select Create Orchestration Group.
2. On General, enter a name and set the two timeouts. Orchestration Group timeout (in minutes) limits how long all members together may take; Orchestration Group member timeout (in minutes) limits a single device.
3. Add the members (both cluster nodes, from the same site).
4. Choose the rule. For a cluster, use Specify the maintenance sequence and order the nodes.
5. On the Script Picker page, add the pre-script and post-script (paste them, open a .ps1 file, or browse to an approved script from the Scripts list) and set each Script timeout (in seconds).

3. Size the timeouts#

The member timeout must cover the pre-script, update installation, the restart and the post-script on your slowest node. Measure a normal patch night and add a margin. In sequence mode the group timeout needs to cover every member in turn, so it should be at least the member timeout multiplied by the number of members, and the whole run must fit inside the maintenance window. Otherwise the last node simply waits for the next window.

4. Approve the scripts#

Since version 2111, pre-scripts and post-scripts don't run until someone approves them, and editing an approved script puts it back to Waiting for approval. The Scripts tab of the orchestration group shows each script's approval status. If your hierarchy requires a second approver for scripts, plan for that before patch night (see CMPivot and Run Scripts).

5. Deploy updates as usual#

Nothing changes in the deployment itself: your automatic deployment rule or software update group targets the server collection with a deadline, and the maintenance windows on the server collections decide when installation can start. The orchestration group only controls who goes next. For a ring design, see automatic deployment rules for Patch Tuesday.

Verify#

1. Group members: in the Orchestration Group node, open the group's members and check each device's orchestration state and when it finished. The nodes should finish one after the other, never together.
2. Deployment status: under Monitoring › Deployments, every member reports Compliant.
3. On each server: no deployed updates should still be pending, and the cluster should be healthy:

Get-CimInstance -Namespace root\ccm\ClientSDK -ClassName CCM_SoftwareUpdate |
    Select-Object ArticleID, Name, EvaluationState
Get-ClusterNode | Select-Object Name, State

Good looks like this: the first command returns nothing, and every node is Up.

Tips and troubleshooting#

• A member doesn't start: check the maintenance window first (deployments waiting for a maintenance window), then whether an earlier member is still running or has failed.
• Scripts never run: they're unapproved, or they were edited after approval.
• A member failed or completed and you need to rerun it: right-click it and select Reset Orchestration Group Member, which clears states such as Complete or Failed.
• Updates installed out of order: someone selected Install in Software Center, which bypasses orchestration. For these servers, set the deployment's user notifications to hide in Software Center, or brief the server owners.
• Updates install but report failures: troubleshoot the client side as usual with WUAHandler.log, UpdatesDeployment.log and UpdatesHandler.log (see updates not installing), and look up codes in the software update error codes quick reference.

References#

• About orchestration groups (Microsoft Learn)
• Create and use orchestration groups (Microsoft Learn)