DAG patching with maintenance mode checklist

Before you start

Start with a quick health check:

• An elevated Exchange Management Shell on the server you're patching (EX01 in the examples). The DAG scripts can also run remotely, but the machine running them needs the Failover Cluster Management tools (RSAT-Clustering).
• A healthy passive copy of every database that's active on EX01, and another Mailbox server (EX02.contoso.com here) to receive redirected mail.
• The CU or SU that matches your current build, downloaded in advance and checked against Microsoft's Exchange build numbers page.
• A note of EX01's current activation policy, because the stop script resets it to Unrestricted.

Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus |
  Format-Table Name,Status,CopyQueueLength,ReplayQueueLength -AutoSize
Test-ReplicationHealth -Identity EX01
Get-MailboxServer EX01 | Format-List DatabaseCopyAutoActivationPolicy

Checklist

1. 1

   Drain transport

   powershell

   Set-ServerComponentState EX01 -Component HubTransport -State Draining -Requester Maintenance
   Restart-Service MSExchangeTransport

   The restart makes the Transport service pick up the draining state straight away. Microsoft's procedure restarts only MSExchangeTransport; Front End Transport is a stateless proxy and doesn't queue mail.

2. 2

   Move databases and pause the cluster node

   powershell

   cd $ExScripts
   .\StartDagServerMaintenance.ps1 -ServerName EX01 -MoveComment Maintenance -PauseClusterNode

   The script can run for several minutes without printing anything, so be patient. Both DAG scripts accept -WhatIf if you want a dry run first.

3. 3

   Redirect queued messages

   powershell

   Redirect-Message -Server EX01 -Target EX02.contoso.com -Confirm:$false
   Get-Queue -Server EX01 | Where-Object { $_.MessageCount -gt 0 }

   The target must be the FQDN of another Mailbox server that isn't in maintenance. Repeat the Get-Queue check until it returns nothing, or only Shadow Redundancy queues.

4. 4

   Take the server offline

   powershell

   Set-ServerComponentState EX01 -Component ServerWideOffline -State Inactive -Requester Maintenance

   If your load balancer probes each protocol's healthcheck.htm page, those probes now fail and EX01 drops out of the pool. If it only checks that a port is open, disable EX01 in the load balancer yourself.

5. 5

   Confirm, then patch

   powershell

   Get-ServerComponentState EX01 | Format-Table Component,State -AutoSize
   Get-MailboxServer EX01 | Format-List DatabaseCopyAutoActivationPolicy
   Get-ClusterNode EX01 | Format-List Name,State
   Get-MailboxDatabaseCopyStatus -Server EX01 | Where-Object { $_.Status -eq "Mounted" }

   You're ready when only Monitoring and RecoveryActionsEnabled are still active, the policy shows Blocked, the node is Paused and the last command returns nothing.

6. 6

   Bring the server back

   powershell

   Set-ServerComponentState EX01 -Component ServerWideOffline -State Active -Requester Maintenance
   cd $ExScripts
   .\StopDagServerMaintenance.ps1 -ServerName EX01
   Set-ServerComponentState EX01 -Component HubTransport -State Active -Requester Maintenance
   Restart-Service MSExchangeTransport

   The stop script resumes the cluster node, sets the activation policy to Unrestricted and resumes the server's database copies. If your original policy was different, put it back, for example Set-MailboxServer EX01 -DatabaseCopyAutoActivationPolicy IntrasiteOnly.

7. 7

   Rebalance, then move to the next server

   Since Exchange 2016 CU2, the DAG's PreferenceMoveFrequency setting rebalances active databases automatically. To rebalance immediately instead:

   powershell

   cd $ExScripts
   .\RedistributeActiveDatabases.ps1 -DagName DAG01 -BalanceDbsByActivationPreference

Verify

Get-ServerComponentState EX01 | Where-Object { $_.State -ne "Active" }
Get-ClusterNode | Format-Table Name,State
Get-MailboxDatabaseCopyStatus -Server EX01 |
  Format-Table Name,Status,CopyQueueLength,ReplayQueueLength -AutoSize
Test-ServiceHealth EX01
Get-Command ExSetup.exe | ForEach-Object { $_.FileVersionInfo.FileVersion }

Good looks like this:

• Only ForwardSyncDaemon and ProvisioningRps are listed; both are inactive by default on-premises.
• Every cluster node is Up.
• EX01's copies are Healthy or Mounted, with low queue lengths.
• RequiredServicesRunning is True for every role listed.
• The ExSetup version matches the build Microsoft lists for the update you installed, and a test message flows through EX01.

Tips & gotchas

• Components still inactive after a failed update? Setup records its own requester. Microsoft's fix is to set ServerWideOffline, Monitoring and RecoveryActionsEnabled back to Active with -Requester Functional. The LocalStates and RemoteStates properties of Get-ServerComponentState show which requester set each state.
• Two-member DAGs. While one member is paused or rebooting, the other carries every database with no redundancy, and quorum depends on the file share witness. Check that the witness server is reachable before you reboot.
• Prefer the scripts to hand-typed sequences. They handle the PAM move and the activation block for you, and the start script rolls itself back if something fails.
• Review mitigations after patching. An update can make an Emergency Mitigation service mitigation unnecessary; see Exchange Emergency Mitigation service and AMSI for the checks.