Exchange components stuck Inactive after an update: ServerWideOffline and requesters

You install a cumulative or security update, the server restarts, and then mail stops flowing or Outlook on the web is unreachable through that server. A common cause is server component state: Exchange's own on/off switches for each protocol. In this post I'll show how to read Get-ServerComponentState, work out which requester is holding a component Inactive, and bring the server back without fighting Exchange's safety mechanisms.

Symptoms#

• Mail stops flowing through the server after the update, and queues on other servers that point to it start growing.
• Load balancer health probes fail, so the server drops out of the pool; clients that connect to it directly can't reach Outlook on the web, EWS or ActiveSync.
• The POP3 and IMAP4 services stop on their own. That's by design while their proxy components are Inactive.
• Health Checker shows the yellow Exchange Server Maintenance warning instead of "Server is not in Maintenance Mode".

The Exchange Management Shell confirms it:

Get-ServerComponentState -Identity EX01 | Where-Object State -ne 'Active' | Format-Table Component, State

Component          State
---------          -----
ServerWideOffline  Inactive
HubTransport       Inactive
FrontendTransport  Inactive
...

The lines to look for are ServerWideOffline Inactive, HubTransport Inactive and FrontendTransport Inactive.

Why it happens#

Every Exchange server has a set of components, such as HubTransport, FrontendTransport, OwaProxy, EwsProxy and ActiveSyncProxy, and each one is Active, Inactive or Draining. ServerWideOffline is the master switch: setting it Inactive takes every component offline at once, which is exactly what maintenance mode does.

Each change is recorded with a requester: HealthApi, Maintenance, Sidelined, Functional or Deployment. A component is only Active when no requester holds it Inactive, so setting it Active as Maintenance won't release a hold placed by another requester. The states are stored twice: in the server's registry (LocalStates) and in Active Directory (RemoteStates).

After an update, components usually stay down for one of three reasons:

• Maintenance mode was never ended. The patching procedure set ServerWideOffline Inactive as Maintenance, and the closing command didn't run. Setup doesn't end your maintenance mode for you.
• The update was interrupted. When setup fails part-way, services can be left Disabled, and admins report components left Inactive as well. If the hold isn't from Maintenance, it didn't come from your maintenance procedure.
• Managed Availability stepped in. HealthApi means Exchange's built-in health engine took a protocol offline because its probes kept failing, for example after the update broke a certificate binding or a config file. Here the component state is the symptom, not the problem.

ForwardSyncDaemon and ProvisioningRps are normally Inactive on on-premises servers; leave those two alone.

How to fix it#

1. Find out who holds the component#

$c = Get-ServerComponentState -Identity EX01 -Component ServerWideOffline
$c.LocalStates  | Format-List
$c.RemoteStates | Format-List
Search-AdminAuditLog -Cmdlets Set-ServerComponentState -StartDate (Get-Date).AddDays(-7) |
  Format-List RunDate, Caller, ObjectModified, CmdletParameters

Each state entry shows a requester, its state and a timestamp. The admin audit log shows whether a person or a patching script ran the cmdlet. Repeat for any other component that's still Inactive.

2. Make sure the update really finished#

Don't bring a half-upgraded server back into service. Check the build with Get-Command ExSetup.exe | ForEach-Object { $_.FileVersionInfo }, check services with Test-ServiceHealth, and look for leftover setup Watermark or Action registry values. If setup didn't complete, finish it first: see recovering a failed Exchange upgrade, which also covers restoring Disabled services.

3. Release the hold with the same requester#

For maintenance mode, end it the way it was started:

Set-ServerComponentState -Identity EX01 -Component ServerWideOffline -State Active -Requester Maintenance
Set-ServerComponentState -Identity EX01 -Component HubTransport -State Active -Requester Maintenance

On a DAG member, also resume the cluster node and database activation as described in patching a DAG without downtime.

If step 1 showed a different requester and the update is complete, release the hold with that requester, for example:

Set-ServerComponentState -Identity EX01 -Component ServerWideOffline -State Active -Requester Functional

Avoid -LocalOnly and -RemoteOnly here. They write to only the registry or only Active Directory, which is how the two copies end up disagreeing.

4. Restart what depends on the state#

Start POP3 and IMAP4 again if you use them, because they may not come back by themselves once PopProxy and ImapProxy are Active. If mail doesn't resume within a few minutes, restart MSExchangeTransport and MSExchangeFrontEndTransport, and check that the queue database drive has enough free space.

Verify the fix#

• Get-ServerComponentState -Identity EX01 | Where-Object State -ne 'Active' returns only ForwardSyncDaemon and ProvisioningRps.
• The Health Checker reports "Server is not in Maintenance Mode" and no difference between LocalStates and RemoteStates.
• Invoke-WebRequest https://ex01.contoso.com/owa/healthcheck.htm -UseBasicParsing returns status code 200, and the load balancer marks the server healthy.
• Queues on the other servers drain (Get-Queue), and a test message shows RECEIVE and DELIVER or SEND events in Get-MessageTrackingLog.

Prevent it next time#

• Script maintenance mode as a pair: whatever turns it on also turns it off and then runs the not-Active check above.
• Run Health Checker after every CU and SU; its maintenance check compares LocalStates and RemoteStates for you.
• Treat a HealthApi hold as an alert to investigate, not a setting to override.
• Finish failed updates with setup rather than by flipping component states on a half-upgraded server.

References#

• Set-ServerComponentState (Microsoft Learn)
• Health Checker maintenance check (Microsoft CSS-Exchange)
• Fix failed Exchange Server updates (Microsoft Learn)