DAG cluster problems: lost quorum, file share witness errors and Event ID 1135

Every Exchange database availability group (DAG) runs on a Windows failover cluster that Exchange creates and manages for you. When that cluster drops a member or loses quorum, databases fail over unexpectedly or dismount on every server at once. In this post I'll explain how DAG quorum and the file share witness work, how to read Event ID 1135 and 1177, how to fix witness errors, and why Datacenter Activation Coordination (DAC) mode belongs on multi-site DAGs.

Symptoms#

In the System log of the members that stayed up (source Microsoft-Windows-FailoverClustering):

Event ID 1135
Cluster node 'EX02' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Event ID 1177
The Cluster service is shutting down because quorum was lost. ...

On the Exchange side, a 1135 usually comes with databases failing over away from the removed member. Lost quorum (1177) is worse: the cluster service stops on the remaining members and databases dismount across the DAG. Test-ReplicationHealth flags checks such as ClusterService, DagMembersUp or QuorumGroup; an unreachable witness looks like this:

Server  Check            Result    Error
------  -----            ------    -----
EX01    FileShareQuorum  *FAILED*  Failures:...

Witness problems often show up earlier, when you create the DAG, add members or change the witness:

The task was unable to create the default witness directory on server FSW01. Please manually specify a witness directory.
The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server FSW01.
Unable to access file shares on witness server FSW01. Until this problem is corrected the database availability group may be more vulnerable to failures. You can use the Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: The network path was not found.

The details are in the DAG task logs under C:\ExchangeSetupLogs\DagTasks on the server where you ran the command.

Why it happens#

How DAG quorum works#

The cluster keeps running only while a majority of its votes can talk to each other. Exchange picks the model for you: with an odd number of members, each server votes (node majority); with an even number, the witness server adds a tie-breaking vote (node and file share majority). The witness is used only when the member count is even, and only for quorum.

Windows Server's dynamic quorum can adjust votes when members leave one at a time, but don't design around it: a missing witness in an even-sized DAG is a real loss of resilience.

Why a node gets removed (1135)#

Cluster members exchange heartbeats on UDP port 3343, and a member that misses enough of them is removed by the others. Typical reasons are network drops, NIC driver, firmware or teaming problems, virtual machines paused by host operations, an overloaded host, firewalls or WAN trouble between sites, and plain reboots or crashes. Microsoft's 1135 guide suggests a useful pattern check: if every node reports losing the same node, look at that node; if the nodes in one site all lose the other site, look at the link between the sites.

Why the witness fails#

The witness server can't be a DAG member and must be in the same Active Directory forest. Exchange creates and secures the witness directory and share itself, so if the witness isn't an Exchange server, the Exchange Trusted Subsystem group must be in its local Administrators group. Failures come from that membership missing, the witness server being down, SMB (TCP 445) blocked from the DAG members, or someone deleting the share or folder while tidying up.

How to fix it#

Run the cluster cmdlets below on a DAG member.

1. Get the picture#

Get-DatabaseAvailabilityGroup DAG01 -Status |
  Format-List Servers,OperationalServers,PrimaryActiveManager,WitnessServer,WitnessDirectory,WitnessShareInUse,DatacenterActivationMode
Test-ReplicationHealth -Identity EX01 | Where-Object { $_.Result.Value -ne "Passed" } |
  Format-List Check,Result,Error
Get-ClusterNode | Format-Table Name,State,NodeWeight,DynamicWeight -AutoSize
Get-ClusterQuorum | Format-List QuorumResource,QuorumType

2. Investigate Event ID 1135#

Get-WinEvent -FilterHashtable @{ LogName = 'System'; ProviderName = 'Microsoft-Windows-FailoverClustering'; Id = 1135,1177,1069 } -MaxEvents 50 |
  Format-Table TimeCreated,Id,Message -Wrap
New-Item -ItemType Directory -Path C:\ClusterLogs -Force | Out-Null
Get-ClusterLog -Destination C:\ClusterLogs -TimeSpan 120 -UseLocalTime

Compare the times across members and against the removed node's own logs: a reboot, a backup window, a VM migration or a NIC reset in the same minute is usually the answer. Then:

• Run the cluster network validation tests, which don't disrupt a running DAG: Test-Cluster -Node EX01,EX02 -Include "Network".
• Update NIC drivers and firmware, review teaming, and make sure UDP 3343 is open between all members.
• On virtual machines, don't use hypervisor snapshots or checkpoints (Microsoft doesn't support them for Exchange), and make sure the host isn't overcommitted.
• Treat raising the heartbeat thresholds (Get-Cluster | Format-List *SubnetDelay,*SubnetThreshold) as a last resort: it hides network problems and slows detection of real failures.

3. Fix the file share witness#

# From each DAG member
Test-NetConnection FSW01 -Port 445

# On the witness server, if it isn't an Exchange server
Get-LocalGroupMember -Group Administrators
Add-LocalGroupMember -Group Administrators -Member "CONTOSO\Exchange Trusted Subsystem"

# Re-apply (or move) the witness from the Exchange Management Shell
Set-DatabaseAvailabilityGroup -Identity DAG01 -WitnessServer FSW01.contoso.com -WitnessDirectory "C:\DAGFileShareWitnesses\DAG01.contoso.com"

# Back on a DAG member
Get-ClusterResource | Where-Object { $_.ResourceType.Name -eq "File Share Witness" } |
  Format-Table Name,State,OwnerGroup -AutoSize

Exchange recreates the folder and share if they're missing. In an even-sized DAG the witness resource should be Online; an odd-sized DAG normally has no witness resource at all. Admins have reported the Exchange Trusted Subsystem warning appearing even when the membership is correct, so if you've added the group, confirm with the DagTasks log and the share itself before changing anything else. A few rules:

• Don't use a domain controller as the witness. On a DC, the local Administrators group is the domain's built-in Administrators group, so you'd be giving Exchange servers admin rights over the domain.
• Another Exchange server outside the DAG makes a good witness: the Exchange Trusted Subsystem permissions are already in place.
• Don't change quorum or witness settings in Failover Cluster Manager. Exchange manages them, and Microsoft doesn't support using a DAG's cluster for anything other than the DAG.

4. Recover from lost quorum#

Quorum returns once a majority of votes can talk again. Bring members or the witness back first, then check the databases:

Get-MailboxDatabaseCopyStatus * |
  Where-Object { $_.Status -in "Dismounted","Failed","ServiceDown" } |
  Format-Table Name,Status -AutoSize
Mount-Database DB01

If a whole site is gone and the survivors can't reach a majority, don't force quorum with the cluster tools. Follow Exchange's datacenter switchover procedure instead: Stop-DatabaseAvailabilityGroup for the failed site, then Restore-DatabaseAvailabilityGroup for the surviving one, which shrinks the cluster to the surviving members and can use the alternate witness server configured on the DAG.

5. Turn on DAC mode#

Set-DatabaseAvailabilityGroup -Identity DAG01 -DatacenterActivationMode DagOnly

DAC mode is off by default. With it on, a member that starts up can't mount databases until it can talk to other DAG members. That prevents split brain, where both halves of a DAG mount the same databases after a site outage. Enable it on any DAG with members in more than one site.

Verify the fix#

Test-ReplicationHealth -Identity EX01
Get-ClusterNode | Format-Table Name,State -AutoSize
Get-DatabaseAvailabilityGroup DAG01 -Status | Format-List OperationalServers,WitnessShareInUse,PrimaryActiveManager

Good looks like ClusterService, DagMembersUp, ClusterNetwork, QuorumGroup and FileShareQuorum all Passed; every node Up; all members listed in OperationalServers; and, in an even-sized DAG, WitnessShareInUse showing Primary. No new 1135 or 1177 events should appear in the following days.

Prevent it next time#

• Place the witness where it survives the failures you care about; for a DAG spanning two sites, consider a third location both sites can reach.
• Treat the witness server as part of Exchange: monitor it and document that its witness folder must stay.
• Run Test-ReplicationHealth on a schedule and alert on any failed check.
• Patch members one at a time with maintenance mode, which pauses the cluster node properly: Patching a DAG without downtime.
• If members stay in the cluster but copies fall behind, see High CopyQueueLength or ReplayQueueLength in a DAG.

References#

• Manage database availability groups (Microsoft Learn)
• Troubleshoot cluster issue with Event ID 1135 (Microsoft Learn)
• Monitor database availability groups (Microsoft Learn)