Database copy FailedAndSuspended in an Exchange DAG: find the cause and reseed safely

When a passive copy in a database availability group (DAG) shows FailedAndSuspended, Exchange has stopped trying to repair it, and that database is running with one copy fewer than you designed for. In this post I'll show how to read the real reason from the copy status and event logs, fix what broke the copy, reseed it with Update-MailboxDatabaseCopy, and prove it's healthy again on Exchange Server SE and Exchange 2019 (Exchange 2016 works the same way).

Symptoms#

The quickest check is the copy status of the affected database:

Get-MailboxDatabaseCopyStatus -Identity DB01 |
  Format-Table Name,Status,CopyQueueLength,ReplayQueueLength -AutoSize

Name       Status              CopyQueueLength ReplayQueueLength
----       ------              --------------- -----------------
DB01\EX01  Mounted                           0                 0
DB01\EX02  FailedAndSuspended             1874                 0

Test-ReplicationHealth on the passive server flags the copy in one or both of the copy checks:

Server  Check            Result    Error
------  -----            ------    -----
EX02    DBCopySuspended  *FAILED*  Failures:...
EX02    DBCopyFailed     *FAILED*  Failures:...

You'll usually also see:

• The copy queue keeps growing, because no logs reach the failed copy.
• In the Exchange admin center, Servers › Databases shows the failed state for that copy in the details pane.
• If you reseed several copies from the same source at once, the seed fails with SeedInProgressException error FE1C6491 (covered in step 4).

The difference between the two failure states matters. A copy that is only Failed isn't suspended, so the Replication service keeps checking it and returns it to Healthy by itself once the problem clears. FailedAndSuspended means the system decided the failure needs an administrator, for example because it found divergence between the active and passive copies that it can't resolve. It won't retry on its own.

Why it happens#

Microsoft lists the situations that require a reseed, and they explain most FailedAndSuspended copies:

• A failover lost data and left the old copy diverged and unrecoverable.
• A log file is corrupt and can't be replayed into the passive copy.
• Any copy of the database went through an offline defragmentation (eseutil /d).
• The database's log generation sequence was reset to 1.

Behind those, the usual triggers are storage problems on the passive server (a failed or full disk, I/O errors, a missing mount point), log files deleted by hand or quarantined by antivirus before every copy had them, and restores or repairs done on one copy without thinking about the others. Network outages normally produce DisconnectedAndHealthy or Failed instead, which recover without a reseed once the path is back; for that case see High CopyQueueLength or ReplayQueueLength in a DAG.

How to fix it#

1. Read the error the copy reports#

Get-MailboxDatabaseCopyStatus -Identity "DB01\EX02" |
  Format-List Name,Status,ErrorMessage,ErrorEventId,SuspendComment,LastStatusTransitionTime,DiskFreeSpacePercent

Then open Event Viewer on the passive server at Applications and Services Logs › Microsoft › Exchange › MailboxDatabaseFailureItems, which logs failures that affect replicated databases, and the HighAvailability channel next to it. From the shell:

Get-WinEvent -LogName "Microsoft-Exchange-MailboxDatabaseFailureItems/Operational" -MaxEvents 20 |
  Format-Table TimeCreated,Id,Message -Wrap

Match what you find to a cause:

2. Fix the cause first#

Reseeding onto a disk that is still failing just gives you another failed copy hours later. Check the passive server's System log for disk and controller errors, make sure the volumes have room for a full copy of the database plus its logs, and confirm those folders are still excluded from antivirus scanning.

3. Try one resume if the cause was temporary#

If the problem was transient and is now fixed (a volume that filled up and has been cleaned, a disk that came back), one resume is worth trying:

Resume-MailboxDatabaseCopy -Identity "DB01\EX02"

Microsoft notes that resuming a copy that is still in a bad state just makes replication fail and suspends the copy again. If it drops back to FailedAndSuspended, stop and reseed.

4. Reseed the copy#

Two prerequisites: the copy must be suspended (a FailedAndSuspended copy already is; a Failed one needs Suspend-MailboxDatabaseCopy first), and the Remote Registry service must be running on the server that holds the passive copy.

# Only needed if the copy is Failed but not suspended
Suspend-MailboxDatabaseCopy -Identity "DB01\EX02" -SuspendComment "Reseed" -Confirm:$false

Get-Service RemoteRegistry -ComputerName EX02
Update-MailboxDatabaseCopy -Identity "DB01\EX02" -DeleteExistingFiles

-DeleteExistingFiles clears the old copy's files on the target before seeding; without it the seed fails when it finds them. If the cmdlet still complains about files in the folder, move them out manually. Useful variations:

• -SourceServer EX03 seeds from a healthy passive copy, for example one in the same site so the seed doesn't cross the WAN.
• -Network DAG01\ReplicationDagNetwork01 picks the DAG network the seed uses.
• -ManualResume leaves the copy suspended after seeding, so you can check it before running Resume-MailboxDatabaseCopy.
• -BeginSeed starts the seed and returns at once. Without it, keep the shell open: closing it ends the seed.
• -DatabaseOnly is required when you seed several copies from the same source at once; otherwise the task fails with SeedInProgressException error FE1C6491.

If a whole server lost its disks, Update-MailboxDatabaseCopy -Server EX02 reseeds every copy on it. The EAC works too: in Servers › Databases, select the database, then choose Suspend and Update under the passive copy in the details pane. While seeding runs, the target shows Seeding and the source shows SeedingSource; the duration depends on database size, bandwidth, latency and storage speed.

Verify the fix#

Get-MailboxDatabaseCopyStatus -Identity DB01 |
  Format-Table Name,Status,CopyQueueLength,ReplayQueueLength,ContentIndexState -AutoSize
Test-ReplicationHealth -Identity EX02

Good looks like this: after a short Initializing phase, the reseeded copy shows Healthy, both queues are at or near zero, and every Test-ReplicationHealth check returns Passed. On Exchange 2016, ContentIndexState should be Healthy as well; Exchange 2019 and SE keep search data inside the database, so there's no separate catalog to check (see Content index Failed or Unknown). For final proof, activate the reseeded copy in a quiet period with Move-ActiveMailboxDatabase DB01 -ActivateOnServer EX02, open a test mailbox, then move the database back.

Prevent it next time#

• Never delete transaction logs by hand. Let backups or circular logging truncate them, so Exchange can coordinate truncation across all copies.
• Keep antivirus exclusions for database, log and search folders in place, and re-check them after every agent upgrade.
• Watch disk space. Managed Availability's low-space monitor only watches volumes that hold databases and logs, with a default threshold of 180 GB. You can tune it with the SpaceMonitorLowSpaceThresholdInMB value under HKLM\SOFTWARE\Microsoft\ExchangeServer\v15\Replay\Parameters.
• Check copy health every day. A scheduled Test-ReplicationHealth catches a failed copy long before a second failure turns it into an outage.
• Patch with maintenance mode so updates don't leave copies suspended; see Patching a DAG without downtime.

References#

• Monitor database availability groups (Microsoft Learn)
• Update a mailbox database copy (Microsoft Learn)
• Update-MailboxDatabaseCopy (Microsoft Learn)