Third-party transport agents breaking Exchange mail flow (Trend Micro, ESET, Kaspersky)

On-server mail security (Trend Micro ScanMail for Exchange, ESET Mail Security for Microsoft Exchange Server or Kaspersky Security for Microsoft Exchange Servers, for example), signature and disclaimer tools plug into Exchange as transport agents: code that runs inside the transport process for every message. When one is slow, crashes or doesn't support your Exchange build, mail stalls on the whole server. Here's how to find the culprit and fix it on Exchange Server SE and 2019.

Symptoms#

• Mail through one server slows down or stops, and Get-Queue shows its Submission queue growing.
• Application log warnings from MSExchange Extensibility, event ID 1050, naming an agent and an event:

The execution time of agent '<agent name>' exceeded <n> milliseconds while handling event '<event>'
... However, Transport will continue processing this message.

ScanMail example (Trend Micro):
The execution time of agent 'ScanMail Routing Agent' exceeded X milliseconds while handling event 'OnResolvedMessage'
for message with InternetMessageId: 'Not Available'.

• Crashes: event 1000 shows Faulting application name: EdgeTransport.exe ... Faulting module name: <third-party agent DLL>, often with event 4999 and, in the System log, The Microsoft Exchange Transport service terminated unexpectedly.
• Back pressure (452 4.3.1 Insufficient system resources) as EdgeTransport.exe memory grows.
• The Transport service won't start after a product was removed, with Event ID 1052 (MSExchange Extensibility) or Event ID 16023 (MSExchange Extensibility) in the Application log; ESET documents both for agents left registered after its product was uninstalled.
• It began right after an Exchange cumulative update (CU), security update (SU) or vendor update.

Why it happens#

A transport agent is a .NET assembly built with the Exchange Transport Agents SDK and registered in agents.config (%ExchangeInstallPath%TransportRoles\Shared) on each server. Exchange loads it into the transport process and calls it at SMTP receive events such as OnEndOfData and categorizer events such as OnSubmittedMessage and OnResolvedMessage. The Transport service accepts every agent type, the Front End Transport service only SMTP receive agents (not on OnEndOfData), and the Mailbox Transport service no third-party agents.

Because agents run in-process and in line with mail flow:

• A slow agent holds every message at its event. Event 1050 means an agent spent unusually long on one event; transport carries on, but a backlog builds. Trend Micro documents one cause for its ScanMail Routing Agent: Detect new spam sources listing domains that aren't yours as organizational MX records.
• An unhandled exception in agent code can bring down EdgeTransport.exe; a message that keeps crashing it lands in the poison queue.
• Memory-hungry scanning inflates the process until back pressure refuses mail.
• Version drift: vendors support specific Exchange builds. A CU or SU replaces Exchange's files underneath the agent, and a version the vendor hasn't released for that build can fail straight after patching.

How to fix it#

1. List the agents on every transport service#

# Transport service (the default, -TransportService Hub)
Get-TransportAgent | Format-Table Identity, Enabled, Priority -AutoSize

# Front End Transport service
Get-TransportAgent -TransportService FrontEnd | Format-Table Identity, Enabled, Priority -AutoSize

# Agents that handled mail since the last service start
Get-TransportPipeline | Format-List

Run it on each server. The Microsoft agents you'll see are the Transport Rule, DLP Policy, Retention Policy, Supervisory Review, Malware, Text Messaging and System Probe Drop agents, plus the anti-spam agents if you ran Install-AntiSpamAgents.ps1. Anything else came with a third-party product.

2. Tie the errors to an agent#

Event 1050 names the agent. For crashes, match the faulting module in event 1000 to an agent's DLL:

Get-WinEvent -FilterHashtable @{LogName='Application'; ProviderName='MSExchange Extensibility'; Id=1050} -MaxEvents 20 |
    Format-List TimeCreated, Message
Get-WinEvent -FilterHashtable @{LogName='Application'; Id=1000} -MaxEvents 50 |
    Where-Object Message -like '*EdgeTransport.exe*' | Format-List TimeCreated, Message

# The DLL behind each registered agent
Select-String -Path (Join-Path $env:ExchangeInstallPath 'TransportRoles\Shared\agents.config') -Pattern '\.dll' |
    ForEach-Object { $_.Line.Trim() }

# Agent version and exact Exchange build
(Get-Item 'C:\Program Files\Vendor\Agent\VendorAgent.dll').VersionInfo.FileVersion
Get-Command ExSetup.exe | ForEach-Object { $_.FileVersionInfo.ProductVersion }

If the faulting module is an Exchange or .NET file, use the event 4999 crash guide instead.

3. Disable the suspect agent to prove it#

Disable-TransportAgent -Identity "Vendor Routing Agent"
Restart-Service MSExchangeTransport

# An agent in the Front End Transport service
Disable-TransportAgent -Identity "Vendor Receive Agent" -TransportService FrontEnd
Restart-Service MSExchangeFrontEndTransport

If a product registers several agents, disable them all, then re-enable one at a time. Follow the vendor's procedure where one exists. Microsoft documents that the agent cmdlets work by editing agents.config, so they also help when the Transport service won't start: ESET, for example, has you remove its ESET Filtering Agent and ESET Filtering AV Agent with Uninstall-TransportAgent in that situation, and edit the file only as a last resort.

If mail flows and the events stop, you have your culprit. If not, re-enable the agent and check file-level antivirus exclusions, back pressure and the poison and Submission queues.

4. Fix the cause#

• Update the product to a version the vendor supports on your exact Exchange build, then Enable-TransportAgent -Identity "Vendor Routing Agent" and restart the service.
• Correct the slow setting. For ScanMail's event 1050, Trend Micro's fix is at Spam Prevention › Content Scanning › New Spam Sources › Detect new spam sources in the ScanMail console: list only your own MX FQDNs or gateway IP addresses.
• Change the order only if the vendor says so. Agents run in event order; on the same event, priority 1 runs first: Set-TransportAgent -Identity "Vendor Signature Agent" -Priority 12, then restart.
• Remove leftovers of uninstalled products with the vendor's uninstaller, or Uninstall-TransportAgent -Identity "Old Vendor Agent" and a restart.

Verify the fix#

# Submission drains
Get-Queue -Identity "$env:COMPUTERNAME\Submission" | Format-List MessageCount, Status, Velocity

# No new 1050 warnings in the last 4 hours (no output = good)
Get-WinEvent -FilterHashtable @{LogName='Application'; ProviderName='MSExchange Extensibility'; Id=1050; StartTime=(Get-Date).AddHours(-4)} -ErrorAction SilentlyContinue

# The worker process stays up
(Get-Process EdgeTransport).StartTime

# A test message flows in seconds
Get-MessageTrackingLog -Sender test@contoso.com -Start (Get-Date).AddMinutes(-15) |
    Sort-Object Timestamp | Format-Table Timestamp, EventId, Source, ServerHostname

Good looks like: each agent in the state you intended, a re-enabled agent back in Get-TransportPipeline once mail has passed, Submission near zero and the same EdgeTransport.exe start time for days.

For agents that change messages (signatures, disclaimers), pipeline tracing snapshots a message before and after each agent: Set-TransportService EX01 -PipelineTracingEnabled $true -PipelineTracingSenderAddress test@contoso.com, send a test, read the folder in PipelineTracingPath, then switch it off and delete the snapshots, which are full message copies.

Prevent it next time#

• Keep an inventory of agents per server (vendor, DLL version, priority) and refresh it after every change.
• Before each CU or SU, check the vendor's compatibility statement for that build and its advice on suspending agents during updates; afterwards, confirm the agents are enabled and send a test (see the CU pre-flight checklist).
• Run as few agents as possible. If a gateway in front of Exchange already filters mail, decide whether you need on-box scanning too; for signatures, compare cloud signature services.
• File-level antivirus needs Microsoft's exclusions, and AMSI integration scans HTTP requests, not SMTP mail; Get-TransportAgent controls neither.

References#

• Transport agents in Exchange Server (Microsoft Learn)
• Slow mail delivery in Exchange that has transport rules configured (Microsoft Learn, event 1050 example)
• MSExchange Extensibility Event ID 1050 - ScanMail for Exchange (Trend Micro)
• Microsoft Transport Agent service will not start after removing ESET Mail Security for Microsoft Windows Exchange Server (KB3026) (ESET)