DLP and encryption gateways for Exchange Server (Forcepoint, Trellix): routing and loops

Inline DLP and encryption products such as Forcepoint DLP, Trellix DLP Network Prevent and email encryption gateways ask the same of Exchange Server: send outbound mail to them, then let them pass it on or hand it back. Get the hand-off wrong and you get loops, mail that quietly skips the scanner, or queues that fill when the appliance is down. This vendor-neutral guide covers patterns, connectors and tests.

How it works#

How Exchange picks a route#

For an external recipient, Exchange drops connectors whose size limit is smaller than the message, keeps those that are enabled, in scope for the local Active Directory site and matching the recipient's address space, takes the most specific address space, and then prefers the lowest aggregate cost (site link costs plus the address space cost, 1 to 100). Remaining ties go to the source server reachable in the fewest hops (the local server first), then to the connector whose name sorts first. Four consequences:

• Routing never looks at how a message arrived, and Exchange Server's mail flow rules have no action that picks a send connector. A gateway that hands mail back to the servers that sent it creates a loop ending in 554 5.4.6 Hop count exceeded - possible mail loop.
• Connectors with more specific address spaces (hybrid, partner, archive) bypass a * connector to the gateway.
• A message larger than the gateway connector's MaxMessageSize can leave, uninspected, through another enabled * connector.
• Mail flow rules can act on message headers, which is how verdict headers work.

How the products hand mail on#

• Forcepoint DLP: the protector can act as an MTA. In the Data Security module of the Forcepoint Security Manager (Settings › Deployment › System Modules), you set the protector's SMTP service mode to Mail Transfer Agent (MTA), choose Monitoring or Blocking, and set the next hop MTA (smart host), the maximum message size (33 MB by default) and the networks allowed to send through it. The next hop must allow the protector to relay.
• Trellix DLP Network Prevent (formerly McAfee DLP Prevent) delivers scanned mail to a configured smart host with an X-RCIS-Action header (ALLOW when nothing matched), and the smart host carries out the action. If that smart host is the same Exchange tier that sent the message, you have the loop pattern.
• Symantec Network Prevent for Email has forwarding and reflecting modes; see Symantec DLP with Exchange.

Step-by-step#

1. Choose the pattern#

A scoped connector (-IsScopedConnector $true) is only a candidate for servers in its own Active Directory site, which stops the site B relay servers routing back to the gateway. Nothing else should submit mail to them.

2. Build the connector to the gateway#

New-SendConnector -Name "Outbound via DLP gateway" -Usage Custom -AddressSpaces "SMTP:*;1" `
  -SmartHosts dlpgw1.contoso.com,dlpgw2.contoso.com -DNSRoutingEnabled $false `
  -SmartHostAuthMechanism None -SourceTransportServers EX01,EX02 -MaxMessageSize 30MB
Get-SendConnector | Format-Table Name,AddressSpaces,SmartHosts,MaxMessageSize,IsScopedConnector,Enabled -AutoSize

Set MaxMessageSize no higher than the gateway accepts, and disable other * connectors these servers could use, so oversized mail fails visibly instead of leaving uninspected. For the separate-tier design, add -IsScopedConnector $true here and create a scoped internet connector sourced on the site B servers.

3. Accept the return traffic (separate tier only)#

New-ReceiveConnector -Name "From DLP gateway" -Server RELAY01 -TransportRole FrontendTransport -Usage Custom `
  -Bindings 0.0.0.0:25 -RemoteIPRanges 10.0.5.21,10.0.5.22 -AuthMechanism ExternalAuthoritative -PermissionGroups ExchangeServers

This "Externally Secured" connector treats the gateway as a trusted internal source, so limit RemoteIPRanges to the gateway's addresses; that more specific match also wins over the default connectors.

4. Act on verdict headers#

Where the product reports its verdict in a header, mail flow rules on the servers receiving scanned mail act on it. Trellix's values are ALLOW, BLOCK, QUART, ENCRYPT, BOUNCE, REDIR, NOTIFY, BYPASS and SCANFAIL (a message that couldn't be analyzed); when several rules match, the highest-priority value wins. Two example rules, plus one that strips the header from inbound internet mail: Trellix replaces an existing X-RCIS header on mail it scans, but Exchange's header firewall doesn't remove vendor X-headers, so a sender could plant a verdict on mail that never passes the scanner:

New-TransportRule -Name "DLP verdict BLOCK" -FromScope InOrganization `
  -HeaderMatchesMessageHeader "X-RCIS-Action" -HeaderMatchesPatterns "BLOCK" `
  -RejectMessageReasonText "Blocked by the data loss prevention policy" -RejectMessageEnhancedStatusCode "5.7.1"
New-TransportRule -Name "DLP verdict QUART" -FromScope InOrganization `
  -HeaderMatchesMessageHeader "X-RCIS-Action" -HeaderMatchesPatterns "QUART" -RedirectMessageTo dlp-review@contoso.com
New-TransportRule -Name "Strip X-RCIS-Action from external mail" -FromScope NotInOrganization -RemoveHeader "X-RCIS-Action"

For ENCRYPT, Exchange can apply a rights management template with -ApplyRightsProtectionTemplate, which needs IRM (see IRM with the RMS connector), or tag the message for a downstream encryption gateway.

5. Decide fail-open or fail-closed#

• Gateway side: Forcepoint asks what to do when an unspecified error stops analysis, Permit traffic or Block traffic. Trellix marks such messages SCANFAIL and leaves the decision to your rules.
• Exchange side: when no smart host answers, the queue goes to Retry with 441 4.4.1 Error encountered while communicating with primary target IP address, and messages wait until they expire (two days by default): fail-closed. List at least two gateway nodes as smart hosts. For a manual fail-open, keep a disabled bypass connector and swap it in, knowing that mail sent meanwhile isn't inspected.
• Timeouts: if sessions drop while large attachments are scanned, the send protocol log shows which side closed the connection (see SMTP protocol logging); compare that with the gateway's timeouts.

6. Add TLS#

Set-SendConnector "Outbound via DLP gateway" -RequireTLS $true -TlsAuthLevel CertificateValidation

The gateway's certificate must chain to a CA your Exchange servers trust, and the gateway must trust its own next hop.

Verify#

Get-TransportService | Get-MessageTrackingLog -Recipients test@fabrikam.com -Start (Get-Date).AddHours(-1) -ResultSize Unlimited |
  Sort-Object Timestamp | Format-Table Timestamp,ServerHostname,EventId,Source,ConnectorId -AutoSize

• Forward design: one SEND through Outbound via DLP gateway per message. Separate tier: that SEND, one RECEIVE on From DLP gateway and one SEND through the site B connector. Repeating RECEIVE and SEND pairs mean a loop.
• Send one message just under the size limit and one just over: the first is scanned, the second must not leave through another connector.
• Check the external copy's headers for the product's verdict or pass header.
• With one gateway node stopped, the queue shouldn't go to Retry.

Tips and gotchas#

• Journal reports to an external archive also match *. Give the archive its own connector with a specific address space so DLP can't block compliance copies (see journaling to a third-party archive).
• Edge Transport servers keep their own, unreplicated mail flow rules; verdict rules that must run there go on each Edge server.
• Gateways in front of inbound mail are a different job: see putting an email security gateway in front of Exchange.

References#

• Connector selection in external message routing (Microsoft Learn)
• Send connectors in Exchange Server (Microsoft Learn)
• Trellix DLP Network Prevent X-RCIS-Action header behavior (Trellix documentation)
• Protector SMTP service: Mail Transfer Agent (MTA) tab (Forcepoint documentation)