Custom sensitive information types in Purview: regex, keywords, confidence and testing

Built-in sensitive information types (SITs) know credit card and passport numbers, but not your project codes, contract numbers or internal account IDs. A custom SIT teaches Microsoft Purview what that data looks like, and every DLP and auto-labeling policy can then use it. In this post I'll build one in the portal, show the XML rule package for finer control, and test it before any policy depends on it.

Prerequisites#

• An admin account that can manage classifiers in the Microsoft Purview portal (see Purview roles and role groups), plus Security & Compliance PowerShell (Connect-IPPSSession) for testing and XML uploads.
• Real-looking sample text: a few true matches and, just as important, near-misses that must not match.
• Your own regex expertise. Microsoft states that its support engineers can't build content-matching definitions for you.

Step-by-step#

1. Design the pattern before you click#

A SIT is one or more patterns. Each pattern has a primary element (a regular expression with an optional validator, a keyword list, a keyword dictionary or a built-in function), optional supporting elements found within a character proximity window, and a confidence level: Low, Medium or High (65, 75 and 85 in XML). The usual design is a bare regex at Low, regex plus a keyword at High, so policies can notify on Low and block on High.

2. Write a regex Purview will accept#

Purview uses the Boost.RegEx 5.1.3 engine and enforces a few rules:

• No positional anchors such as ^ and $.
• Exactly one capturing group; use non-capturing groups for prefixes and suffixes, and put variants inside the single group with |.
• Lookbehinds must be fixed length, and a pattern can't start or end with | or .{0,m}, or put an unbounded repeater on a group.

For a project code such as PRJ-204518: \b(PRJ-[0-9]{6})\b. Word boundaries (\b) are safer than \s delimiters, especially if the SIT later becomes an exact data match primary element.

3. Create the SIT in the portal#

1. Go to Information Protection › Classifiers › Sensitive info types and choose Create sensitive info type.
2. Enter a name and description, then choose Create pattern.
3. Pick the pattern's default confidence level, set the regex as the primary element and set Character proximity.
4. Add supporting elements (for example a keyword list with "project code" and "confidential project") and any additional checks, then choose Create.
5. Add a second, lower-confidence pattern without the keyword, choose the recommended confidence level for the SIT, and save.

Custom SITs created this way land in the Microsoft.SCCManaged.CustomRulePack rule package.

4. Or copy and tune a built-in SIT#

Select a built-in SIT and choose Copy, then edit the copy's patterns, for example adding a keyword requirement to cut noise. A few SITs can't be copied (including the EU and U.S. driver's license types and the ICD code types), and a SIT with ten text processors has to be copied with PowerShell instead.

5. Use a rule package for full control#

XML lets you exclude false-positive keywords, require several unique matches, or manage SITs as code. A trimmed example (generate your own GUIDs with [guid]::NewGuid()):

<?xml version="1.0" encoding="utf-16"?>
<RulePackage xmlns="http://schemas.microsoft.com/office/2011/mce">
 <RulePack id="00000000-0000-0000-0000-000000000001">
  <Version major="1" minor="0" build="0" revision="0"/>
  <Publisher id="00000000-0000-0000-0000-000000000002"/>
  <Details defaultLangCode="en-us"><LocalizedDetails langcode="en-us">
   <PublisherName>Contoso</PublisherName><Name>Contoso SITs</Name><Description>Project codes</Description>
  </LocalizedDetails></Details>
 </RulePack>
 <Rules>
  <Entity id="00000000-0000-0000-0000-000000000003" patternsProximity="300" recommendedConfidence="75">
   <Pattern confidenceLevel="65"><IdMatch idRef="Regex_project"/></Pattern>
   <Pattern confidenceLevel="85">
    <IdMatch idRef="Regex_project"/><Match idRef="Keyword_project"/>
    <Any minMatches="0" maxMatches="0"><Match idRef="Keyword_not_real"/></Any>
   </Pattern>
  </Entity>
  <Regex id="Regex_project">\b(PRJ-[0-9]{6})\b</Regex>
  <Keyword id="Keyword_project"><Group matchStyle="word"><Term>project code</Term><Term>confidential project</Term></Group></Keyword>
  <Keyword id="Keyword_not_real"><Group matchStyle="word"><Term>sample</Term><Term>template</Term></Group></Keyword>
  <LocalizedStrings><Resource idRef="00000000-0000-0000-0000-000000000003">
   <Name default="true" langcode="en-us">Contoso project code</Name>
   <Description default="true" langcode="en-us">Detects Contoso project codes</Description>
  </Resource></LocalizedStrings>
 </Rules>
</RulePackage>

The Any block with minMatches="0" maxMatches="0" means "none of these words nearby". Every entity needs recommendedConfidence, or policies that use the SIT can't be saved. Save the file with Unicode encoding and upload it:

Connect-IPPSSession
New-DlpSensitiveInformationTypeRulePackage -FileData ([System.IO.File]::ReadAllBytes('C:\SIT\ContosoSITs.xml'))
Get-DlpSensitiveInformationType -Identity "Contoso project code" | Format-List Name,Publisher

To change it later, raise the Version numbers and upload with Set-DlpSensitiveInformationTypeRulePackage; without a new version number the package isn't deployed. If the validator spots a performance risk, the upload is refused with one of these messages:

Generic quantifiers which match more content than expected (e.g., '+', '*')
Lookaround assertions
Complex grouping in conjunction with general quantifiers

Verify#

Test text before any policy uses the SIT:

$r = Test-DataClassification -TextToClassify "Kickoff for confidential project PRJ-204518 is Monday." -ClassificationNames "Contoso project code"
$r.ClassificationResults
$t = Test-TextExtraction -FileData ([System.IO.File]::ReadAllBytes('C:\SIT\sample.docx'))
(Test-DataClassification -TestTextExtractionResults $t.ExtractedResults).ClassificationResults

Good looks like your SIT listed with a count and the high confidence level when the keyword is present, the low level without it, and no result for your near-miss samples. Then:

• Add the SIT to a DLP policy in simulation and review the matches; see from simulation mode to enforcement.
• Existing SharePoint and OneDrive content is only classified with the new SIT after a recrawl, or after an on-demand classification scan.

Tips and gotchas#

• Long or huge keyword lists: move them into a keyword dictionary.
• Prefer matchStyle="word"; use string only for parts of words or Asian languages.
• Email: the body and each attachment are separate items, so the keyword must sit in the same item as the match.
• Your own records: for exact customer or employee data, exact data match beats a regex; for noisy matches see tuning DLP false positives.

References#

• Create custom sensitive information types (Microsoft Learn)
• Create a custom sensitive information type using PowerShell (Microsoft Learn)
• Sensitive information type limits (Microsoft Learn)