Extending detection: custom recognizers and languages

Generic detection cannot know your national identifier formats, your internal reference numbers or your language out of the box. TokenVeil is built to be extended for exactly this, without touching the rest of the system.

Two layers of detection

Detection combines two layers, and you can extend either one:

• Deterministic recognizers. Pattern and checksum rules for structured values: emails, IPs, IBANs, card numbers, phone numbers and national or company identifiers. These are high precision, because a checksum or a fixed shape is a proof rather than a guess.
• Named-entity recognition. A statistical model for values that have no fixed shape: person names, organizations and locations in free text. This is what catches a surname in the middle of a sentence.

The Community edition covers the deterministic layer. The commercial edition adds a trained NER model for free-text entities.

Adding your own recognizers

You can add custom recognizers for the identifiers that are specific to your organization or country: a national personal ID (with its checksum), a company registration number, policy numbers, customer IDs and internal reference formats. A recognizer is a pattern, optionally with a checksum or a validation rule, so it detects your identifiers with high precision and no false positives on look-alike numbers.

For known fixed values, such as internal project codenames, there is also a custom-terms mechanism: you list the exact strings to always anonymize, per user or across the whole instance.

Adding languages

The engine ships with French and English. The named-entity layer is model driven, so a new language is added by bringing in a detection model for that language alongside the existing ones, plus any language-specific recognizers. This is how a market-specific pack (a national NER model and its national identifier recognizers) is built, and it is a natural deliverable of a pilot for a new market.