Persian Search Needs More Than Unicode Normalization

A support specialist searches for پیگیری, but the document store misses a letter containing پيگيري. The team has already enabled Unicode normalization. Someone proposes a more aggressive cleanup function, applied everywhere—including external case identifiers. A retrieval problem is about to become an equality-policy change.

This is an illustrative situation, not a customer incident. For teams building Persian enterprise search or a document-connected AI assistant, the useful decision is specific: which differences should this field ignore, for this operation? The answer needs an explicit matching contract, examples that must match, examples that must remain distinct, and an index migration that preserves the original evidence.

Decide what a match is allowed to mean

A title search can reasonably return the same candidates for common Persian keyboard variants. That does not make two documents identical. A name search can suggest several customer records without proving they refer to one person. An identifier issued by another system has its own contract; the search team's preferred cleanup rules do not override it.

The W3C string-matching draft explains that visually identical strings can retain different code-point sequences even after normalization. At our review date, this is a Working Draft, not a final Recommendation. Its relevant distinction is useful: visual similarity is not a complete comparison specification.

Write a small policy for each field before selecting an implementation:

These are ZharfAI design recommendations, not universal defaults. Give each policy an owner who understands the field's consequences. Our broader Persian AI localization guide covers language and presentation architecture; this guide concentrates on the narrower, testable equality decisions within retrieval.

Inspect six pairs before trusting a normalization label

Unicode Standard Annex 15 distinguishes canonical normalization from compatibility normalization. NFC uses canonical equivalence; NFKC additionally applies compatibility transformations. Neither is a general Persian spelling repair. We compared the following pairs with JavaScript's String.normalize; equality here means identical resulting character sequences, not identical people or documents.

These are locally executed teaching fixtures, not a search-engine evaluation. The runtime was Node.js 25.6.1 with Unicode 17.0 data and ICU 78.2; the reviewed standard is Unicode 18.0. Record the results in your actual environment too. A product's “Unicode support” label does not specify your matching policy.

The pairs expose two different mistakes. Switching from NFC to NFKC does not implement the desired yeh or kaf mapping. Yet it can erase another distinction, such as the circle around a digit. Compatibility normalization is not simply a stronger setting on a Persian-correctness dial. Choose it for a defined purpose, not because the acronym contains an extra letter.

Treat the half-space as a token-boundary decision

Persian text often contains the zero-width non-joiner, U+200C, commonly called a half-space. In the test word above, removing it produces میرود; replacing it with an ASCII space produces می رود. Those are different strings entering tokenization. Whether they eventually retrieve similar results depends on the rest of the processing chain.

Elastic's documented Persian analyzer example maps U+200C to U+0020 before tokenization. Its subsequent filters include digit handling, Arabic and Persian normalization, stopwords and stemming. This is an analyzer configuration, not an instruction to rewrite every Persian field in storage.

Inspect the actual field's intermediate text, tokens, token positions and phrase-query behavior. One successful single-word lookup is insufficient evidence for a multiword phrase. A boundary change can alter adjacency requirements. Test the index and query paths together, including their explicit configuration; matching analyzer names across environments do not prove matching behavior.

Do not remove every invisible character with one broad regular expression. First identify the character and its role. Direction controls, a non-joiner and an ordinary space are not interchangeable categories. A small documented transformation is easier to evaluate than a cleanup function whose name conceals several unrelated operations.

Scope language-specific rules to language-specific fields

A shared script does not imply a shared orthography. An enterprise collection can mix Persian prose, Arabic passages, regional names and product codes. Applying a Persian title policy to all of them merely because the glyphs look familiar is an unsupported extension of scope.

Doctor and colleagues' research on graphemic normalization of the Perso-Arabic script examines script variation, including visually ambiguous but canonically distinct letters, across eight languages in particular NLP tasks. It provides context for language-sensitive treatment, not evidence that one universal mapping improves every Persian company's search.

Our recommendation is to name and version the processing profile for each intended use. A profile records transformations, their order, digit policy, non-joiner treatment and tokenization settings. Keep the field's declared language separate from a content-language estimate. If automatic language detection chooses a profile, test uncertainty and unlabelled input rather than silently treating all script-family text as Persian.

The assistant should not invent a fresh equality rule on every request. It can explain a mismatch to a user, while reproducible code and versioned configuration decide how characters are transformed. That division also makes failures debuggable: the team can reproduce an input under the same profile without reconstructing a model's improvised interpretation.

Let one search key point to several originals

Construct two documents with different stable IDs. Give one the title نامه ۱۲ and the other نامه 12. If the title-search profile folds Persian digits into ASCII digits, both titles produce the same search key. That collision may be desirable: a user could reasonably expect either digit style to retrieve both documents.

The error would be making that derived key the document's unique identity. One document could replace the other, or result deduplication could collapse two genuine records. The search relationship is one key to potentially several stable document IDs. Deduplicate repeated appearances of the same document by its actual identity, not by its simplified title.

Permissions remain attached to the real document and authorization context. Key equality must not widen access. If the assistant takes an action on a result, it must pass the selected stable identifier to the tool, not the now-ambiguous transformed title. Preserve each original title in the result so the user can inspect the distinction.

This is also why broader candidate retrieval is different from record merging. Our customer-record matching guide examines the additional identity evidence. Here, we have only assembled candidates. Better retrieval does not grant permission to consolidate records or transfer authority between them.

Check database equality separately from search analysis

The PostgreSQL 18 collation documentation describes nondeterministic collations that can consider different byte sequences equal. The provider and settings determine which distinctions disappear; deterministic = false is not a Persian-normalization switch. The documentation also identifies performance costs and restrictions on some operations.

The design consequence is to test direct equality, joins, uniqueness constraints, exact lookup and full-text retrieval separately. They need not behave identically, but their differences must be intentional. Making name retrieval more forgiving must not silently redefine identifier uniqueness elsewhere in the application.

Before changing a column's comparison policy or its index, enumerate values that become equal under the proposed rule. Classify each collision group as permitted, requiring review, or blocking the change. Do not infer database behavior from glyph appearance or from the JavaScript fixture above. We have not benchmarked PostgreSQL or Elasticsearch in a deployed environment for this article.

For a consequential field, retain the approved identifier contract even when search offers a more tolerant suggestion path. A user can select a candidate and then confirm its exact identity. There is no need to force every interface to use the same permissive comparison merely for implementation convenience.

Build both must-match and must-not-match tests

An acceptance set needs positive pairs and protected negative pairs. Record the field, profile version, precise input sequences, expected outcome and reason for the expectation. The same two strings can be equivalent for title retrieval and distinct within an identifier namespace. A test without its field and operation is incomplete.

Start with the six pairs above, then add all three digit families, non-joiner versus space versus no separator, and authorized examples of actual product failures. Use fabricated values when they preserve the failure mechanism; test fixtures should not become an unnecessary repository of personal information.

For sensitive identifiers, inspect visual confusion separately. Unicode Technical Standard 39 describes confusable skeletons as a detection mechanism, not a normalization to use as general identifier text. A similarity alert must not itself merge accounts or rewrite the user's original value.

Test useful invariants as well as individual pairs. Applying the same profile twice should not change the result again. The declared profile should behave consistently across ingestion and querying. Every derived key must retain a path back to its source document. Idempotence alone is weak evidence: a function that turns every input into an empty string passes that test while destroying retrieval.

Inspect result identities, not just the assistant's fluent answer. For the two-title fixture, check that both authorized documents survive indexing, remain distinguishable and are referenced correctly by actions. Include a case where access permits only one of them. A matching improvement is not acceptable if it changes that boundary.

Migrate the index and query profile as one generation

Our proposed implementation retains a referenceable source and builds a derived search representation carrying the document ID, source revision, field and profile version. Preserve source material under the organization's access and retention policy; preservation is not a justification for keeping all data indefinitely.

When the profile changes, rebuild from the authoritative source, not from a previously simplified key. Lost distinctions cannot reliably be recovered from that key. Compare old and new generations on identical queries, while also accounting for source updates and deletions through an agreed cutover point. An index that handles historical examples but misses recent changes is not ready.

Select query processing together with the index generation. Switching an index alias while leaving an incompatible query profile active changes only half the contract. Rollback must restore the pair. Partition or invalidate cached results by their relevant generation so that old matching decisions do not masquerade as new ones.

Evidence locations need their own check. Deleting or replacing characters can shift offsets. A retrieved passage must resolve to the original text and correct location before the assistant presents a verbatim quotation. Preserve a tested position mapping where appropriate; otherwise locate and verify the passage again in the source. Do not display a transformed search string as though it were the untouched evidence.

Measure useful matches, not just more results

For queries with independently known relevant documents, measure retrieval coverage and the relevance of returned results separately. Track collision groups too: how many search keys now correspond to multiple original values or documents? An increase is not automatically a defect. Interpret it against the field policy and reviewed examples.

Break results down by transformation, input source, language and field. An attractive aggregate can hide identifier regressions behind title-search gains. Inspect latency, index size, ranking quality and quotation fidelity alongside coverage. Without independent judgments about relevant documents, do not advertise a measured real-world recall rate.

Rerun a small set of known documents and protected pairs after changes to analyzers, Unicode libraries, extraction, input channels or fonts. A font change may affect only appearance; record that distinction so a display issue is not mistaken for a comparison issue. Diagnostic reports need code points and processing versions, not indiscriminate copies of confidential content.

Finally, a well-tested matching contract still does not prove the repository itself is complete. Our guide to empty search results covers scope and completion evidence. The narrower success here is knowing which distinctions the system deliberately ignored to find a document—and which it preserved to identify that document or quote its evidence correctly.

Source notes — reviewed September 26, 2026

The executed character comparisons, constructed document pair and migration recommendations are ZharfAI analysis and teaching examples. They are not customer deployment results or service-performance benchmarks. The sources support particular mechanisms and distinctions, not a guarantee of product search quality.

• Unicode UAX #15: revision 58, Unicode 18.0, August 12, 2026; normalization forms.
• W3C, Character Model: String Matching: First Public Working Draft, July 16, 2026; work in progress, not a final Recommendation.
• Elastic, language analyzers: rolling documentation without a fixed page-publication date; Persian analyzer example accessed on the review date.
• PostgreSQL 18, collation support: versioned documentation without a specific publication date for this page; database comparison semantics.
• Unicode UTS #39: revision 34, August 27, 2026; confusable detection and limits on skeleton use.
• Doctor and colleagues, Graphemic Normalization of the Perso-Arabic Script: first submitted October 21, 2022; version three January 29, 2024; language and orthographic variation.