A public library indexes a collection of names, place descriptions, scanned documents and recorded interviews. One visitor searches a phrase and finds the intended result; another types a version with different accents or word boundaries and gets silence. A search engine that treats every document as an English sentence and every query as a simple substring will reliably disappoint somebody.
The complication begins before machine learning or ranking. The system must decide what the units of language are.
The boundary between words is not universal §
Unicode's text segmentation work[1] describes rules for grapheme, word and sentence boundaries. A search analyzer must also account for language, normalization, punctuation and the domain's relevant vocabulary. Different writing systems do not all use spaces between words in the same way.
An exact-name lookup, a full-text discovery query and a semantic similarity search need different matching contracts. Collapsing them into one fuzzy search box can help someone find a rough topic while making exact evidence hard to retrieve.
A worked source-search failure §
Suppose an article quotes a historical committee title, but the archive indexes only a current translated title. A keyword search returns nothing. A broad embedding search might find thematically similar documents, but could also return records that never contained the quoted phrase.
The right system distinguishes exact phrase evidence, normalized matches, translations and semantic neighbors. A highlighted result should reflect a real passage from the indexed source, not a text fragment invented to explain a score.
Ranking is an editorial decision made by software §
Even a technically correct index has choices about freshness, field boosts, language preferences, popularity, source authority and user access. A research system that heavily boosts repeated copies may hide the original observer, while a commerce system that boosts availability without verifying stock may mislead a buyer.
A useful search receipt can identify its index generation, the corpus actually searched, applied filters, query interpretation and source passage. This is especially important when a result will become an input to an AI-generated answer.
Security is inside retrieval, not after it §
A private document should not become searchable merely because its title resembles a public record. Permission checks must bound candidate generation and later rendering; filtering only the final visible card can leak counts, snippets or other sensitive clues.
Aurekai's essay on dependent evidence adds the epistemic layer: five results copied from one notice are not five independent observations. Search can improve discovery without answering how much evidence a claim has.
Search is therefore a living representation of what the institution knows, can show and can distinguish—not merely a function that accepts a query string.
Sources and further reading
References point to original material linked in the essay, not independently invented citations.
- Unicode's text segmentation workunicode.org↩
Related reading
More articles that explore this subject from another angle.
Textual and semantic retrieval make different promises about source identity.
Follow this connection ↗Another lens: Data consistencyPage three did not mean the same thing twiceSoftware and operations connects search to data consistency without making their records interchangeable.
Follow this connection ↗Another essay leads hereThe spreadsheet import was also a programCSV serialization, spreadsheets, field types and downstream execution reveal why a simple export is a trust boundary.
Read the earlier argument ↗