Two scanners can examine the same source file and disagree about the identity of a finding. One saw CRLF line endings; another saw LF. One received a composed Unicode character; another received an equivalent sequence of code points. Hashing the raw inputs gives different IDs even when the human believes the finding did not change.

The real design question comes before the hash.

Decide the equivalence relation §

Write x ~ y when two representations should count as the same underlying subject. A canonicalization C should satisfy: if x ~ y, then C(x) = C(y). But we also need to avoid collapsing meaningful differences.

Absent data is not always an empty string. A missing customer address and a blank address entered by the customer can mean different things. Reordering fields in a map may be harmless; reordering an ordered workflow is not.

Unicode Standard Annex #15[1] defines forms of textual normalization. MDN's normalization examples[2] demonstrate how visually equivalent strings can have different underlying bytes.

Version the contract §

An identity function might hash the canonical representation alongside a schema version. That makes changes in interpretation visible. It does not remove the need to map old IDs to new ones when records are migrated.

Consider a security finding that moves five lines down after a comment is inserted. If its identity is tied to the line number, the issue can disappear and reappear as a new one without being fixed. If identity ignores too much context, two genuinely different findings can collapse.

Our earlier SARIF and Matrix discussions pointed toward testing these cases explicitly across Unicode, paths, null values, ordering and schema versions.

A digest is a useful implementation mechanism. The institutional decision is which changes preserve identity and which create a new subject. That decision has to be written down before the checksum can help.

Sources and further reading

References point to original material linked in the essay, not independently invented citations.

  1. Unicode Standard Annex #15unicode.org↩
  2. MDN's normalization examplesdeveloper.mozilla.org↩
Across the Journal

Related reading

More articles that explore this subject from another angle.

Browse more articles in the Reading Guide ↗

About this article. Mathematical research notes from archived experiments and external literature. The linked external material is cited for context, not as an endorsement. View editorial source ↗