jaccard-tokenizer-splits-on-hyphens-and-colons
IN premise
`_tokenize_id` splits belief IDs on hyphens and colons into token sets for Jaccard similarity comparison, so `foo-bar-baz` and `foo-bar-qux` share 2/4 tokens (0.5 similarity)
Summary
The similarity matching system breaks belief IDs apart at hyphens and colons to compare them token by token. This means beliefs with overlapping name segments will score as partially similar even if they refer to completely different concepts, which affects how search results are ranked and which beliefs surface as related.
Details
| Source | entries/2026/05/11/tests-test_derive.md |