is_token_segmentation
def
A list of tokens forms a valid segmentation of a word y over vocabulary V and single-character alphabet if their concatenation is y and each token belongs to V or has length 1.
Statement
def is_token_segmentation {α : Type*} (V : Set (List α)) (y : List α) (tokens : List (List α)) : Prop
- Defined in
- arXiv:2611.15074, Section 2.1, lines 213-216
- Verified
- 13 Nov 2026
- Axioms
- none listed
- Used by 2
- suffix_segmentation_bound_by_chunk_vocabularyFix q in {2, ..., M}. For every alphabet and every nonempty word w of length at most M, there exists a vocabulary V_w of substrings of w…
- is_token_segmentation_singletonEvery word y admits a valid token segmentation into singletons over any vocabulary V.
Read back from the Lean
Given a type α, a set of lists V ⊆ List α, a list y : List α, and a list of lists tokens : List (List α), is_token_segmentation V y tokens asserts that:
1. Flattening (concatenating) the lists in tokens yields y, and
2. Every list t in tokens either belongs to V or has length equal to 1.
Written by a model that saw only the Lean, never the English above. If the two disagree, that is worth a challenge.
Lean source view module on GitHub
/-- Predicate that a list of tokens is a valid segmentation over vocabulary V and singletons. -/ def is_token_segmentation {α : Type*} (V : Set (List α)) (y : List α) (tokens : List (List α)) : Prop := tokens.flatten = y ∧ ∀ t ∈ tokens, t ∈ V ∨ t.length = 1