AFTD Auto-Formalizing Theoretical Domains

is_token_segmentation

def

A list of tokens forms a valid segmentation of a word y over vocabulary V and single-character alphabet if their concatenation is y and each token belongs to V or has length 1.

Statement

def is_token_segmentation {α : Type*} (V : Set (List α)) (y : List α) (tokens : List (List α)) : Prop
Defined in
arXiv:2611.15074, Section 2.1, lines 213-216
Verified
13 Nov 2026
Axioms
none listed
Used by 2

Read back from the Lean

Given a type α, a set of lists V ⊆ List α, a list y : List α, and a list of lists tokens : List (List α), is_token_segmentation V y tokens asserts that: 1. Flattening (concatenating) the lists in tokens yields y, and 2. Every list t in tokens either belongs to V or has length equal to 1.

Written by a model that saw only the Lean, never the English above. If the two disagree, that is worth a challenge.

Lean source view module on GitHub

/-- Predicate that a list of tokens is a valid segmentation over vocabulary V and singletons. -/
def is_token_segmentation {α : Type*} (V : Set (List α)) (y : List α) (tokens : List (List α)) : Prop := tokens.flatten = y ∧ ∀ t ∈ tokens, t ∈ V ∨ t.length = 1
See it in the graphDiscuss this resultChallenge it