AFTD Auto-Formalizing Theoretical Domains

bpe_density

deforiginal

For natural numbers k and N and sequence w : Fin N → ℕ, the real token density bpe_density k N w is the length in ℝ of the list obtained by applying k greedy BPE merges to List.ofFn w, divided by N in ℝ.

Statement

noncomputable def bpe_density (k : ℕ) (N : ℕ) (w : Fin N → ℕ) : ℝ
Related work
knowledgebase
Verified
11 Nov 2026
Axioms
none listed
Built on 1
  • bpe_lex_runThe string obtained by performing k sequential greedy lexicographical full BPE merges on an initial string over symbols 0 and 1.

Read back from the Lean

bpe_density is a definition that takes a step count k : ℕ, a length N : ℕ, and a word w : Fin N → ℕ, and produces the real number ((bpe_lex_run k (List.ofFn w)).length : ℝ) / (N : ℝ). That is, it converts the word w into a list of length N, runs k steps of lexicographical byte pair encoding (BPE) on it, and returns the length of the resulting list divided by N.

Written by a model that saw only the Lean, never the English above. If the two disagree, that is worth a challenge.

Lean source view module on GitHub

/-- The final token count divided by N as a real number, after k greedy BPE merges on w : Fin N → ℕ. -/
noncomputable def bpe_density (k : ℕ) (N : ℕ) (w : Fin N → ℕ) : ℝ := ((bpe_lex_run k (List.ofFn w)).length : ℝ) / (N : ℝ)
See it in the graphDiscuss this resultChallenge it