memit-generation-entropy-formula

IN premise — summaries/2026/08/24/meng-2022-medit-sR-references.md

Created 2026-08-25T02:58:14+00:00

Generation Entropy (GE) is computed as −(⅔ Σ f₂(k) log₂ f₂(k) + ⁴⁄₃ Σ f₃(k) log₂ f₃(k)), using weighted 2-gram and 3-gram frequency distributions to detect repetitive generation.

Summary

This defines a specific metric for measuring how repetitive a text generation is by looking at how evenly 2-word and 3-word patterns are distributed across the output, with longer patterns weighted more heavily. In practice, it gives the system a concrete, calculable way to flag when a generator is falling into predictable loops versus producing genuinely varied text.