python-offside-rule-indent-dedent

IN premise — entries/2026/06/21/wiki-Tokenization_lexical_analysis-chunk-1.md

Created 2026-06-21T09:50:10+00:00

Python's off-side rule is implemented at the lexer level: the lexer maintains a stack of indent levels and emits INDENT/DEDENT tokens, making the lexical grammar context-sensitive rather than context-free.

Summary

Python handles its indentation-based block structure by having the tokenizer itself track the current nesting depth and inject special marker tokens into the stream, so the tokenizer is stateful and depends on what came before rather than simply scanning characters independently. This matters for any tool that tries to re-tokenize a code fragment, highlight syntax incrementally, or parse incomplete snippets, because the same indentation sequence can yield different results depending on prior context.