lexing-two-stages-scanning-evaluation

Summary

Breaking raw text into tokens is not one monolithic step; it splits into first figuring out where boundaries fall and then assigning each chunk a type and concrete value. This matters because the two concerns can be debugged, optimized, or swapped independently, so a mistake in recognizing where one symbol ends does not get tangled up with a mistake in interpreting what that symbol means.