Compiler From Scratch: Phase 1 - Tokenizer Generator 018: Fixing errors at EndOfText, add more tests

Streamed on 2024-11-15 (https://www.twitch.tv/thediscouragerofhesitancy)

Zero Dependencies Programming!

The testing from last time revealed some bugs when the text buffer ends in certain ways. Today I debugged that issue and got all 120 tests to run. It related to how we were looking for the end of the text buffer and also how the LAZY tokenizer build differed from the BATCH tokenizer build.

Once that was done I tackled a couple of loose ends. The first loose end was to parse the TokenizerRule actions out of the string and into some Boolean flags. Eventually we will add more actions the tokenizer can take, and this makes that future work a little easier.

Lastly, there were a few places in the generated code that were logging errors. This is not good. The errors (and any warnings) should be saved in the Context. So I made that change. The only tricky bit here is the TxtBuf must be declared before the Context and can't reference it directly. Rather, the Context passes in a callback for when an error needs to be reported to it.

Then we added one more test case to be run each test execution, this time looking for "unexpected character" errors. It was a simple test to implement at the end of the day.