Compiler From Scratch: Phase 1 - Tokenizer Generator 005: Anchoring, NFA to DFA, and diagrams

Streamed on 2024-08-16 (https://www.twitch.tv/thediscouragerofhesitancy)

Zero Dependencies Programming!

I worked through adding Anchoring support to our Patter-to-NFA converter, then upgraded our NFA output options to include a Graphviz file. The graph based images are very nice for visualization and understanding the NFA.

Then I grabbed some old code from my sandbox and brought it in to convert the NFA into a DFA. There was some modification necessary, but it wasn't too bad. We added the same Graphviz output capability to the DFA. There was some refactoring of both NFA and DFA to bring them more into alignment with each other, and additional changes were needed to support StartOfLine and EndOfLine in the DFA, but overall it wasn't too bad.

There are still some output issues I have with the Graphviz output: the patterns as stored aren't escaped properly for Graphviz, so escaped characters don't render out right, but that is something we'll work on later.

Lastly, we started trying to build the NFAs from our TokenizationDefinition, but there is a very strange bug that I ran into that I didn't have fixed by the end of the stream. Next time.

I did have to trim off the last couple of minutes to fit Rumble's 15GB limit, but you didn't miss anything.