Compiler From Scratch: Phase 1 - Tokenizer Generator 008: Code generating Context and Token

Streamed on 2024-09-06 (https://www.twitch.tv/thediscouragerofhesitancy)

Zero Dependencies Programming!

Today we generated more code to support the tokenizer we will be generating. We generated a class to handle Context. This is a simple object that keeps all the shared state and intermediate results which are in use throughout the different stages. For starters it contains our text buffer and a list of error and warning messages. This class will grow as we need more features.

We also generated the Token class that the Tokenizer will generate. This class contains all the information about where to find the token in the text buffer and a token identifier.

Then we got a start on generating the Tokenizer itself. We have stubbed in some of the simple parts, but there are two main processing methods we could go through: lazy or batch evaluation. Both have different performance implications, so we will generate both inside of ifdef checks so we can test them both.

In all this coding, we are writing out text, and large sections are just text with no modification which we could get from a separate file and not clutter our main code generation. The idea of having code snippets in an asset folder, then an asset pipeline script which embeds the code into our code would mean we could edit our snippets easier and also not count as much of that code towards our line count. We might work on this next week ... unless I get ambitious and excited about it and do it on my own before the next stream. That happens sometimes. Especially when I don't get to work on my compiler more than once a week and start to get the shakes.