/Programming

Building a programming language sounds like something reserved for people with a PhD in compilers, but you can actually have one working in an afternoon, with Python and a bit of patience. It won't compete with Rust and it won't replace anything you use at work — it's just a serious kind of toy project, the kind that teaches you more about how computers actually interpret code than any online course. I'll walk through the whole path, with code you can run right now, for a small but real language: variables, math, prints, and a simple loop. Before writing a single line, it's worth deciding what the language will do and what it will deliberately skip. It's tempting to start thinking about functions, classes, types, everything — and that's exactly how these projects die before ever printing "hello world." We'll call it **Crumb**, and it's going to do exactly four things: store values in variables, do simple math, print stuff to the screen, and repeat a block of instructions a fixed number of times. A Crumb program looks like this: ``` var x = 5 var y = 3 print x + y loop 3 print x x = x + 1 end ``` That's already enough to play with, and the code to run it fits comfortably in a single Python file. Any language, no matter how small, goes through three stages to turn text into a result. First the text is broken into meaningful pieces — the so-called tokens. Then those tokens are organized into a structure that represents the program's logic, usually a tree. Finally, that tree is walked and each instruction actually gets executed. These are called, respectively, the lexer (or tokenizer), the parser, and the interpreter. We'll build all three, one after the other. The lexer takes a line of text and returns a list of tokens. For Crumb, a token can be a keyword (`var`, `print`, `loop`, `end`), a variable name, a number, an operator (`+`, `-`, `=`), or end of line. Nothing fancy is needed — splitting by spaces already handles most cases, as long as you're careful with operators glued to numbers. ```python def tokenize(line): line = line.replace("+", " + ").replace("-", " - ") return line.split() ``` Yes, it's naive. A real lexer would use regular expressions and handle strings, comments, decimal numbers, and so on. But for Crumb this is enough, and there's something satisfying about seeing how far you can get with so little code. Next comes the parser, which is where things get more interesting. The parser reads the list of tokens for each line and decides what kind of statement it is, turning it into a data structure the interpreter can understand. I'm not going to build a full abstract syntax tree here (that's what GCC or CPython would do under the hood) — instead I'll use something simpler, a Python dictionary that describes the instruction. It's a deliberate simplification, and it works great for toy languages. ```python def parse_line(tokens): if not tokens: return None if tokens[0] == "var": # var x = 5 name = tokens[1] value = parse_expression(tokens[3:]) return {"type": "var", "name": name, "value": value} if tokens[0] == "print": value = parse_expression(tokens[1:]) return {"type": "print", "value": value} if tokens[0] == "loop": times = int(tokens[1]) return {"type": "loop_start", "times": times} if tokens[0] == "end": return {"type": "end"} if len(tokens) >= 2 and tokens[1] == "=": # x = x + 1 name = tokens[0] value = parse_expression(tokens[2:]) return {"type": "assignment", "name": name, "value": value} raise ValueError(f"couldn't parse this line: {tokens}") def parse_expression(tokens): # supports "5", "x", "x + y", "x - 3", nothing more complex than that if len(tokens) == 1: return {"type": "term", "value": tokens[0]} return { "type": "operation", "left": {"type": "term", "value": tokens[0]}, "operator": tokens[1], "right": {"type": "term", "value": tokens[2]}, } ``` Notice that `parse_expression` only knows how to handle a single operator at a time — no `x + y - z` on one line. That's a real limitation, and it's the kind of limitation anyone attempting this will run straight into. The proper fix, if you ever want to take it further, is a recursive-descent parser with operator precedence. For now, let's leave it as is and move on to the part that actually brings the language to life. The interpreter walks through the list of already-parsed instructions and executes each one, keeping a dictionary of variables (what real languages call an "environment" or "scope"). The trickiest part is the `loop`, since it needs to know where its matching `end` is in order to repeat the right block of instructions. ```python def eval_term(term, variables): value = term["value"] if value.lstrip("-").isdigit(): return int(value) return variables.get(value, 0) def eval_expression(expression, variables): if expression["type"] == "term": return eval_term(expression, variables) left = eval_term(expression["left"], variables) right = eval_term(expression["right"], variables) if expression["operator"] == "+": return left + right if expression["operator"] == "-": return left - right raise ValueError("unknown operator") def run(instructions): variables = {} i = 0 while i < len(instructions): instruction = instructions[i] if instruction["type"] == "var" or instruction["type"] == "assignment": variables[instruction["name"]] = eval_expression(instruction["value"], variables) elif instruction["type"] == "print": print(eval_expression(instruction["value"], variables)) elif instruction["type"] == "loop_start": times = instruction["times"] # find the matching "end" end_idx = i + 1 depth = 1 while depth > 0: if instructions[end_idx]["type"] == "loop_start": depth += 1 elif instructions[end_idx]["type"] == "end": depth -= 1 if depth > 0: end_idx += 1 block = instructions[i + 1:end_idx] for _ in range(times): run(block) i = end_idx i += 1 ``` All that's left is to put it together: read the file line by line, tokenize, parse, and hand the result to `run`. ```python def load_and_run(path): with open(path, encoding="utf-8") as f: lines = f.readlines() instructions = [] for line in lines: line = line.strip() if not line: continue tokens = tokenize(line) instruction = parse_line(tokens) if instruction: instructions.append(instruction) run(instructions) if __name__ == "__main__": import sys load_and_run(sys.argv[1]) ``` Save all of this into a file called `crumb.py`, save the example program above into a file called `example.crumb`, and run `python crumb.py example.crumb`. You should see `8` show up (the result of `x + y`), followed by `5`, `6`, and `7` — the three passes of the loop, with `x` growing on each iteration. If that ran, you now have a working programming language, written by you, from scratch. From here, the fun part is taking Crumb apart and putting it back together bigger. An `if` is the obvious next step, and it uses exactly the same logic as `loop`: find the matching block, decide whether to run it or skip over it. Multiplication and division require taking `parse_expression` seriously, with a grammar that respects precedence — it's worth looking up "recursive-descent parsing" once you get there, since it's the pattern practically every real language uses under the hood. Strings as a data type force the lexer to recognize quotes instead of blindly splitting on spaces. And if you really want to go further, you could swap Python's `print()` for actual bytecode generation, or even compilation to C — but that's a different afternoon, or a different life. What matters is that this little exercise shows what's happening behind any language you use day to day. When Python throws you a syntax error, that's its parser rejecting something that doesn't match the grammar. When a variable "disappears" outside a block, that's the runtime environment doing scoping. Once you've built your own version, rough as it may be, none of that is magic anymore.