Research
Notes on the things I'm reading, reimplementing, or trying to work out.
This is where I keep longer-form notes on topics I haven’t written a proper post about yet - half-formed ideas, reading notes, and small experiments.
What’s on the go
Regular expressions and finite automata
Working through the equivalence between regular expressions, finite automata, and regular grammar, and building a small regex engine to make the theory concrete. The write-up exists but is still a draft, so it’s not linked here yet.
Information theory and combinatorics
Permutation counting and the surprising amount of structure hiding in “how many ways”. Also written up, and also still a draft.
Attention and sequence models
Working through the transformer architecture from the paper outward, including the implementations that came after it.
Self-replication and von Neumann probes
Where the self-replicating machine idea came from, what it demands of a description of computation, and how it keeps turning up in robotics, cellular automata and the philosophy of mind.
Higher-order derivatives in fine-tuning
Using second and higher-order derivatives in gradient descent and backpropagation rather than only the first derivative, and what the extra curvature information buys you when fine-tuning against sparse data. Work in progress.
Have something you’d like me to look into? Open an issue.
Nothing here yet.