あけましておめでとうございます🎍

Hmm

Tried to get a zipcar parked in ice a couple inches thick... my Californian self gave up

writing is hard. writing on a deadline is hard.

advent of code has brought to life two wolves in me i forgot existed (competitive at all costs and eternal imposter)

All the things you need to know to pretrain an LLM at home*!

Gave a workshop at Uni Bern: starts with scaling laws and goes to web scale data processing and finishes training with 4D parallelism and ZeRO.

*assuming your home includes an H100 cluster

read a comment under an ai image generator app ad saying: "these things impress you at first, but you get frustrated when you want to control something about the result and discover that it's more difficult than you thought" and i feel the same way with most ai products nowadays

The Extinction Illusion: The left side and the right side have the same number of black dots

journals.sagepub.com/doi/abs/10.1...

it's another beautiful day in cambridge, ma

#EMNLP has a nice set of tokenization/subword modeling papers this year.

It's a good mix of tokenization algorithms, tokenization evaluation, tokenization-free methods, and subword embedding probing. Lmk if I missed some!

Here is a list with links + presentation time (in chronological order).