SURREALISM, n. Pure prescient prefiguration, by which it is intended to express the design of indie video games.

There is so much low hanging fruit in thinking carefully about what’s happening in a transformer / LLM.

Did it really take two years for someone to point out that RoPE will (obviously?) not decay activation with relative distance for random key value pairs?

From: arxiv.org/pdf/2410.06205