Lightnews — Scholar-powered news

Kyle Lo

@kylelo.bsky.social

6.5K followers 590 following 500 posts

language model pretraining research @ai2.bsky.social, Co-lead of Data for OLMo w/ @soldaini.net, statistics @uw, open science, tabletop, seattle, he/him,🧋 kyleclo.com

Posts Media Videos Starter Packs

Pinned

Kyle Lo @kylelo.bsky.social · Mar 13

we released olmo 32b today! ☺️

🐟our largest & best fully open model to-date
🐠right up there w similar size weights-only models from big companies on popular benchmarks
🐡but we used way less compute & all our data, ckpts, code, recipe are free & open

made a nice plot of our post-trained results!✌️

1 5 38

Kyle Lo @kylelo.bsky.social · 1d

nice read thx for sharing! I think the piece could use a follow up / complement discussing misaligned incentives that push scientists to compete rather than collaborate (notably, the section on data fragmentation)

1 6

Kyle Lo @kylelo.bsky.social · 1d

bye #colm2025 big fan of the montreal bagels 🥯 hot take I like them better than

Kyle Lo @kylelo.bsky.social · 3d

lol so much love for prepost-postpre training

Kyle Lo @kylelo.bsky.social · 3d

any other fans of pre-pretraining?

1 2

Kyle Lo @kylelo.bsky.social · 4d

come say hi at posters this morning for OLMo 2 and fluid benchmarking posters 👋 and dont miss @valentinhofmann.bsky.social's talk in morning #colm2025 @ai2.bsky.social vry proud of my gifs

2 6

Kyle Lo @kylelo.bsky.social · 4d

@josephc.bsky.social @mariaa.bsky.social and I are at poster #21

findings from large scale survey of 800 researchers on how they use LMs in their research #colm2025

2 13

Kyle Lo @kylelo.bsky.social · 6d

flyin to #colm2025 along w bunch of the @ai2.bsky.social team

come chat w me about pretraining horror stories, data & evals, what we're cookin for next olmo, etc

made a 🔥 poster for thursday sess, come say hi

1 11

Kyle Lo @kylelo.bsky.social · 7d

same flight lol I just got to airport way too early

1 1

Kyle Lo @kylelo.bsky.social · 7d

5 am airport for the only direct flight from seattle to montreal #colm2025

1 12

Kyle Lo @kylelo.bsky.social · 10d

ya totes, see u there!

Kyle Lo @kylelo.bsky.social · 10d

synthetic data mimics real data's rough shape, modality, types, schema, etc. but with fake values. models these days are quite proficient at operating over data of this type & generating reasonable code; the main contrib here is system design to replace the repetitive data exploratory workflow

Kyle Lo @kylelo.bsky.social · 10d

hehe i didnt do anythin!

core is data voyager (arxiv.org/abs/2402.13610) but local LM instead of GPT

it generates code (map-reduce-filter) that transforms data (csvs), a federated platform executes & returns some output back to system. system repeatedly interprets + generates more code

Data-driven Discovery with Large Generative Models

With the accumulation of data at an unprecedented rate, its potential to fuel scientific discovery is growing exponentially. This position paper urges the Machine Learning (ML) community to exploit th...

arxiv.org

2 1

Kyle Lo @kylelo.bsky.social · 10d

not my project but I rlly like it

working w cancer research center to analyze clinical data, but private data cant leave the center.

so the team developed a tool that generates code for remote execution by the cancer center, developed on synthetic data, and now tested for realsies 🤩

1 4

Kyle Lo @kylelo.bsky.social · 10d

nice post! reminds me of this ICLR2023 paper also had a short discussion about other architectures not seeing as big diff between adam/sgd as seen w modern transformers arxiv.org/abs/2304.139...

Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Be

The success of the Adam optimizer on a wide array of architectures has made it the default in settings where stochastic gradient descent (SGD) performs poorly. However, our theoretical understanding o...

arxiv.org

Kyle Lo @kylelo.bsky.social · 24d

had to explain to first time submitter why AC recommended accept ended up as reject 😮‍💨 been publishing long enough that i get why such things happen but can be rough

Kyle Lo @kylelo.bsky.social · 24d

oh dang, missed this paper, this is rlly nice thx!

Kyle Lo @kylelo.bsky.social · 24d

high dim + discrete space (tokens). back in soft prompts days, gradients made high dim easier to handle cuz continuous space. high dim search wout gradients is tough

1 4

Kyle Lo @kylelo.bsky.social · 25d

LM benchmark design requires 3 decisions, how to:
🐟 select test cases
🐠 score LM on each test
🦈 aggregate scores to estimate perf

fluid benchmarking is simple:
🍣 find max informative test cases
🍥 estimate 'ability', not simple avg perf

why care? turn ur grey noisy benchmarks to red ones!

2 5

Kyle Lo @kylelo.bsky.social · Sep 12

scathing takedown of recent K2 Think model

"evaluates on data it was trained on, relies on an external model and additional samples for its claimed performance gains, and artificially reduces the scores of compared models"

www.sri.inf.ethz.ch/blog/k2think

Debunking the Claims of K2-Think

K2-Think is a recently released LLM that claims performance on par with GPT-OSS 120B and DeepSeek v3.1, despite having fewer parameters. As we discuss below, the reported gains are overstated, relying...

www.sri.inf.ethz.ch

1 6