Sneha Kudugunta @NeurIPS2024
@snehaark.bsky.social
tpu go brr @deep-mind.bsky.social @uwcse.bsky.social | varying proportions of AI and mediocre jokes (not mutually exclusive) | she/her/hers
Reposted by Sneha Kudugunta @NeurIPS2024
📢Thrilled to introduce ATLAS 🗺️: the largest multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer:
🌍 Is scaling diff by lang?
🧙♂️ Can we model the curse of multilinguality?
⚖️ Pretrain vs finetune from checkpoint?
🔀 X-lingual transfer scores across langs?
1/🧵
🌍 Is scaling diff by lang?
🧙♂️ Can we model the curse of multilinguality?
⚖️ Pretrain vs finetune from checkpoint?
🔀 X-lingual transfer scores across langs?
1/🧵
October 28, 2025 at 2:03 PM
📢Thrilled to introduce ATLAS 🗺️: the largest multilingual scaling study to-date—we ran 774 exps (10M-8B params, 400+ languages) to answer:
🌍 Is scaling diff by lang?
🧙♂️ Can we model the curse of multilinguality?
⚖️ Pretrain vs finetune from checkpoint?
🔀 X-lingual transfer scores across langs?
1/🧵
🌍 Is scaling diff by lang?
🧙♂️ Can we model the curse of multilinguality?
⚖️ Pretrain vs finetune from checkpoint?
🔀 X-lingual transfer scores across langs?
1/🧵
I will be at poster #2507 w/ my co-authors in East Exhibit Hall A-C at #NeurIPS2024 chatting about MatFormer and elastic models today at 4.30pm!
Come by, or reach out if you want to chat about pretraining, scaling laws or conditional computation!
arxiv.org/abs/2310.07707
Come by, or reach out if you want to chat about pretraining, scaling laws or conditional computation!
arxiv.org/abs/2310.07707
December 11, 2024 at 9:42 PM
I will be at poster #2507 w/ my co-authors in East Exhibit Hall A-C at #NeurIPS2024 chatting about MatFormer and elastic models today at 4.30pm!
Come by, or reach out if you want to chat about pretraining, scaling laws or conditional computation!
arxiv.org/abs/2310.07707
Come by, or reach out if you want to chat about pretraining, scaling laws or conditional computation!
arxiv.org/abs/2310.07707