Lightnews — Scholar-powered news

Marzena Karpinska

@markar.bsky.social

3.8K followers 930 following 65 posts

#nlp researcher interested in evaluation including: multilingual models, long-form input/output, processing/generation of creative texts previous: postdoc @ umass_nlp phd from utokyo https://marzenakrp.github.io/

marzenakrp.github.io

Posts Media Videos Starter Packs

Marzena Karpinska @markar.bsky.social · 1d

Probably for the best, they had serious overflows because of this...

Marzena Karpinska @markar.bsky.social · 1d

Wasn't this what ACL did this year?

Marzena Karpinska @markar.bsky.social · 7d

Come to talk with us today about the evaluation of long form multilingual generation at the second poster session #COLM2025

📍4:30–6:30 PM / Room 710 – Poster #8

2 6

Marzena Karpinska @markar.bsky.social · 8d

Off to #COLM fake Fuji looks really good today.
本物は下からしか見たことがないが、今日は少なくとも偽物が上から見えて嬉しい。

Marzena Karpinska @markar.bsky.social · 8d

I feel like it was worth waking up early

Marzena Karpinska @markar.bsky.social · 8d

Wait how come, I'm flying direct at 7am..

Reposted by Marzena Karpinska

Yoav Goldberg @yoavgo.bsky.social · Aug 27

When reading AI reasoning text (aka CoT), we (humans) form a narrative about the underlying computation process, which we take as a transparent explanation of model behavior. But what if our narratives are wrong? We measure that and find it usually is.

Now on arXiv: arxiv.org/abs/2508.16599

Humans Perceive Wrong Narratives from AI Reasoning Texts

A new generation of AI models generates step-by-step reasoning text before producing an answer. This text appears to offer a human-readable window into their computation process, and is increasingly r...

arxiv.org

4 22 85

Reposted by Marzena Karpinska

Tom Kocmi @kocmitom.bsky.social · Aug 23

📊 Preliminary ranking of WMT 2025 General Machine Translation benchmark is here!

But don't draw conclusions just yet - automatic metrics are biased for techniques like metric as a reward model or MBR. The official human ranking will be part of General MT findings at WMT.

arxiv.org/abs/2508.14909

Preliminary Ranking of WMT25 General Machine Translation Systems

We present the preliminary ranking of the WMT25 General Machine Translation Shared Task, in which MT systems have been evaluated using automatic metrics. As this ranking is based on automatic evaluati...

arxiv.org

1 4 9

Marzena Karpinska @markar.bsky.social · Aug 20

Happy to see this work accepted to #EMNLP2025! 🎉🎉🎉

1 12

Reposted by Marzena Karpinska

EMNLP @emnlpmeeting.bsky.social · Aug 20

✨We are thrilled to announce that over 3200 papers have been accepted to #EMNLP2025 ✨

This includes over 1800 main conference papers and over 1400 papers in findings!

Congratulations to all authors!! 🎉🎉🎉

2 29

Reposted by Marzena Karpinska

Jessy Li @jessyjli.bsky.social · Aug 12

The Echoes in AI paper showed quite the opposite with also a story continuation setup.
Additionally, we present evidence that both *syntactic* and *discourse* diversity measures show strong homogenization that lexical and cosine used in this paper do not capture.

2 13 60

Marzena Karpinska @markar.bsky.social · Aug 16

Definitely!

Marzena Karpinska @markar.bsky.social · Aug 16

At the same time I wish that whoever sparked this interest in data distribution would also help them with the design...

1 1

Marzena Karpinska @markar.bsky.social · Aug 16

Absolutely! Looking forward to seeing QUDsim at COLM!

2 1

Marzena Karpinska @markar.bsky.social · Aug 15

The issue is always what, which humans in what circumstances

1 2

Marzena Karpinska @markar.bsky.social · Aug 15

I think there are quite a few undergraduate students on this preprint and maybe there was a need for a bit more mentoring. The comparison to writingprompts is just one of the issues (amateur writers in very different conditions than normal writing + very short outputs).

1 3

Marzena Karpinska @markar.bsky.social · Aug 8

Check out the full leaderboard here: novelchallenge.github.io

We'll be updating the dataset with new books and claims within the next few months!

NoCha leaderboard

novelchallenge.github.io

Marzena Karpinska @markar.bsky.social · Aug 8

GPT-5 lands first place on NoCha, our long-context book understanding benchmark.

That said, this is a tiny improvement (~1%) over o1-preview, which was released almost one year ago. Have long-context models hit a wall?

Accuracy of human readers is >97%... Long way to go!

Screenshot of benchmark with gpt-5 on top with 68.46% accuracy.

1 6 18

Reposted by Marzena Karpinska

Ankita @ankitagupta.bsky.social · Jul 28

🗓️29 July, 4 PM: Automated main concept generation for narrative discourse assessment in aphasia. w/
@marisahudspeth.bsky.social, Polly Stokes, Jacquie Kurland, and @brenocon.bsky.social

📍Hall 4/5.

Come by to chat about argumentation, narrative texts, policy & law, and beyond! #ACL2025NLP

3 7

Reposted by Marzena Karpinska

Ankita @ankitagupta.bsky.social · Jul 28

Excited to present two papers at #ACL2025!

🗓️30 July, 11 AM: 𝛿-Stance: A Large-Scale Real World Dataset of Stances in Legal Argumentation. w/ Douglas Rice and @brenocon.bsky.social

📍At Hall 4/5. 🧵👇

1 3 7

Reposted by Marzena Karpinska

Abhilasha Ravichander @lasha.bsky.social · Jul 22

📣 Life update: Thrilled to announce that I’ll be starting as faculty at the Max Planck Institute for Software Systems this Fall!

I’ll be recruiting PhD students in the upcoming cycle, as well as research interns throughout the year: lasharavichander.github.io/contact.html

13 12 89

Marzena Karpinska @markar.bsky.social · Jul 23

Congratulations 👏🎉

1 1

Reposted by Marzena Karpinska

EMNLP @emnlpmeeting.bsky.social · Jul 17

For EMNLP 2025’s special theme of "Advancing our Reach: Interdisciplinary Recontextualization of NLP", we are organizing a panel of experts, and would like input from the community at large as we prepare. Please take a moment to fill in this survey: forms.office.com/r/pWFFA0Gss1

5 8

Reposted by Marzena Karpinska

Melanie Mitchell @melaniemitchell.bsky.social · Jul 12

A new definition for AGI just dropped, and it is a bad one.

Dr Abeba Birhane @abeba.bsky.social · Jul 12

lord grant me the courage to write with the confidence a mediocre white man

screenshot reads: Many (not all) insiders now say AGI — artificial general intelligence — stands a good chance of happening in the next few years. AGI is a generative AI model that could, on intellectually oriented tests, outperform human experts on 90% of questions. That doesn’t mean AI will be able to dribble a basketball, make GDP grow by 40% a year or, for that matter, destroy us. Still, AGI would be an impressive accomplishment — and over time, however slowly, it will change our world.

8 27 170

Marzena Karpinska @markar.bsky.social · Jul 8

Now accepted to #COLM2025 @colmweb.org
🇨🇦🎉

Yekyung Kim @yekyung.bsky.social · Mar 5

Is the needle-in-a-haystack test still meaningful given the giant green heatmaps in modern LLM papers?

We create ONERULER 💍, a multilingual long-context benchmark that allows for nonexistent needles. Turns out NIAH isn't so easy after all!

Our analysis across 26 languages 🧵👇