Hye Sun Yun
@hyesunyun.bsky.social
PhD candidate in CS at Northeastern University | NLP + HCI for health | she/her 🏃♀️🧅🌈
I am at CHIL this week to present my poster (Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?) on Thursday June 26.
Looking forward to connecting and sharing our work on spin with the CHIL community!
Looking forward to connecting and sharing our work on spin with the CHIL community!
June 26, 2025 at 12:52 AM
I am at CHIL this week to present my poster (Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?) on Thursday June 26.
Looking forward to connecting and sharing our work on spin with the CHIL community!
Looking forward to connecting and sharing our work on spin with the CHIL community!
Huge thanks to my amazing co-authors!
Karen Y.C. Zhang, @ramezkouzy.bsky.social, @ijmarshall.bsky.social, @jessyjli.bsky.social, & @byron.bsky.social [7/7]
Check out our full findings here: arxiv.org/abs/2502.07963
Karen Y.C. Zhang, @ramezkouzy.bsky.social, @ijmarshall.bsky.social, @jessyjli.bsky.social, & @byron.bsky.social [7/7]
Check out our full findings here: arxiv.org/abs/2502.07963
Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
Medical research faces well-documented challenges in translating novel treatments into clinical practice. Publishing incentives encourage researchers to present "positive" findings, even when empirica...
arxiv.org
February 15, 2025 at 2:34 AM
Huge thanks to my amazing co-authors!
Karen Y.C. Zhang, @ramezkouzy.bsky.social, @ijmarshall.bsky.social, @jessyjli.bsky.social, & @byron.bsky.social [7/7]
Check out our full findings here: arxiv.org/abs/2502.07963
Karen Y.C. Zhang, @ramezkouzy.bsky.social, @ijmarshall.bsky.social, @jessyjli.bsky.social, & @byron.bsky.social [7/7]
Check out our full findings here: arxiv.org/abs/2502.07963
Can we fix this? We tested zero-shot prompts to reduce LLMs' susceptibility to spin.
Good news: prompts that encouraged reasoning reduced their tendency to overstate trial results! 🛠️
Careful design is key to improving evidence synthesis for clinical decisions. [6/7]
Good news: prompts that encouraged reasoning reduced their tendency to overstate trial results! 🛠️
Careful design is key to improving evidence synthesis for clinical decisions. [6/7]
February 15, 2025 at 2:34 AM
Can we fix this? We tested zero-shot prompts to reduce LLMs' susceptibility to spin.
Good news: prompts that encouraged reasoning reduced their tendency to overstate trial results! 🛠️
Careful design is key to improving evidence synthesis for clinical decisions. [6/7]
Good news: prompts that encouraged reasoning reduced their tendency to overstate trial results! 🛠️
Careful design is key to improving evidence synthesis for clinical decisions. [6/7]
When we asked LLMs to simplify abstracts into plain language, they often propagated spin into their summaries. This means LLMs could unintentionally mislead patients and non-experts about the effectiveness of treatments. 😱 [5/7]
February 15, 2025 at 2:34 AM
When we asked LLMs to simplify abstracts into plain language, they often propagated spin into their summaries. This means LLMs could unintentionally mislead patients and non-experts about the effectiveness of treatments. 😱 [5/7]
We asked LLMs how favorably they perceived a treatment’s results (0-10 scale). Even though LLMs could detect spin, they were far more influenced by it than human experts.
Meaning: LLMs believed spun abstracts presented more favorable results! 😬 [4/7]
Meaning: LLMs believed spun abstracts presented more favorable results! 😬 [4/7]
February 15, 2025 at 2:34 AM
We asked LLMs how favorably they perceived a treatment’s results (0-10 scale). Even though LLMs could detect spin, they were far more influenced by it than human experts.
Meaning: LLMs believed spun abstracts presented more favorable results! 😬 [4/7]
Meaning: LLMs believed spun abstracts presented more favorable results! 😬 [4/7]
When we prompted 22 LLMs to identify spin in medical abstracts, we found that they were moderately to strongly capable of detecting spin.
However, things got interesting when we asked LLMs to interpret the results… [3/7]
🔽
However, things got interesting when we asked LLMs to interpret the results… [3/7]
🔽
February 15, 2025 at 2:34 AM
When we prompted 22 LLMs to identify spin in medical abstracts, we found that they were moderately to strongly capable of detecting spin.
However, things got interesting when we asked LLMs to interpret the results… [3/7]
🔽
However, things got interesting when we asked LLMs to interpret the results… [3/7]
🔽
So what is spin?
Spin refers to reporting strategies that make experimental treatments appear more beneficial than they actually are—often distracting from nonsignificant results.
Example:
❌ “The treatment shows a promising trend toward significance…”
✅ “No significant difference was found.”
[2/7]
Spin refers to reporting strategies that make experimental treatments appear more beneficial than they actually are—often distracting from nonsignificant results.
Example:
❌ “The treatment shows a promising trend toward significance…”
✅ “No significant difference was found.”
[2/7]
February 15, 2025 at 2:34 AM
So what is spin?
Spin refers to reporting strategies that make experimental treatments appear more beneficial than they actually are—often distracting from nonsignificant results.
Example:
❌ “The treatment shows a promising trend toward significance…”
✅ “No significant difference was found.”
[2/7]
Spin refers to reporting strategies that make experimental treatments appear more beneficial than they actually are—often distracting from nonsignificant results.
Example:
❌ “The treatment shows a promising trend toward significance…”
✅ “No significant difference was found.”
[2/7]
As someone interested in an academic position post-PhD, I found this post very helpful. Thank you for sharing your wisdom and advice.
January 24, 2025 at 5:45 PM
As someone interested in an academic position post-PhD, I found this post very helpful. Thank you for sharing your wisdom and advice.
The application form says it is no longer accepting responses. Is the application closed now?
January 6, 2025 at 11:54 PM
The application form says it is no longer accepting responses. Is the application closed now?