Abhinav Rao
LightNews LightNews
aethersura.bsky.social
Abhinav Rao
@aethersura.bsky.social
130 followers 210 following 0 posts
Master's @LTI, CMU
Posts Replies Media Videos
Reposted by Abhinav Rao
shaily99.bsky.social
I will be at #EMNLP2024 presenting our work on "Extrinsic Evaluation of Cultural Competence in Large Language Models" in Poster Session 12 on Thursday 2-3:30 PM.

In this work we take the first steps towards asking whether LLMs can cater to diverse cultures in *user-facing generative* tasks.

[1/7]
Paper titled “Extrinsic Evaluation of Cultural Competence in Large Language Models” by Shaily Bhatt and Fernando Diaz. Along with a figure showing an example from our data. We have two tasks: Question Answering and Story Generation. We collected outputs for 345 QA and 35 story topics, 2 temperatures, 6 LLMs and 193 nationalities. The image shows two example outputs, one from India and one from the USA. The QA example shows outputs for the topic of “legislature”, in the US output words like “United States”, “Senate”, and “House of Representatives” are highlighted. The India output has “India”, “Lok Sabha (House of the People)” and “Rajya Sabha (Council of States)” highlighted. In the case of the story, outputs for India and the US for the topic of “honesty” are shown. For the US, the words “America”, “Tommy”, “park”, and “shiny red apple” are highlighted, while for India, the words “India”, “Raj”, and “mango tree” are highlighted.
November 9, 2024 at 5:24 PM