Dj Ripley
LightNews LightNews
djripley.bsky.social
Dj Ripley
@djripley.bsky.social
2.7K followers 990 following 1.5K posts
Scholar, teacher, DJ, radical, junglist.
Former streetmedic, current assoc. prof. More at http://larisa-mann.com
She/her.
I wrote : RUDE CITIZENSHIP: copyright, Jamaican popular music & reverberations of colonial power https://tinyurl.com/RudeCBook
Posts Replies Media Videos
Reposted by Dj Ripley
ketanjoshi.co
Is there no simple gut instinct at work here? "This thing lies by design, and unchangeably based on its fundamental technological construction, and is therefore unsuited to replacing human science communication"

Nope. Just gets shrugged off as a minor whoopsie
November 25, 2025 at 7:11 PM
Reposted by Dj Ripley
ketanjoshi.co
The more I think about this, the more it really, really, REALLY bothers me

We've reached a point where a text-based reference fabrication engine - comfortably the most effective machine liar we've ever invented - is being pushed by an ultra-polluting tech company and accepted by climate scientists
ketanjoshi.co Ketan Joshi @ketanjoshi.co · 5d
Deeply absurd. This Google PDF published on a blog (arxiv, not peer reviewed) claims an LLM is "PhD level" but in most cases the MAJORITY of reference URLs were invalid or inaccessible.

A PhD sitting down and just fabricating >50% of sources = career ending

arxiv.org/abs/2511.11597
CLINB: A Climate Intelligence Benchmark for Foundational Models Michelle Chen Huebscher1 , Katharine Mach2 , Aleksandar Stanić1 , Markus Leippold1,3, Ben Gaiarin1 , Zeke Hausfather4 , Elisa Rawat , Erich Fischer5 , Massimiliano Ciaramita1 , Joeri Rogelj6 , Christian Buck1 , Lierni Sestorain Saralegui1 and Reto Knutti5 1Google DeepMind, 2University of Miami, 3University of Zurich, 4Stripe, 5ETH Zurich, 6 Imperial College London Evaluating how Large Language Models (LLMs) handle complex, specialized knowledge remains a critical challenge. We address this through the lens of climate change by introducing CLINB, a benchmark that assesses models on open-ended, grounded, multimodal question answering tasks with clear requirements for knowledge quality and evidential support. CLINB relies on a dataset of real users’ questions and evaluation rubrics curated by leading climate scientists. We implement and validate a model-based evaluation process and evaluate several frontier models. Our findings reveal a critical dichotomy. Frontier models demonstrate remarkable knowledge synthesis capabilities, often exhibiting PhD-level understanding and presentation quality. They outperform “hybrid" answers curated by domain experts assisted by weaker models. However, this performance is countered by failures in grounding. The quality of evidence varies, with substantial hallucination rates for references and images. We argue that bridging this gap between knowledge synthesis and verifiable attribution is essential for the deployment of AI in scientific workflows and that reliable, interpretable benchmarks like CLINB are needed to progress towards building trustworthy AI systems. Total Reference URLs Generated claude-opus-4-1 claude-sonnet-4 gpt-5 hybrid gemini-2.5-pro gemini-2.5-flash o3 0.0 0.2 0.4 0.6 0.8 1.0 Proportion Reference URL Status hybrid gemini-2.5-pro claude-opus-4-1 o3 gemini-2.5-flash claude-sonnet-4 0 200 400 600 800 1000 Count of URLs Total Image URLs Generated hybrid claude-opus-4-1 gemini-2.5-flash gemini-2.5-pro claude-sonnet-4 o3 0.0 0.2 0.4 0.6 0.8 1.0 Proportion Image URL Status Status OK INACCESSIBLE_CONTENT INVALID_URL ERROR Figure 3 | Number of reference (top), and image (bottom), URLs and their status. Ablations We perform several ablation studies with the autorater (Table 4). Notably, removing the question-specific rubrics from the prompt changes the results only in the bottom half, with the Hybrid answers overtaken by Gemini 2.5 Flash and Claude Sonnet 4. This suggests that the additional resolution provided by the rubrics applies primarily to the kind of responses used to develop the rubrics. Or, in other words, that rubrics are far from complete. Hence, it is important that rubrics adapt to new data as better models become availab A New Expert-Grounded Benchmark for Scientific AI We introduce CLINB, a benchmark for modelbased evaluation of frontier models on complex, multimodal scientific communication. Its core is a new dataset of real-world climate questions paired with data-driven, question-specific evaluation rubrics, curated and validated by leading climate scientists through a novel three-phase, human-in-the-loop process.2 PhD-Level Synthesis vs. Attribution Failures Frontier models demonstrate remarkable knowledge synthesis, often exhibiting a PhD-level understanding. However, this performance masks a critical inadequacy in grounding. We report substantial hallucination rates for references (10% to 25%) and even more failures for images (50% to 80% in certain settings), exposing a major gap between synthesis and verifiable attribution. Insights into Human-AI Collaboration Dynamics Autonomous frontier models surpass ’hybrid’ answers (curated by experts using weaker AI assistance), revealing the assisting model’s capability—not human oversight—as the primary bottleneck. Counter-intuitively, highly motivated non-specialists (our ’Advocates’) who deeply engage with AI tools can produce higher-quality answers than domain experts who engage less with AI during answer curation. A Validated Methodology for Scalable Oversight We validate a rigorous, rubric-based autorater. Ablation studies demonstrate that structured prompts and automated evidence-checking are essential for mitigating inherent LLM judge biases. This process is hampered by inaccessible sources (up to 50%). Furthermore, we identify evaluation challenges, including model familiarity bias in human raters and the limitations of rubrics to generalize across models.
November 25, 2025 at 7:01 PM
djripley.bsky.social
my mom quotes HER mom who used to say "the earth should swallow them up! lightning should strike"!
In honor of their heritage I also would add "may they be chandeliers, hanging by day and burning by night"
November 25, 2025 at 7:58 PM