Data Elixir
@dataelixir.com
Data Elixir is a weekly newsletter with curated data science picks from around the web. Subscribe at dataelixir.com and follow us here for selections between issues. Covering machine learning, data visualization, analytics, and strategy.
Sometimes the best polars pattern is knowing when to exit the DataFrame. partitionby() splits data into a dict of frames, letting you process with list comprehensions. Cleaner than forcing everything through mapgroups() when further wrangling isn't needed.
Python Rgonomics: User-defined functions in polars | Emily Riederer
Polars provides a consistent API for conducting transformations against a DataFrame. But what do you do when you need to apply a user-defined function beyond the native API? This post surveys the…
www.emilyriederer.com
November 24, 2025 at 1:37 PM
Sometimes the best polars pattern is knowing when to exit the DataFrame. partitionby() splits data into a dict of frames, letting you process with list comprehensions. Cleaner than forcing everything through mapgroups() when further wrangling isn't needed.
Most devs treat AI coding agents like infinite context machines. Reality: a 200k token window fills fast. The /compact feature is a trap. Better approach: /clear + document state in markdown, then resume. Treat context like disk space. You need a cleanup strategy.
How I Use Every Claude Code Feature
A brain dump of all the ways I've been using Claude Code.
blog.sshh.io
November 23, 2025 at 4:15 PM
Most devs treat AI coding agents like infinite context machines. Reality: a 200k token window fills fast. The /compact feature is a trap. Better approach: /clear + document state in markdown, then resume. Treat context like disk space. You need a cleanup strategy.
Most modern dimensionality reduction (t-SNE, UMAP, Isomap) shares a pattern: represent data as a graph capturing local similarity, then embed to preserve that structure. It's graphs all the way down.
A Visual Introduction to Dimensionality Reduction with Isomap
"To deal with hyper-planes in a 14-dimensional space, visualize a 3D space and say 'fourteen' to yourself very loudly. Everyone does it." - Geoffrey Hinton
alechelbling.com
November 22, 2025 at 3:47 AM
Most modern dimensionality reduction (t-SNE, UMAP, Isomap) shares a pattern: represent data as a graph capturing local similarity, then embed to preserve that structure. It's graphs all the way down.
Reposted by Data Elixir
I curated some readings for class on "data tensions" and the list felt worth sharing. Come on a tour of datasets, books, the web, and AI with me...
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
Torching the Modern-Day Library of Alexandria
“Somewhere at Google there is a database containing 25 million books and nobody is allowed to read them.”
www.theatlantic.com
November 14, 2025 at 4:39 PM
I curated some readings for class on "data tensions" and the list felt worth sharing. Come on a tour of datasets, books, the web, and AI with me...
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
Everyone's rushing to pgvector for "simple" vector search in Postgres. This reality check shows what actually happens at scale: indexing nightmares and performance walls. Simple isn't always sustainable in production.
The Case Against pgvector | Alex Jacobs
What happens when you try to run pgvector in production and discover all the things the blog posts conveniently forgot to mention
alex-jacobs.com
November 14, 2025 at 2:01 AM
Everyone's rushing to pgvector for "simple" vector search in Postgres. This reality check shows what actually happens at scale: indexing nightmares and performance walls. Simple isn't always sustainable in production.
When healthcare becomes algorithmic, what gets optimized out? This Guardian essay asks the hard question about AI spreading through diagnostics and therapy: are we trading care quality for efficiency without realizing the cost?
What we lose when we surrender care to algorithms | Eric Reinhart
A dangerous faith in AI is sweeping American healthcare – with consequences for the basis of society itself
www.theguardian.com
November 13, 2025 at 4:16 AM
When healthcare becomes algorithmic, what gets optimized out? This Guardian essay asks the hard question about AI spreading through diagnostics and therapy: are we trading care quality for efficiency without realizing the cost?
Thinking Machines Lab solved a problem everyone accepted as unsolvable: LLM nondeterminism at temperature 0. Same prompt, same model, 1000 runs → 80 different outputs. With batch-invariant kernels? Bitwise identical every time. Open sourced. www.distributedthoughts.org/will-i-make-...
Will I Make It To The Restaurant Before The Soup Dumplings Get Cold? (And Other Problems In Machine Learning)
I'm chronically late. Not because I want to be rude - I feel terrible about it every single time - but because I'm catastrophically bad at predicting how long it takes to get anywhere. Turns out…
www.distributedthoughts.org
November 8, 2025 at 3:47 AM
Thinking Machines Lab solved a problem everyone accepted as unsolvable: LLM nondeterminism at temperature 0. Same prompt, same model, 1000 runs → 80 different outputs. With batch-invariant kernels? Bitwise identical every time. Open sourced. www.distributedthoughts.org/will-i-make-...
Most marketplaces have SKUs. Etsy has 100M+ unique items with no standard attributes. How do you build filters when one listing is a "porcelain sculpture that looks like a t-shirt" and dimensions live in random photo text? www.etsy.com/codeascraft/...
www.etsy.com
November 6, 2025 at 3:37 AM
Most marketplaces have SKUs. Etsy has 100M+ unique items with no standard attributes. How do you build filters when one listing is a "porcelain sculpture that looks like a t-shirt" and dimensions live in random photo text? www.etsy.com/codeascraft/...
GeoUtil converts between GeoJSON, TopoJSON, Shapefile, KML, WKT, and CSV without touching a server. TopoJSON compression alone cuts file sizes 80%+ while preserving topology. All free, all browser-based. geoutil.com
GeoUtil — Free Online Map & Geography Tools
All-in-one online geography toolkit. Measure distance & area, convert GeoJSON, TopoJSON, JSON, merge or minify files, and more — fast, free, and browser-based.
geoutil.com
October 31, 2025 at 2:47 PM
GeoUtil converts between GeoJSON, TopoJSON, Shapefile, KML, WKT, and CSV without touching a server. TopoJSON compression alone cuts file sizes 80%+ while preserving topology. All free, all browser-based. geoutil.com
Debugging constraint problems is backwards: remove constraints until something works, then figure out what broke. No stack traces, just an "unsatisfiable." Forces you to think differently about what you're actually asking the system to solve. www.righto.com/2025/10/solv...
Solving the NYTimes Pips puzzle with a constraint solver
The New York Times recently introduced a new daily puzzle called Pips . You place a set of dominoes on a grid, satisfying various condition...
www.righto.com
October 30, 2025 at 3:04 PM
Debugging constraint problems is backwards: remove constraints until something works, then figure out what broke. No stack traces, just an "unsatisfiable." Forces you to think differently about what you're actually asking the system to solve. www.righto.com/2025/10/solv...
Most interpretable models sacrifice accuracy. Most accurate models are black boxes. TRUST breaks this trade-off by combining parametric and non-parametric approaches and offers full prediction explanations without losing performance. Published at PRICAI 2025.
trust-free
Transparent, Robust & Ultra-Sparse Trees (TRUST™) - Free Version
pypi.org
October 29, 2025 at 2:37 PM
Most interpretable models sacrifice accuracy. Most accurate models are black boxes. TRUST breaks this trade-off by combining parametric and non-parametric approaches and offers full prediction explanations without losing performance. Published at PRICAI 2025.
Data products shouldn't live forever by default. Netflix treats outdated metrics like deprecated software and actively sunset them. The cost of maintaining zombie datasets? Lost trust and accumulated technical debt that blocks innovation.
Data as a Product: Applying a Product Mindset to Data at Netflix
Introduction: What if we treated data with the same care and intentionality as a consumer-facing product? Adopting a “data as a product”…
netflixtechblog.medium.com
October 29, 2025 at 12:13 AM
Data products shouldn't live forever by default. Netflix treats outdated metrics like deprecated software and actively sunset them. The cost of maintaining zombie datasets? Lost trust and accumulated technical debt that blocks innovation.
Reposted by Data Elixir
What will you be using this #30DayMapChallenge? Cadence is offering its users £2,500 worth of prizes this #30daymapchallenge. And every user (new or existing) gets a free Professional upgrade for November! Learn more: cadence.cityscience.com/blog/30-day-...
30 Day Map Challenge
The 30 Day Map Challenge with Cadence – November 2025 This November, Cadence is proud to support the 30 Day Map Challenge – a global celebration of maps, creativity and storytelling. Whether …
cadence.cityscience.com
October 24, 2025 at 12:20 PM
What will you be using this #30DayMapChallenge? Cadence is offering its users £2,500 worth of prizes this #30daymapchallenge. And every user (new or existing) gets a free Professional upgrade for November! Learn more: cadence.cityscience.com/blog/30-day-...
Why does neural network training almost never fail? Pure combinatorics. A 6K parameter network contains 10^1089 possible sparse subnetworks. That's 10^900 solutions per atom in the universe. We're not smart, we're just brute forcing.
Sparse Networks and Lottery Winners
Embedding Space is a blog about machine learning and artificial intelligence.
embedding-space.github.io
October 24, 2025 at 2:47 PM
Why does neural network training almost never fail? Pure combinatorics. A 6K parameter network contains 10^1089 possible sparse subnetworks. That's 10^900 solutions per atom in the universe. We're not smart, we're just brute forcing.
The best part about #30DayMapChallenge is that it's tool-agnostic. Whether you're using QGIS, Python's geopandas, R's rayshader, or even Blender for 3D visualizations, the focus is on creativity over tech stack. No programming required.
30DayMapChallenge
Daily mapping challenge happening every November!
30daymapchallenge.com
October 23, 2025 at 1:06 PM
The best part about #30DayMapChallenge is that it's tool-agnostic. Whether you're using QGIS, Python's geopandas, R's rayshader, or even Blender for 3D visualizations, the focus is on creativity over tech stack. No programming required.
Stop fitting separate binomial models to compositional data! Your predictions that 130% of respondents chose option A reveal fundamental model misspecification. Dirichlet regression with Gaussian processes respects constraints.
Compositional modeling of plant communities with Dirichlet regression | GAMbler
Compositional data appears everywhere in scientific research, yet many analysts fall back on problematic approaches that ignore fundamental mathematical constraints. I demonstrate how Dirichlet…
ecogambler.netlify.app
October 23, 2025 at 2:37 AM
Stop fitting separate binomial models to compositional data! Your predictions that 130% of respondents chose option A reveal fundamental model misspecification. Dirichlet regression with Gaussian processes respects constraints.
AI systems are now generating, testing, and validating their own hypotheses. DeepMind's Co-Scientist and Stanford's Virtual Lab represent something new: AI as actual scientific collaborator, not just a fancy search engine.
State of AI Report 2025
The State of AI Report analyses the most interesting developments in AI. Read and download here.
www.stateof.ai
October 17, 2025 at 2:47 PM
AI systems are now generating, testing, and validating their own hypotheses. DeepMind's Co-Scientist and Stanford's Virtual Lab represent something new: AI as actual scientific collaborator, not just a fancy search engine.
Black and white, hand-drawn data viz countering visual noise at one of NYC's busiest hubs. Smart choice. Sometimes the most effective data art isn't about adding more complexity but finding clarity in the chaos through intentional restraint.
‘A Data Love Letter to the Subway’
A data-driven animation for Fulton Center commissioned by MTA Arts & Design for its 40th anniversary.
www.pentagram.com
October 17, 2025 at 1:01 AM
Black and white, hand-drawn data viz countering visual noise at one of NYC's busiest hubs. Smart choice. Sometimes the most effective data art isn't about adding more complexity but finding clarity in the chaos through intentional restraint.
The metrics that matter most are the hardest to measure. Bookings take weeks to materialize, but clicks happen instantly. The trap: optimizing for clicks can actually decrease bookings. Correlation isn't causation, especially in A/B tests.
How to estimate correlation between metrics from past A/B tests
Authors: Miha Gazvoda, Christina Katsimerou
booking.ai
October 15, 2025 at 2:37 PM
The metrics that matter most are the hardest to measure. Bookings take weeks to materialize, but clicks happen instantly. The trap: optimizing for clicks can actually decrease bookings. Correlation isn't causation, especially in A/B tests.
Psychology's dirty secret: "data available upon request" usually means data not available at all. Researchers found systematic patterns in why data disappears over time. Open-washing is real and it's undermining reproducibility.
LnuOpen | Meta-Psychology
Many journals now require data sharing and require articles to include a Data Availability Statement. However, several studies over the past two decades have shown that promissory notes about data…
open.lnu.se
October 15, 2025 at 12:13 AM
Psychology's dirty secret: "data available upon request" usually means data not available at all. Researchers found systematic patterns in why data disappears over time. Open-washing is real and it's undermining reproducibility.
Reposted by Data Elixir
Today my @nytimes.com colleagues and I are launching a new series called Lost Science. We interview US scientists who can no longer discover something new about our world, thanks to this year‘s cuts. Here is my first interview with a scientist who studied bees and fires. Gift link: nyti.ms/3IWXbiE
nyti.ms
October 8, 2025 at 11:29 PM
Today my @nytimes.com colleagues and I are launching a new series called Lost Science. We interview US scientists who can no longer discover something new about our world, thanks to this year‘s cuts. Here is my first interview with a scientist who studied bees and fires. Gift link: nyti.ms/3IWXbiE
Parquet is showing its age. CMU researchers built F3 - a columnar format that embeds WebAssembly decoders directly in files. Universal compatibility without the usual compatibility hell. Smart approach for modern ML workloads.
db.cs.cmu.edu/papers/2025/...
db.cs.cmu.edu/papers/2025/...
db.cs.cmu.edu
October 13, 2025 at 1:43 PM
Parquet is showing its age. CMU researchers built F3 - a columnar format that embeds WebAssembly decoders directly in files. Universal compatibility without the usual compatibility hell. Smart approach for modern ML workloads.
db.cs.cmu.edu/papers/2025/...
db.cs.cmu.edu/papers/2025/...
Mathematicians feared nuclear winter would freeze Earth, but it turns out CO2 might do it instead. The math behind climate tipping points is fascinating and terrifyingly unpredictable. Sometimes the thing you're not watching is the real threat.
The Math of Climate Change Tipping Points | Quanta Magazine
Tipping points in our climate predictions are both wildly dramatic and wildly uncertain. Can mathematicians make them useful?
www.quantamagazine.org
October 10, 2025 at 3:43 PM
Mathematicians feared nuclear winter would freeze Earth, but it turns out CO2 might do it instead. The math behind climate tipping points is fascinating and terrifyingly unpredictable. Sometimes the thing you're not watching is the real threat.
Reposted by Data Elixir
Data dictionary template: osf.io/ynqcu
Project summary template: osf.io/q6g8d
Dataset level README template: osf.io/tk4cb
Project summary template: osf.io/q6g8d
Dataset level README template: osf.io/tk4cb
October 8, 2025 at 2:54 PM
Data dictionary template: osf.io/ynqcu
Project summary template: osf.io/q6g8d
Dataset level README template: osf.io/tk4cb
Project summary template: osf.io/q6g8d
Dataset level README template: osf.io/tk4cb
"Silicon samples" - using LLMs to generate fake survey responses instead of recruiting humans. Sounds efficient until you realize small model tweaks completely flip your results. Shortcuts in research usually aren't.
The threat of analytic flexibility in using large language models to simulate human data: A call to attention
Social scientists are now using large language models to create "silicon samples" - synthetic datasets intended to stand in for human respondents, aimed at revolutionising human subjects research.…
arxiv.org
October 9, 2025 at 1:08 PM
"Silicon samples" - using LLMs to generate fake survey responses instead of recruiting humans. Sounds efficient until you realize small model tweaks completely flip your results. Shortcuts in research usually aren't.