Data Elixir
@dataelixir.com
Data Elixir is a weekly newsletter with curated data science picks from around the web. Subscribe at dataelixir.com and follow us here for selections between issues. Covering machine learning, data visualization, analytics, and strategy.
Learning SQL is like learning a foreign language: you need to read more variations than you'll actually write. Learn disciplined canonical syntax for your own queries, but understand the messy dialects others use.
A modern guide to SQL JOINs
There are many SQL JOINs guides and tutorials, but this one takes a different approach. We try to avoid misleading wording and imagery, and we structure the material in a different way. The goal of…
kb.databasedesignbook.com
December 5, 2025 at 2:01 AM
Learning SQL is like learning a foreign language: you need to read more variations than you'll actually write. Learn disciplined canonical syntax for your own queries, but understand the messy dialects others use.
Side projects still open more doors than traditional applications in data roles. The trick: build small, interesting things that signal skills and make your work discoverable. Especially relevant given the current job market.
Make Things, Tell People
On side projects and finding work
presentofcoding.substack.com
December 1, 2025 at 1:37 PM
Side projects still open more doors than traditional applications in data roles. The trick: build small, interesting things that signal skills and make your work discoverable. Especially relevant given the current job market.
Most time-to-event metrics are broken. Amazon thought customer support wait times were under 1 min. Bezos called in: 10+ minutes. Why? They only measured customers who stayed on hold long enough to be served. The ones who hung up? Never counted. www.counting-stuff.com/why-you-are-...
Why You Are (Probably) Measuring Time Wrong: Why do we need to use Survival Analysis more
Author: Michał Chorowski [Hey everyone! A guest post from Michał this week! This newsletter is always willing to host/share data-related content, so if you've created anything you'd like to share,…
www.counting-stuff.com
November 29, 2025 at 4:02 PM
Most time-to-event metrics are broken. Amazon thought customer support wait times were under 1 min. Bezos called in: 10+ minutes. Why? They only measured customers who stayed on hold long enough to be served. The ones who hung up? Never counted. www.counting-stuff.com/why-you-are-...
Hot take: Your analysts doing "some basic data engineering" is killing your analytics function. The MTA hired 5 dedicated data engineers and it unlocked everything else. Stop asking data scientists to maintain pipelines. www.mta.info/article/less...
Lessons learned in starting a central data team
Learn how the MTA succeeded in setting up a central data team and a general purpose, cloud-based platform for data analytics.
www.mta.info
November 29, 2025 at 3:47 AM
Hot take: Your analysts doing "some basic data engineering" is killing your analytics function. The MTA hired 5 dedicated data engineers and it unlocked everything else. Stop asking data scientists to maintain pipelines. www.mta.info/article/less...
Sometimes the best polars pattern is knowing when to exit the DataFrame. partitionby() splits data into a dict of frames, letting you process with list comprehensions. Cleaner than forcing everything through mapgroups() when further wrangling isn't needed.
Python Rgonomics: User-defined functions in polars | Emily Riederer
Polars provides a consistent API for conducting transformations against a DataFrame. But what do you do when you need to apply a user-defined function beyond the native API? This post surveys the…
www.emilyriederer.com
November 24, 2025 at 1:37 PM
Sometimes the best polars pattern is knowing when to exit the DataFrame. partitionby() splits data into a dict of frames, letting you process with list comprehensions. Cleaner than forcing everything through mapgroups() when further wrangling isn't needed.
Most devs treat AI coding agents like infinite context machines. Reality: a 200k token window fills fast. The /compact feature is a trap. Better approach: /clear + document state in markdown, then resume. Treat context like disk space. You need a cleanup strategy.
How I Use Every Claude Code Feature
A brain dump of all the ways I've been using Claude Code.
blog.sshh.io
November 23, 2025 at 4:15 PM
Most devs treat AI coding agents like infinite context machines. Reality: a 200k token window fills fast. The /compact feature is a trap. Better approach: /clear + document state in markdown, then resume. Treat context like disk space. You need a cleanup strategy.
Most modern dimensionality reduction (t-SNE, UMAP, Isomap) shares a pattern: represent data as a graph capturing local similarity, then embed to preserve that structure. It's graphs all the way down.
A Visual Introduction to Dimensionality Reduction with Isomap
"To deal with hyper-planes in a 14-dimensional space, visualize a 3D space and say 'fourteen' to yourself very loudly. Everyone does it." - Geoffrey Hinton
alechelbling.com
November 22, 2025 at 3:47 AM
Most modern dimensionality reduction (t-SNE, UMAP, Isomap) shares a pattern: represent data as a graph capturing local similarity, then embed to preserve that structure. It's graphs all the way down.
Reposted by Data Elixir
I curated some readings for class on "data tensions" and the list felt worth sharing. Come on a tour of datasets, books, the web, and AI with me...
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
Torching the Modern-Day Library of Alexandria
“Somewhere at Google there is a database containing 25 million books and nobody is allowed to read them.”
www.theatlantic.com
November 14, 2025 at 4:39 PM
I curated some readings for class on "data tensions" and the list felt worth sharing. Come on a tour of datasets, books, the web, and AI with me...
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
We'll start with this piece on the Google Books project: the hopes, dreams, disasters, and aftermath of building a public library on the internet.
1/n
Everyone's rushing to pgvector for "simple" vector search in Postgres. This reality check shows what actually happens at scale: indexing nightmares and performance walls. Simple isn't always sustainable in production.
The Case Against pgvector | Alex Jacobs
What happens when you try to run pgvector in production and discover all the things the blog posts conveniently forgot to mention
alex-jacobs.com
November 14, 2025 at 2:01 AM
Everyone's rushing to pgvector for "simple" vector search in Postgres. This reality check shows what actually happens at scale: indexing nightmares and performance walls. Simple isn't always sustainable in production.
When healthcare becomes algorithmic, what gets optimized out? This Guardian essay asks the hard question about AI spreading through diagnostics and therapy: are we trading care quality for efficiency without realizing the cost?
What we lose when we surrender care to algorithms | Eric Reinhart
A dangerous faith in AI is sweeping American healthcare – with consequences for the basis of society itself
www.theguardian.com
November 13, 2025 at 4:16 AM
When healthcare becomes algorithmic, what gets optimized out? This Guardian essay asks the hard question about AI spreading through diagnostics and therapy: are we trading care quality for efficiency without realizing the cost?
Thinking Machines Lab solved a problem everyone accepted as unsolvable: LLM nondeterminism at temperature 0. Same prompt, same model, 1000 runs → 80 different outputs. With batch-invariant kernels? Bitwise identical every time. Open sourced. www.distributedthoughts.org/will-i-make-...
Will I Make It To The Restaurant Before The Soup Dumplings Get Cold? (And Other Problems In Machine Learning)
I'm chronically late. Not because I want to be rude - I feel terrible about it every single time - but because I'm catastrophically bad at predicting how long it takes to get anywhere. Turns out…
www.distributedthoughts.org
November 8, 2025 at 3:47 AM
Thinking Machines Lab solved a problem everyone accepted as unsolvable: LLM nondeterminism at temperature 0. Same prompt, same model, 1000 runs → 80 different outputs. With batch-invariant kernels? Bitwise identical every time. Open sourced. www.distributedthoughts.org/will-i-make-...
Most marketplaces have SKUs. Etsy has 100M+ unique items with no standard attributes. How do you build filters when one listing is a "porcelain sculpture that looks like a t-shirt" and dimensions live in random photo text? www.etsy.com/codeascraft/...
www.etsy.com
November 6, 2025 at 3:37 AM
Most marketplaces have SKUs. Etsy has 100M+ unique items with no standard attributes. How do you build filters when one listing is a "porcelain sculpture that looks like a t-shirt" and dimensions live in random photo text? www.etsy.com/codeascraft/...
GeoUtil converts between GeoJSON, TopoJSON, Shapefile, KML, WKT, and CSV without touching a server. TopoJSON compression alone cuts file sizes 80%+ while preserving topology. All free, all browser-based. geoutil.com
GeoUtil — Free Online Map & Geography Tools
All-in-one online geography toolkit. Measure distance & area, convert GeoJSON, TopoJSON, JSON, merge or minify files, and more — fast, free, and browser-based.
geoutil.com
October 31, 2025 at 2:47 PM
GeoUtil converts between GeoJSON, TopoJSON, Shapefile, KML, WKT, and CSV without touching a server. TopoJSON compression alone cuts file sizes 80%+ while preserving topology. All free, all browser-based. geoutil.com
Debugging constraint problems is backwards: remove constraints until something works, then figure out what broke. No stack traces, just an "unsatisfiable." Forces you to think differently about what you're actually asking the system to solve. www.righto.com/2025/10/solv...
Solving the NYTimes Pips puzzle with a constraint solver
The New York Times recently introduced a new daily puzzle called Pips . You place a set of dominoes on a grid, satisfying various condition...
www.righto.com
October 30, 2025 at 3:04 PM
Debugging constraint problems is backwards: remove constraints until something works, then figure out what broke. No stack traces, just an "unsatisfiable." Forces you to think differently about what you're actually asking the system to solve. www.righto.com/2025/10/solv...
Most interpretable models sacrifice accuracy. Most accurate models are black boxes. TRUST breaks this trade-off by combining parametric and non-parametric approaches and offers full prediction explanations without losing performance. Published at PRICAI 2025.
trust-free
Transparent, Robust & Ultra-Sparse Trees (TRUST™) - Free Version
pypi.org
October 29, 2025 at 2:37 PM
Most interpretable models sacrifice accuracy. Most accurate models are black boxes. TRUST breaks this trade-off by combining parametric and non-parametric approaches and offers full prediction explanations without losing performance. Published at PRICAI 2025.
Data products shouldn't live forever by default. Netflix treats outdated metrics like deprecated software and actively sunset them. The cost of maintaining zombie datasets? Lost trust and accumulated technical debt that blocks innovation.
Data as a Product: Applying a Product Mindset to Data at Netflix
Introduction: What if we treated data with the same care and intentionality as a consumer-facing product? Adopting a “data as a product”…
netflixtechblog.medium.com
October 29, 2025 at 12:13 AM
Data products shouldn't live forever by default. Netflix treats outdated metrics like deprecated software and actively sunset them. The cost of maintaining zombie datasets? Lost trust and accumulated technical debt that blocks innovation.
Reposted by Data Elixir
What will you be using this #30DayMapChallenge? Cadence is offering its users £2,500 worth of prizes this #30daymapchallenge. And every user (new or existing) gets a free Professional upgrade for November! Learn more: cadence.cityscience.com/blog/30-day-...
30 Day Map Challenge
The 30 Day Map Challenge with Cadence – November 2025 This November, Cadence is proud to support the 30 Day Map Challenge – a global celebration of maps, creativity and storytelling. Whether …
cadence.cityscience.com
October 24, 2025 at 12:20 PM
What will you be using this #30DayMapChallenge? Cadence is offering its users £2,500 worth of prizes this #30daymapchallenge. And every user (new or existing) gets a free Professional upgrade for November! Learn more: cadence.cityscience.com/blog/30-day-...
Why does neural network training almost never fail? Pure combinatorics. A 6K parameter network contains 10^1089 possible sparse subnetworks. That's 10^900 solutions per atom in the universe. We're not smart, we're just brute forcing.
Sparse Networks and Lottery Winners
Embedding Space is a blog about machine learning and artificial intelligence.
embedding-space.github.io
October 24, 2025 at 2:47 PM
Why does neural network training almost never fail? Pure combinatorics. A 6K parameter network contains 10^1089 possible sparse subnetworks. That's 10^900 solutions per atom in the universe. We're not smart, we're just brute forcing.
The best part about #30DayMapChallenge is that it's tool-agnostic. Whether you're using QGIS, Python's geopandas, R's rayshader, or even Blender for 3D visualizations, the focus is on creativity over tech stack. No programming required.
30DayMapChallenge
Daily mapping challenge happening every November!
30daymapchallenge.com
October 23, 2025 at 1:06 PM
The best part about #30DayMapChallenge is that it's tool-agnostic. Whether you're using QGIS, Python's geopandas, R's rayshader, or even Blender for 3D visualizations, the focus is on creativity over tech stack. No programming required.
Stop fitting separate binomial models to compositional data! Your predictions that 130% of respondents chose option A reveal fundamental model misspecification. Dirichlet regression with Gaussian processes respects constraints.
Compositional modeling of plant communities with Dirichlet regression | GAMbler
Compositional data appears everywhere in scientific research, yet many analysts fall back on problematic approaches that ignore fundamental mathematical constraints. I demonstrate how Dirichlet…
ecogambler.netlify.app
October 23, 2025 at 2:37 AM
Stop fitting separate binomial models to compositional data! Your predictions that 130% of respondents chose option A reveal fundamental model misspecification. Dirichlet regression with Gaussian processes respects constraints.
AI systems are now generating, testing, and validating their own hypotheses. DeepMind's Co-Scientist and Stanford's Virtual Lab represent something new: AI as actual scientific collaborator, not just a fancy search engine.
State of AI Report 2025
The State of AI Report analyses the most interesting developments in AI. Read and download here.
www.stateof.ai
October 17, 2025 at 2:47 PM
AI systems are now generating, testing, and validating their own hypotheses. DeepMind's Co-Scientist and Stanford's Virtual Lab represent something new: AI as actual scientific collaborator, not just a fancy search engine.
Black and white, hand-drawn data viz countering visual noise at one of NYC's busiest hubs. Smart choice. Sometimes the most effective data art isn't about adding more complexity but finding clarity in the chaos through intentional restraint.
‘A Data Love Letter to the Subway’
A data-driven animation for Fulton Center commissioned by MTA Arts & Design for its 40th anniversary.
www.pentagram.com
October 17, 2025 at 1:01 AM
Black and white, hand-drawn data viz countering visual noise at one of NYC's busiest hubs. Smart choice. Sometimes the most effective data art isn't about adding more complexity but finding clarity in the chaos through intentional restraint.
The metrics that matter most are the hardest to measure. Bookings take weeks to materialize, but clicks happen instantly. The trap: optimizing for clicks can actually decrease bookings. Correlation isn't causation, especially in A/B tests.
How to estimate correlation between metrics from past A/B tests
Authors: Miha Gazvoda, Christina Katsimerou
booking.ai
October 15, 2025 at 2:37 PM
The metrics that matter most are the hardest to measure. Bookings take weeks to materialize, but clicks happen instantly. The trap: optimizing for clicks can actually decrease bookings. Correlation isn't causation, especially in A/B tests.
Psychology's dirty secret: "data available upon request" usually means data not available at all. Researchers found systematic patterns in why data disappears over time. Open-washing is real and it's undermining reproducibility.
LnuOpen | Meta-Psychology
Many journals now require data sharing and require articles to include a Data Availability Statement. However, several studies over the past two decades have shown that promissory notes about data…
open.lnu.se
October 15, 2025 at 12:13 AM
Psychology's dirty secret: "data available upon request" usually means data not available at all. Researchers found systematic patterns in why data disappears over time. Open-washing is real and it's undermining reproducibility.
Reposted by Data Elixir
Today my @nytimes.com colleagues and I are launching a new series called Lost Science. We interview US scientists who can no longer discover something new about our world, thanks to this year‘s cuts. Here is my first interview with a scientist who studied bees and fires. Gift link: nyti.ms/3IWXbiE
nyti.ms
October 8, 2025 at 11:29 PM
Today my @nytimes.com colleagues and I are launching a new series called Lost Science. We interview US scientists who can no longer discover something new about our world, thanks to this year‘s cuts. Here is my first interview with a scientist who studied bees and fires. Gift link: nyti.ms/3IWXbiE