Nikhil Benesch
@benesch.bsky.social
Systems engineer @turbopuffer.bsky.social. Former CTO @materialize.com.
They’re calling it our most boring feature yet.
boring(n): mundane; works exactly as expected; the highest praise you can give a database
boring(n): mundane; works exactly as expected; the highest praise you can give a database
by popular demand, now introducing one-dimensional vectors 🎈
July 22, 2025 at 3:48 PM
They’re calling it our most boring feature yet.
boring(n): mundane; works exactly as expected; the highest praise you can give a database
boring(n): mundane; works exactly as expected; the highest praise you can give a database
tpuf Python client went async today! 🐡 ❤️ 🐍
Async client perf slightly edges out sync client perf under heavy query load.
github.com/turbopuffer/...
Async client perf slightly edges out sync client perf under heavy query load.
github.com/turbopuffer/...
June 10, 2025 at 7:08 PM
tpuf Python client went async today! 🐡 ❤️ 🐍
Async client perf slightly edges out sync client perf under heavy query load.
github.com/turbopuffer/...
Async client perf slightly edges out sync client perf under heavy query load.
github.com/turbopuffer/...
Big day today at turbopuffer HQ.
May 14, 2025 at 2:46 PM
Big day today at turbopuffer HQ.
Reposted by Nikhil Benesch
when we said "coming soon" we really meant it
now puffin' in aws-us-east-1 and aws-eu-central-1
now puffin' in aws-us-east-1 and aws-eu-central-1
March 7, 2025 at 7:41 PM
when we said "coming soon" we really meant it
now puffin' in aws-us-east-1 and aws-eu-central-1
now puffin' in aws-us-east-1 and aws-eu-central-1
Things move fast at turbopuffer. Now puffin' in aws/us-east-1 and aws/eu-central-1 too.
March 7, 2025 at 6:06 PM
Things move fast at turbopuffer. Now puffin' in aws/us-east-1 and aws/eu-central-1 too.
I've been dreaming about conditional writes on S3 for years. I couldn't have asked for a better way to celebrate than getting to ship tpuf on AWS. 🐡
now available: turbopuffer AWS regions ☁️
come find us in us-west-2 and ap-southeast-2
come find us in us-west-2 and ap-southeast-2
March 6, 2025 at 9:40 PM
I've been dreaming about conditional writes on S3 for years. I couldn't have asked for a better way to celebrate than getting to ship tpuf on AWS. 🐡
It’s a rare day that my love of going to battle with build systems pays off like this. Kudos to GCP for a very impressive new SKU. 🐡💨
GCP released a new ARM machine type with NVMe SSDs recently
70% increase in end-to-end production indexing throughput at 20-33% lower machine cost!
70% increase in end-to-end production indexing throughput at 20-33% lower machine cost!
February 26, 2025 at 10:28 PM
It’s a rare day that my love of going to battle with build systems pays off like this. Kudos to GCP for a very impressive new SKU. 🐡💨
Couldn't be more excited to be joining the team at @turbopuffer.bsky.social. 🐡💨
welcoming two excellent additions to team tpuf this week: @arash11gt as a customer engineer, and @nikhilbenesch to the DB team from pasts at Materialize & Cockroach.
we're locking in a p999 eng team to build the best search engine for scale.
we're locking in a p999 eng team to build the best search engine for scale.
February 6, 2025 at 6:26 PM
Couldn't be more excited to be joining the team at @turbopuffer.bsky.social. 🐡💨
Just catching up on my NULL BITMAPS and this is easily the best intuition for write skew I've ever seen described. The analogy to merge skew in a codebase is genius.
this week on NULL BITMAP, intuition for snapshot isolation: it’s like git buttondown.com/jaffray/arch...
Git Workflow is Snapshot-Isolated
We are apparently continuing our series of posts borne from thinking about Amazon DSQL. Today we are talking about skew. Amy and Bill are two developers...
buttondown.com
December 30, 2024 at 10:48 PM
Just catching up on my NULL BITMAPS and this is easily the best intuition for write skew I've ever seen described. The analogy to merge skew in a codebase is genius.
Well well well: www.crunchydata.com/blog/pg_incr...
Incremental pipelines come to Postgres via Crunchy Data! This is like "dbt incremental", not true incremental view maintenance like @materialize.com or Snowflake's dynamic tables, but it's a neat step towards IVM.
Incremental pipelines come to Postgres via Crunchy Data! This is like "dbt incremental", not true incremental view maintenance like @materialize.com or Snowflake's dynamic tables, but it's a neat step towards IVM.
December 19, 2024 at 5:08 AM
Well well well: www.crunchydata.com/blog/pg_incr...
Incremental pipelines come to Postgres via Crunchy Data! This is like "dbt incremental", not true incremental view maintenance like @materialize.com or Snowflake's dynamic tables, but it's a neat step towards IVM.
Incremental pipelines come to Postgres via Crunchy Data! This is like "dbt incremental", not true incremental view maintenance like @materialize.com or Snowflake's dynamic tables, but it's a neat step towards IVM.
Reposted by Nikhil Benesch
The 6th (and last of 2024!) NYC Systems talks are next Thursday! We've got @jaronoff.com of Omlet and @benesch.bsky.social of Materialize speaking. :)
nycsystems.xyz/december-202...
nycsystems.xyz/december-202...
December 12, 2024 at 7:54 PM
The 6th (and last of 2024!) NYC Systems talks are next Thursday! We've got @jaronoff.com of Omlet and @benesch.bsky.social of Materialize speaking. :)
nycsystems.xyz/december-202...
nycsystems.xyz/december-202...
Reposted by Nikhil Benesch
🚀 Postgres Compatibility Index (PCI): Think your shiny new Postgres derivative is "Postgres-compatible"? Test it, score it, and know the truth. 🐘
Thanks @gunnarmorling.dev @tudor.golubenco.org @benesch.bsky.social @deverts.bsky.social for the inspiration.
Blog ==> tinyurl.com/mmfcbczz
Thanks @gunnarmorling.dev @tudor.golubenco.org @benesch.bsky.social @deverts.bsky.social for the inspiration.
Blog ==> tinyurl.com/mmfcbczz
December 10, 2024 at 2:16 AM
🚀 Postgres Compatibility Index (PCI): Think your shiny new Postgres derivative is "Postgres-compatible"? Test it, score it, and know the truth. 🐘
Thanks @gunnarmorling.dev @tudor.golubenco.org @benesch.bsky.social @deverts.bsky.social for the inspiration.
Blog ==> tinyurl.com/mmfcbczz
Thanks @gunnarmorling.dev @tudor.golubenco.org @benesch.bsky.social @deverts.bsky.social for the inspiration.
Blog ==> tinyurl.com/mmfcbczz
Reposted by Nikhil Benesch
First new post in a couple of weeks! There's been a lot of activity around regattastorage.com this week, so I decided to write about the space. tl;dr It's pretty exciting!
The Quest for a Distributed POSIX-Compatible Filesystem
Distributed POSIX filesystems have proven elusive, but we're getting closer. Perhaps that's all we need.
materializedview.io
December 5, 2024 at 8:36 PM
First new post in a couple of weeks! There's been a lot of activity around regattastorage.com this week, so I decided to write about the space. tl;dr It's pretty exciting!
Reposted by Nikhil Benesch
Today's third blog post on Aurora DSQL: this time looking at the write and commit path, and how conflicts are detected and handled: brooker.co.za/blog/2024/12...
DSQL Vignette: Transactions and Durability - Marc's Blog
brooker.co.za
December 5, 2024 at 6:08 PM
Today's third blog post on Aurora DSQL: this time looking at the write and commit path, and how conflicts are detected and handled: brooker.co.za/blog/2024/12...
Reposted by Nikhil Benesch
For anyone keeping score in the open table format wars, AWS launching their s3 table bucket with only Iceberg support seems like a really big deal. Nothing is named in such a way that they can't add support for Delta down the line, but this feels like a huge endorsements of Iceberg (vs. others)
December 4, 2024 at 1:32 AM
For anyone keeping score in the open table format wars, AWS launching their s3 table bucket with only Iceberg support seems like a really big deal. Nothing is named in such a way that they can't add support for Delta down the line, but this feels like a huge endorsements of Iceberg (vs. others)
Reposted by Nikhil Benesch
My second blog post on Aurora DSQL, with a deep dive on how reads work: brooker.co.za/blog/2024/12...
DSQL Vignette: Reads and Compute - Marc's Blog
brooker.co.za
December 4, 2024 at 5:11 PM
My second blog post on Aurora DSQL, with a deep dive on how reads work: brooker.co.za/blog/2024/12...
S3 (Iceberg) Tables is everything I dreamt of, and more. I blogged some long-form thoughts: meltware.com/2024/12/04/s...
I think we're about to see an explosion of data tools (@materialize.com, @clickhouse.com, @duckdb.org, et al.) learn to write Iceberg tables via S3 table buckets.
#databs
I think we're about to see an explosion of data tools (@materialize.com, @clickhouse.com, @duckdb.org, et al.) learn to write Iceberg tables via S3 table buckets.
#databs
A First Look at S3 (Iceberg) Tables
AWS announced S3 Tables today, which brings native support for Apache Iceberg to S3. It’s hard to overstate how exciting this is for the data analytics ecosystem. This post is a quick rundown of my th...
meltware.com
December 4, 2024 at 10:02 AM
S3 (Iceberg) Tables is everything I dreamt of, and more. I blogged some long-form thoughts: meltware.com/2024/12/04/s...
I think we're about to see an explosion of data tools (@materialize.com, @clickhouse.com, @duckdb.org, et al.) learn to write Iceberg tables via S3 table buckets.
#databs
I think we're about to see an explosion of data tools (@materialize.com, @clickhouse.com, @duckdb.org, et al.) learn to write Iceberg tables via S3 table buckets.
#databs
Reposted by Nikhil Benesch
Aurora DSQL uses most of the SQL layer (parser, planner, optimizer, etc) from PostgreSQL itself. The storage layer, concurrency control, replication, etc are ground-up.
December 3, 2024 at 8:18 PM
Aurora DSQL uses most of the SQL layer (parser, planner, optimizer, etc) from PostgreSQL itself. The storage layer, concurrency control, replication, etc are ground-up.
GCP is now the only major cloud provider without a serverless Postgres offering.
As of today AWS has DSQL: aws.amazon.com/about-aws/wh...
Azure partnered with Neon in August: neon.tech/blog/neon-is...
As of today AWS has DSQL: aws.amazon.com/about-aws/wh...
Azure partnered with Neon in August: neon.tech/blog/neon-is...
Announcing Amazon Aurora DSQL (Preview) - AWS
Discover more about what's new at AWS with Announcing Amazon Aurora DSQL (Preview)
aws.amazon.com
December 3, 2024 at 5:57 PM
GCP is now the only major cloud provider without a serverless Postgres offering.
As of today AWS has DSQL: aws.amazon.com/about-aws/wh...
Azure partnered with Neon in August: neon.tech/blog/neon-is...
As of today AWS has DSQL: aws.amazon.com/about-aws/wh...
Azure partnered with Neon in August: neon.tech/blog/neon-is...
Reposted by Nikhil Benesch
IT'S HERE! Amazon Aurora Distributed SQL (DSQL). It's DynamoDB and Postgres merged into a truly serverless relational database.
This is THE announcement of Re:Invent 2024. Everything else is irrelevant.
This is THE announcement of Re:Invent 2024. Everything else is irrelevant.
Announcing Amazon Aurora DSQL (Preview)
AWS introduces Amazon Aurora DSQL, a serverless, distributed SQL database offering unlimited scalability, high availability, and zero infrastructure management...
buff.ly
December 3, 2024 at 5:20 PM
IT'S HERE! Amazon Aurora Distributed SQL (DSQL). It's DynamoDB and Postgres merged into a truly serverless relational database.
This is THE announcement of Re:Invent 2024. Everything else is irrelevant.
This is THE announcement of Re:Invent 2024. Everything else is irrelevant.
S3 gets *native* Iceberg support. 🤯
From the AWS blog: aws.amazon.com/blogs/aws/ne...
We knew something like this was coming (bsky.app/profile/bene...), but it's an even deeper integration than I expected.
From the AWS blog: aws.amazon.com/blogs/aws/ne...
We knew something like this was coming (bsky.app/profile/bene...), but it's an even deeper integration than I expected.
December 3, 2024 at 5:49 PM
S3 gets *native* Iceberg support. 🤯
From the AWS blog: aws.amazon.com/blogs/aws/ne...
We knew something like this was coming (bsky.app/profile/bene...), but it's an even deeper integration than I expected.
From the AWS blog: aws.amazon.com/blogs/aws/ne...
We knew something like this was coming (bsky.app/profile/bene...), but it's an even deeper integration than I expected.
@glaubercosta.bsky.social nails an explanation of 1) why vector databases are mostly a feature not a product but 2) why nonetheless @turbopuffer.bsky.social is on to something.
From a Software Huddle interview a few months back: www.youtube.com/watch?v=a4X7...
From a Software Huddle interview a few months back: www.youtube.com/watch?v=a4X7...
SQLite, Turso, and the State of Databases with Glauber Costa
YouTube video by Software Huddle
www.youtube.com
December 1, 2024 at 8:39 PM
@glaubercosta.bsky.social nails an explanation of 1) why vector databases are mostly a feature not a product but 2) why nonetheless @turbopuffer.bsky.social is on to something.
From a Software Huddle interview a few months back: www.youtube.com/watch?v=a4X7...
From a Software Huddle interview a few months back: www.youtube.com/watch?v=a4X7...
@fasterthanli.me what's the performance of loona like? 👀
I'm in the market for a faster S3 client in Rust and loona is the first plausible project I surfaced. bsky.app/profile/bene...
I'm in the market for a faster S3 client in Rust and loona is the first plausible project I surfaced. bsky.app/profile/bene...
Does it use io_uring? All I want for Christmas is an S3 client that uses io_uring.
November 28, 2024 at 4:50 AM
@fasterthanli.me what's the performance of loona like? 👀
I'm in the market for a faster S3 client in Rust and loona is the first plausible project I surfaced. bsky.app/profile/bene...
I'm in the market for a faster S3 client in Rust and loona is the first plausible project I surfaced. bsky.app/profile/bene...
Awesome work from @antiguru.bsky.social that can cut @materialize.com memory use by 50% on a common class of workload.
Most importantly for our customers: Materialize is now 50% cheaper on these workloads!
Most importantly for our customers: Materialize is now 50% cheaper on these workloads!
Sometimes it's easier to mitigate a problem than solve it: a feature I recently implemented reduces the memory and CPU @materialize.com requires to maintain future updates. Some details why solving the problem is hard and the mitigation easy-ish.
materialize.com/changelog/20...
materialize.com/changelog/20...
Improved performance for temporal filters
Optimization time! We've significantly reduced the cost of using temporal filters in Materialize.
materialize.com
November 27, 2024 at 6:37 PM
Awesome work from @antiguru.bsky.social that can cut @materialize.com memory use by 50% on a common class of workload.
Most importantly for our customers: Materialize is now 50% cheaper on these workloads!
Most importantly for our customers: Materialize is now 50% cheaper on these workloads!
Somehow Starlark inherited all the same Unicode problems that Python 2 had. 🙃
github.com/tidbyt/pixle... cost me more time than I would like to admit.
github.com/tidbyt/pixle... cost me more time than I would like to admit.
Add a .bytes() method to HTTP responses by benesch · Pull Request #1095 · tidbyt/pixlet
The new .bytes() method returns the body as bytes rather than a string. It is modeled after the .json() method, which parses the body as JSON.
The use case is parsing HTTP requests that return bina...
github.com
November 27, 2024 at 4:55 AM
Somehow Starlark inherited all the same Unicode problems that Python 2 had. 🙃
github.com/tidbyt/pixle... cost me more time than I would like to admit.
github.com/tidbyt/pixle... cost me more time than I would like to admit.