Andrey Satarin
@asatarin.bsky.social
Staff SRE at Google. Distributed systems / databases / reliability / correctness. Views my own. Repost / like is not an endorsement.
http://asatarin.github.io
http://asatarin.github.io
Pinned
Andrey Satarin
@asatarin.bsky.social
· Nov 19
Testing Distributed Systems
Curated list of resources on testing distributed systems
asatarin.github.io
Hello, Bluesky users!
I curate and maintain list of resources on testing distributed systems. You might have seen it before. It's a good one, if I may say so myself.
asatarin.github.io/testing-dist...
I curate and maintain list of resources on testing distributed systems. You might have seen it before. It's a good one, if I may say so myself.
asatarin.github.io/testing-dist...
Reposted by Andrey Satarin
#SD25 online continues today!
Join Kyle Kingsbury for Jepsen 18: Serializable Mom
How three unconventional systems ensure--or violate--key safety properties
10am PT / 1pm ET / 7pm CET
www.youtube.com/watch?v=dpTx...
@jepsen.mastodon.jepsen.io.ap.brid.gy
Join Kyle Kingsbury for Jepsen 18: Serializable Mom
How three unconventional systems ensure--or violate--key safety properties
10am PT / 1pm ET / 7pm CET
www.youtube.com/watch?v=dpTx...
@jepsen.mastodon.jepsen.io.ap.brid.gy
Jepsen 18: Serializable Mom by Kyle Kingsbury
YouTube video by TigerBeetle
www.youtube.com
August 18, 2025 at 6:10 AM
#SD25 online continues today!
Join Kyle Kingsbury for Jepsen 18: Serializable Mom
How three unconventional systems ensure--or violate--key safety properties
10am PT / 1pm ET / 7pm CET
www.youtube.com/watch?v=dpTx...
@jepsen.mastodon.jepsen.io.ap.brid.gy
Join Kyle Kingsbury for Jepsen 18: Serializable Mom
How three unconventional systems ensure--or violate--key safety properties
10am PT / 1pm ET / 7pm CET
www.youtube.com/watch?v=dpTx...
@jepsen.mastodon.jepsen.io.ap.brid.gy
Reposted by Andrey Satarin
-------
"Jepsen 18: Serializable Mom", Kyle Kingsbury/@aphyr.com , #sd25
Engineers are tasked with building towers of abstraction, building everything higher and higher above the towering tire fire that is databases.
"I professionally set those tires on fire".
"Jepsen 18: Serializable Mom", Kyle Kingsbury/@aphyr.com , #sd25
Engineers are tasked with building towers of abstraction, building everything higher and higher above the towering tire fire that is databases.
"I professionally set those tires on fire".
June 20, 2025 at 1:22 PM
-------
"Jepsen 18: Serializable Mom", Kyle Kingsbury/@aphyr.com , #sd25
Engineers are tasked with building towers of abstraction, building everything higher and higher above the towering tire fire that is databases.
"I professionally set those tires on fire".
"Jepsen 18: Serializable Mom", Kyle Kingsbury/@aphyr.com , #sd25
Engineers are tasked with building towers of abstraction, building everything higher and higher above the towering tire fire that is databases.
"I professionally set those tires on fire".
Reposted by Andrey Satarin
It's day two of Systems Distributed, hosted by @tigerbeetle.com! I'll be liveskeeting all of the talks, except mine (at 11 AM). Since the venue is a film museum, they're setting up special posters for each talk.
#sd25
#sd25
June 20, 2025 at 6:36 AM
It's day two of Systems Distributed, hosted by @tigerbeetle.com! I'll be liveskeeting all of the talks, except mine (at 11 AM). Since the venue is a film museum, they're setting up special posters for each talk.
#sd25
#sd25
Reposted by Andrey Satarin
Greetings from #sd25! It's a two-day single-track conference hosted by
@tigerbeetle.com. I'll be aiming to liveskeet as many of the talks as I can. Super excited to be here!
systemsdistributed.com
@tigerbeetle.com. I'll be aiming to liveskeet as many of the talks as I can. Super excited to be here!
systemsdistributed.com
Systems Distributed '25
A conference to teach systems programming and thinking, and how to apply these ideas. All the way across the stack. From systems languages and compilers, to databases and distributed systems.
systemsdistributed.com
June 19, 2025 at 6:25 AM
Greetings from #sd25! It's a two-day single-track conference hosted by
@tigerbeetle.com. I'll be aiming to liveskeet as many of the talks as I can. Super excited to be here!
systemsdistributed.com
@tigerbeetle.com. I'll be aiming to liveskeet as many of the talks as I can. Super excited to be here!
systemsdistributed.com
Reposted by Andrey Satarin
A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
June 6, 2025 at 10:54 AM
A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]
Reposted by Andrey Satarin
A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
April 29, 2025 at 2:31 PM
A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]
Reposted by Andrey Satarin
All the videos are up at www.hytradboi.com/2025#program now.
HYTRADBOI 2025
HYTRADBOI is a fun online conference about databases, programming languages, and everything in between.
www.hytradboi.com
March 1, 2025 at 6:17 AM
All the videos are up at www.hytradboi.com/2025#program now.
"Correctness at Feldera" talks about various correctness techniques, including:
- machine proof of the underlying DBSP algorithm
- differential testing of the implementation
https://www.feldera.com/blog/correctness-at-feldera
- machine proof of the underlying DBSP algorithm
- differential testing of the implementation
https://www.feldera.com/blog/correctness-at-feldera
Correctness at Feldera
In this blog post, we briefly describe our efforts and development processes that ensure Feldera's engine is correct.
www.feldera.com
January 13, 2025 at 4:49 PM
"Correctness at Feldera" talks about various correctness techniques, including:
- machine proof of the underlying DBSP algorithm
- differential testing of the implementation
https://www.feldera.com/blog/correctness-at-feldera
- machine proof of the underlying DBSP algorithm
- differential testing of the implementation
https://www.feldera.com/blog/correctness-at-feldera
Reposted by Andrey Satarin
Buckle up because we're banging into the new year with my annual retrospective of the last year in databases! Highlights include license change blowback, Databricks vs. Snowflake gangwar, @duckdb.org's shotgun weddings, and buying a quarterback to impress your lover: www.cs.cmu.edu/~pavlo/blog/...
Databases in 2024: A Year in Review
Andy rises from the ashes of his dead startup and discusses what happened in 2024 in the database game.
www.cs.cmu.edu
January 1, 2025 at 2:02 PM
Buckle up because we're banging into the new year with my annual retrospective of the last year in databases! Highlights include license change blowback, Databricks vs. Snowflake gangwar, @duckdb.org's shotgun weddings, and buying a quarterback to impress your lover: www.cs.cmu.edu/~pavlo/blog/...
Reposted by Andrey Satarin
Reposted by Andrey Satarin
This is a sample from a list "testing-sql-databases" I have.
It's more drafty, most likely missing a ton from big tech, startups and academia alike and not published.
It's more drafty, most likely missing a ton from big tech, startups and academia alike and not published.
December 12, 2024 at 6:35 AM
This is a sample from a list "testing-sql-databases" I have.
It's more drafty, most likely missing a ton from big tech, startups and academia alike and not published.
It's more drafty, most likely missing a ton from big tech, startups and academia alike and not published.
Reposted by Andrey Satarin
Good old Microsoft published some work:
- "Deploying a Steered Query Optimizer in Production at Microsoft" dl.acm.org/doi/abs/10.1...
- This great talk "The Cascades Framework for Query Optimization at Microsoft" touches on correctness youtu.be/pQe1LQJiXN0
- "Deploying a Steered Query Optimizer in Production at Microsoft" dl.acm.org/doi/abs/10.1...
- This great talk "The Cascades Framework for Query Optimization at Microsoft" touches on correctness youtu.be/pQe1LQJiXN0
Deploying a Steered Query Optimizer in Production at Microsoft | Proceedings of the 2022 International Conference on Management of Data
You will be notified whenever a record that you have chosen has been cited.
dl.acm.org
December 12, 2024 at 6:35 AM
Good old Microsoft published some work:
- "Deploying a Steered Query Optimizer in Production at Microsoft" dl.acm.org/doi/abs/10.1...
- This great talk "The Cascades Framework for Query Optimization at Microsoft" touches on correctness youtu.be/pQe1LQJiXN0
- "Deploying a Steered Query Optimizer in Production at Microsoft" dl.acm.org/doi/abs/10.1...
- This great talk "The Cascades Framework for Query Optimization at Microsoft" touches on correctness youtu.be/pQe1LQJiXN0
Reposted by Andrey Satarin
Work on correctness of optimizers from Greenplum
- "Automatic capture of minimal, portable, and executable bug repros using AMPERe" dl.acm.org/doi/10.1145/...
- "Testing the accuracy of query optimizers" dl.acm.org/doi/10.1145/...
- "Automatic capture of minimal, portable, and executable bug repros using AMPERe" dl.acm.org/doi/10.1145/...
- "Testing the accuracy of query optimizers" dl.acm.org/doi/10.1145/...
Automatic capture of minimal, portable, and executable bug repros using AMPERe | Proceedings of the Fifth International Workshop on Testing Database Systems
You will be notified whenever a record that you have chosen has been cited.
dl.acm.org
December 12, 2024 at 6:35 AM
Work on correctness of optimizers from Greenplum
- "Automatic capture of minimal, portable, and executable bug repros using AMPERe" dl.acm.org/doi/10.1145/...
- "Testing the accuracy of query optimizers" dl.acm.org/doi/10.1145/...
- "Automatic capture of minimal, portable, and executable bug repros using AMPERe" dl.acm.org/doi/10.1145/...
- "Testing the accuracy of query optimizers" dl.acm.org/doi/10.1145/...
Reposted by Andrey Satarin
Similar work from Databricks:
- "SparkFuzz: searching correctness regressions in modern query engines" dl.acm.org/doi/abs/10.1...
- "Correctness and Performance of Apache Spark SQL" youtu.be/fddBOZxdUKI
- "SparkFuzz: searching correctness regressions in modern query engines" dl.acm.org/doi/abs/10.1...
- "Correctness and Performance of Apache Spark SQL" youtu.be/fddBOZxdUKI
SparkFuzz | Proceedings of the workshop on Testing Database Systems
dl.acm.org
December 12, 2024 at 6:35 AM
Similar work from Databricks:
- "SparkFuzz: searching correctness regressions in modern query engines" dl.acm.org/doi/abs/10.1...
- "Correctness and Performance of Apache Spark SQL" youtu.be/fddBOZxdUKI
- "SparkFuzz: searching correctness regressions in modern query engines" dl.acm.org/doi/abs/10.1...
- "Correctness and Performance of Apache Spark SQL" youtu.be/fddBOZxdUKI
Reposted by Andrey Satarin
You already mentioned "Snowtrail: Testing with Production Queries on a Cloud Database" from Snowflake
dl.acm.org/doi/10.1145/...
dl.acm.org/doi/10.1145/...
Snowtrail | Proceedings of the Workshop on Testing Database Systems
dl.acm.org
December 12, 2024 at 6:35 AM
You already mentioned "Snowtrail: Testing with Production Queries on a Cloud Database" from Snowflake
dl.acm.org/doi/10.1145/...
dl.acm.org/doi/10.1145/...
Reposted by Andrey Satarin
More correctness work from MongoDB, not yet on my list
"Verifying Transactional Consistency of MongoDB"
arxiv.org/abs/2111.14946
"Verifying Transactional Consistency of MongoDB"
arxiv.org/abs/2111.14946
Verifying Transactional Consistency of MongoDB
MongoDB is a popular general-purpose, document-oriented, distributed NoSQL database. It supports transactions in three different deployments: single-document transactions utilizing the WiredTiger stor...
arxiv.org
December 12, 2024 at 6:35 AM
More correctness work from MongoDB, not yet on my list
"Verifying Transactional Consistency of MongoDB"
arxiv.org/abs/2111.14946
"Verifying Transactional Consistency of MongoDB"
arxiv.org/abs/2111.14946
Reposted by Andrey Satarin
MongoDB team did great work on performance:
- "Creating a Virtuous Cycle in Performance Testing at MongoDB" dl.acm.org/doi/10.1145/...
- "The Use of Change Point Detection ..." dl.acm.org/doi/abs/10.1...
- "Fair Benchmarking Considered Difficult" mytherin.github.io/papers/2018-...
- "Creating a Virtuous Cycle in Performance Testing at MongoDB" dl.acm.org/doi/10.1145/...
- "The Use of Change Point Detection ..." dl.acm.org/doi/abs/10.1...
- "Fair Benchmarking Considered Difficult" mytherin.github.io/papers/2018-...
Creating a Virtuous Cycle in Performance Testing at MongoDB | Proceedings of the ACM/SPEC International Conference on Performance Engineering
dl.acm.org
December 12, 2024 at 6:35 AM
MongoDB team did great work on performance:
- "Creating a Virtuous Cycle in Performance Testing at MongoDB" dl.acm.org/doi/10.1145/...
- "The Use of Change Point Detection ..." dl.acm.org/doi/abs/10.1...
- "Fair Benchmarking Considered Difficult" mytherin.github.io/papers/2018-...
- "Creating a Virtuous Cycle in Performance Testing at MongoDB" dl.acm.org/doi/10.1145/...
- "The Use of Change Point Detection ..." dl.acm.org/doi/abs/10.1...
- "Fair Benchmarking Considered Difficult" mytherin.github.io/papers/2018-...
Reposted by Andrey Satarin
To counter balance your argument some big tech work on correctness of (SQL) database in a thread below.
Spanner has incredibly sophisticated random generated checks internally, this just scratches the surface:
- "Randomized Testing of Cloud Spanner"
medium.com/@jcorbett_26...
Spanner has incredibly sophisticated random generated checks internally, this just scratches the surface:
- "Randomized Testing of Cloud Spanner"
medium.com/@jcorbett_26...
Randomized Testing of Cloud Spanner
One of the secrets behind Cloud Spanner quality is randomized testing. SQL databases like Cloud Spanner have complex APIs. Complete unit…
medium.com
December 12, 2024 at 6:35 AM
To counter balance your argument some big tech work on correctness of (SQL) database in a thread below.
Spanner has incredibly sophisticated random generated checks internally, this just scratches the surface:
- "Randomized Testing of Cloud Spanner"
medium.com/@jcorbett_26...
Spanner has incredibly sophisticated random generated checks internally, this just scratches the surface:
- "Randomized Testing of Cloud Spanner"
medium.com/@jcorbett_26...
Reposted by Andrey Satarin
Get your point, but to be fair, this list is not targeting correctness of databases (SQL or otherwise).
It almost entirely excludes anything single node or targeting single threaded execution (like a fuzzer), with some late additions in
asatarin.github.io/testing-dist...
It almost entirely excludes anything single node or targeting single threaded execution (like a fuzzer), with some late additions in
asatarin.github.io/testing-dist...
Testing Distributed Systems
Curated list of resources on testing distributed systems
asatarin.github.io
December 12, 2024 at 6:35 AM
Get your point, but to be fair, this list is not targeting correctness of databases (SQL or otherwise).
It almost entirely excludes anything single node or targeting single threaded execution (like a fuzzer), with some late additions in
asatarin.github.io/testing-dist...
It almost entirely excludes anything single node or targeting single threaded execution (like a fuzzer), with some late additions in
asatarin.github.io/testing-dist...
Reposted by Andrey Satarin
I'd be happy to learn that this is just a gap in my testing knowledge, or that there's a bunch of secret testing systems in big companies that I don't know about, but even just skimming the list of asatarin.github.io/testing-dist... kind of illustrates my point. (Thanks again @asatarin.bsky.social!)
Testing Distributed Systems
Curated list of resources on testing distributed systems
asatarin.github.io
December 12, 2024 at 1:04 AM
I'd be happy to learn that this is just a gap in my testing knowledge, or that there's a bunch of secret testing systems in big companies that I don't know about, but even just skimming the list of asatarin.github.io/testing-dist... kind of illustrates my point. (Thanks again @asatarin.bsky.social!)
Reposted by Andrey Satarin
Deterministic simulation has been entirely pushed by startups. Jepsen is jepsen. All the SQL fuzzing stuff I know of comes from academia or database startups. Larger companies seem to only be winning in the application of formal methods (because they can afford to hire a team of ex-professors?)
December 12, 2024 at 1:04 AM
Deterministic simulation has been entirely pushed by startups. Jepsen is jepsen. All the SQL fuzzing stuff I know of comes from academia or database startups. Larger companies seem to only be winning in the application of formal methods (because they can afford to hire a team of ex-professors?)
Reposted by Andrey Satarin
When I think of advancements in quality and correctness of databases, it feels most things have come from startups or individuals, and not large well-established companies. Which seems... backwards? We talk of startups hacking out code and megacorps crawling to keep a high quality bar.
December 12, 2024 at 1:04 AM
When I think of advancements in quality and correctness of databases, it feels most things have come from startups or individuals, and not large well-established companies. Which seems... backwards? We talk of startups hacking out code and megacorps crawling to keep a high quality bar.
Reposted by Andrey Satarin
Alexey's @clickhouse.com talk here is hilariously off-the-chain. I highly recommend watching it: www.youtube.com/watch?v=jmVx...
You can hear me yelling at him from the audience at 1:09, 2:40, and 7:33.
You can hear me yelling at him from the audience at 1:09, 2:40, and 7:33.
My Database Can Do This by Alexey Milovidov (DBDBD 2024)
YouTube video by DSDSD - Dutch Seminar on Data Systems Design
www.youtube.com
December 3, 2024 at 2:05 PM
Alexey's @clickhouse.com talk here is hilariously off-the-chain. I highly recommend watching it: www.youtube.com/watch?v=jmVx...
You can hear me yelling at him from the audience at 1:09, 2:40, and 7:33.
You can hear me yelling at him from the audience at 1:09, 2:40, and 7:33.
MariaDB is improving their isolation level guarantees following the recent @jepsen.mastodon.jepsen.io.ap.brid.gy report.
"Isolation level violation testing and debugging in MariaDB" post talks about how they specialize Jepsen tests for InnoDB, make it reproducible and debuggable.
"Isolation level violation testing and debugging in MariaDB" post talks about how they specialize Jepsen tests for InnoDB, make it reproducible and debuggable.
Isolation level violation testing and debugging in MariaDB | MariaDB
Explore how the InnoDB team tackles isolation violations in databases, highlighting debugging strategies and a new option for Snapshot isolation compliance.
mariadb.com
November 25, 2024 at 4:49 PM
MariaDB is improving their isolation level guarantees following the recent @jepsen.mastodon.jepsen.io.ap.brid.gy report.
"Isolation level violation testing and debugging in MariaDB" post talks about how they specialize Jepsen tests for InnoDB, make it reproducible and debuggable.
"Isolation level violation testing and debugging in MariaDB" post talks about how they specialize Jepsen tests for InnoDB, make it reproducible and debuggable.
Alexey Gotsman talks about database isolation levels.
Starts off with consistency in general:
- what is strong consistency
- common misunderstandings and pitfalls
- serializability and linearizability
- weak isolation levels
- safety and liveness
Video links below.
Starts off with consistency in general:
- what is strong consistency
- common misunderstandings and pitfalls
- serializability and linearizability
- weak isolation levels
- safety and liveness
Video links below.
November 24, 2024 at 10:09 PM
Alexey Gotsman talks about database isolation levels.
Starts off with consistency in general:
- what is strong consistency
- common misunderstandings and pitfalls
- serializability and linearizability
- weak isolation levels
- safety and liveness
Video links below.
Starts off with consistency in general:
- what is strong consistency
- common misunderstandings and pitfalls
- serializability and linearizability
- weak isolation levels
- safety and liveness
Video links below.