Jepsen
@jepsen.mastodon.jepsen.io.ap.brid.gy
Breaking distributed systems, one fault at a time.
[bridged from https://mastodon.jepsen.io/@jepsen on the fediverse by https://fed.brid.gy/ ]
[bridged from https://mastodon.jepsen.io/@jepsen on the fediverse by https://fed.brid.gy/ ]
The latest Jepsen talk, from Systems Distributed in June, goes live in 15 minutes. We'll be doing a live chat during the premier, if you want to chat about databases and testing. :-)
https://www.youtube.com/watch?v=dpTxWePmW5Y
https://www.youtube.com/watch?v=dpTxWePmW5Y
August 18, 2025 at 4:50 PM
The latest Jepsen talk, from Systems Distributed in June, goes live in 15 minutes. We'll be doing a live chat during the premier, if you want to chat about databases and testing. :-)
https://www.youtube.com/watch?v=dpTxWePmW5Y
https://www.youtube.com/watch?v=dpTxWePmW5Y
A new #Jepsen report! We tested early builds of Capela, an unreleased distributed programming environment, and found twenty-two issues, including four language problems, fourteen crashes or non-fatal panics, performance degradation, and three safety issues including lost update […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
August 7, 2025 at 2:46 PM
A new #Jepsen report! We tested early builds of Capela, an unreleased distributed programming environment, and found twenty-two issues, including four language problems, fourteen crashes or non-fatal panics, performance degradation, and three safety issues including lost update […]
An interview with Kaivalya Apte, on The GeekNarrator podcast. We talk about mapping properties to tests, type I and II errors, performance, LLMs, and more.
https://www.youtube.com/watch?v=IvE1VbOol88
https://www.youtube.com/watch?v=IvE1VbOol88
August 6, 2025 at 3:18 PM
An interview with Kaivalya Apte, on The GeekNarrator podcast. We talk about mapping properties to tests, type I and II errors, performance, LLMs, and more.
https://www.youtube.com/watch?v=IvE1VbOol88
https://www.youtube.com/watch?v=IvE1VbOol88
The video of my BugBash talk, "Jepsen 17: ACID Jazz" is out now! https://www.youtube.com/watch?v=v8cG2hh10SQ
August 5, 2025 at 5:55 PM
The video of my BugBash talk, "Jepsen 17: ACID Jazz" is out now! https://www.youtube.com/watch?v=v8cG2hh10SQ
Antithesis and Jepsen are releasing a glossary of terms useful in distributed systems testing: https://antithesis.com/resources/reliability_glossary/
A distributed systems reliability glossary
A list of key concepts for building and testing reliable distributed systems, with basic definitions and deep references.
antithesis.com
July 15, 2025 at 2:45 PM
Antithesis and Jepsen are releasing a glossary of terms useful in distributed systems testing: https://antithesis.com/resources/reliability_glossary/
June 7, 2025 at 3:54 PM
A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
June 6, 2025 at 10:54 AM
A new #Jepsen report! We worked with TigerBeetle to find seven crashes, elevated latencies during single-node failures, and requests which were retried forever in version 0.16.11. We found only two safety issues: missing results for queries with multiple predicates, and incorrect timestamps in a […]
Jepsen 0.3.9 is now available, including a module for restarting flaky databases, a few improvements to downloading logs, more capable generators, and friendlier error messages. https://github.com/jepsen-io/jepsen/releases/tag/v0.3.9
Release 0.3.9 · jepsen-io/jepsen
A medium-sized release, this version makes several quality-of-life improvements. We (finally!) download log files only once, rather than twice at the end of a run; this should make multi-GB log dow...
github.com
May 7, 2025 at 1:55 PM
Jepsen 0.3.9 is now available, including a module for restarting flaky databases, a few improvements to downloading logs, more capable generators, and friendlier error messages. https://github.com/jepsen-io/jepsen/releases/tag/v0.3.9
A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
April 29, 2025 at 2:31 PM
A small issue in Amazon RDS for PostgreSQL: at the "Repeatable Read” isolation level, which in PostgreSQL normally means Snapshot Isolation, Amazon RDS for PostgreSQL clusters appear to exhibit Long Fork. We observed this behavior in healthy clusters, in versions ranging from 13.15 to 17.4 […]
Added four new phenomena to Jepsen's docs: P4 (Lost Update), A5A (Read Skew), A5B (Write Skew) and Process.
https://jepsen.io/consistency/phenomena#sql
https://jepsen.io/consistency/phenomena#sql
March 30, 2025 at 8:55 PM
Added four new phenomena to Jepsen's docs: P4 (Lost Update), A5A (Read Skew), A5B (Write Skew) and Process.
https://jepsen.io/consistency/phenomena#sql
https://jepsen.io/consistency/phenomena#sql
Jepsen 0.3.8 is now available. It includes a new nemesis for file corruption, and improvements to clock-skew tests. https://github.com/jepsen-io/jepsen/releases/tag/v0.3.8
Release 0.3.8 · jepsen-io/jepsen
The centerpiece of 0.3.8 is a new nemesis for corrupting files: jepsen.nemesis.file. This nemesis can be scoped to specific regions of a file, is aware of chunk structure (e.g. database pages) and ...
github.com
March 6, 2025 at 11:44 PM
Jepsen 0.3.8 is now available. It includes a new nemesis for file corruption, and improvements to clock-skew tests. https://github.com/jepsen-io/jepsen/releases/tag/v0.3.8
What IS Strong Serializability, really? Ever want to try writing your own gossip service. Two open sessions of Jepsen's training classes are coming up: the Distributed Systems Fundamentals class, and (for the first time!) its accompanying workshop […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
March 5, 2025 at 2:44 AM
What IS Strong Serializability, really? Ever want to try writing your own gossip service. Two open sessions of Jepsen's training classes are coming up: the Distributed Systems Fundamentals class, and (for the first time!) its accompanying workshop […]
I'll be speaking on Jepsen at Bug Bash (DC, April 3-4), and Systems Distributed (Amsterdam, June 19-20). Come join!
https://bugbash.antithesis.com/
https://systemsdistributed.com/
https://bugbash.antithesis.com/
https://systemsdistributed.com/
BugBash 2025: software reliability conference
Join us at a tech conference organized by Antithesis to explore reliable software development. April 3-4, 2024 at Yours Truly DC hotel, Washington D.C.
bugbash.antithesis.com
March 5, 2025 at 12:13 AM
I'll be speaking on Jepsen at Bug Bash (DC, April 3-4), and Systems Distributed (Amsterdam, June 19-20). Come join!
https://bugbash.antithesis.com/
https://systemsdistributed.com/
https://bugbash.antithesis.com/
https://systemsdistributed.com/
So, uh, the last time I had to call malloc() was a quarter century ago, and I am struggling to do basic tasks without corrupting the heap. I would love to hire one of you stripey-socked C witches for a very small contract to help finish […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
January 21, 2025 at 11:36 PM
So, uh, the last time I had to call malloc() was a quarter century ago, and I am struggling to do basic tasks without corrupting the heap. I would love to hire one of you stripey-socked C witches for a very small contract to help finish […]
Added descriptions of the SQL isolation level anomalies P0, P1, P2, and P3 to the phenomena page: https://jepsen.io/consistency/phenomena#sql
December 22, 2024 at 9:03 PM
Added descriptions of the SQL isolation level anomalies P0, P1, P2, and P3 to the phenomena page: https://jepsen.io/consistency/phenomena#sql
Released version 0.2.4 of Maelstrom, Jepsen's workbench for writing toy distributed systems: https://github.com/jepsen-io/maelstrom
GitHub - jepsen-io/maelstrom: A workbench for writing toy implementations of distributed systems.
A workbench for writing toy implementations of distributed systems. - jepsen-io/maelstrom
github.com
December 4, 2024 at 7:16 PM
Released version 0.2.4 of Maelstrom, Jepsen's workbench for writing toy distributed systems: https://github.com/jepsen-io/maelstrom
There's a few tickets left for the distributed systems class coming up in just over a week. If you'd like to join, now's the time. :-)
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
Distributed Systems Fundamentals
Learn the theory and practice behind distributed systems, from safety and liveness to deployment and monitoring.
www.eventbrite.com
December 4, 2024 at 6:30 PM
There's a few tickets left for the distributed systems class coming up in just over a week. If you'd like to join, now's the time. :-)
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
Bin Wang put together a Jepsen test for Patroni, a PostgreSQL replication system. All sorts of good stuff in here, including that the cluster can't handle a series of single-node failures: https://www.binwang.me/2024-12-02-PostgreSQL-High-Availability-Solutions-Part-1.html
Jepsen Test on Patroni: A PostgreSQL High Availability Solution | Bin Wang - My Personal Blog
www.binwang.me
December 3, 2024 at 10:30 PM
Bin Wang put together a Jepsen test for Patroni, a PostgreSQL replication system. All sorts of good stuff in here, including that the cluster can't handle a series of single-node failures: https://www.binwang.me/2024-12-02-PostgreSQL-High-Availability-Solutions-Part-1.html
Antithesis, Buf, and Jepsen are running a joint webinar on December 5th. We'll discuss a Kafka protocol safety issue, talk about the challenges of distributed systems testing, and show how Jepsen and Antithesis helped identify critical safety errors in Bufstream. Come watch Antithesis pause […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
November 27, 2024 at 4:08 PM
Antithesis, Buf, and Jepsen are running a joint webinar on December 5th. We'll discuss a Kafka protocol safety issue, talk about the challenges of distributed systems testing, and show how Jepsen and Antithesis helped identify critical safety errors in Bufstream. Come watch Antithesis pause […]
"That's never been my understanding of auto-commit, that would be a crazy default wouldn't it?"
I feel like there's a real need for database user research--building a quantitative and qualitative picture of how actual users understand the systems they use […]
I feel like there's a real need for database user research--building a quantitative and qualitative picture of how actual users understand the systems they use […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
November 12, 2024 at 3:50 PM
"That's never been my understanding of auto-commit, that would be a crazy default wouldn't it?"
I feel like there's a real need for database user research--building a quantitative and qualitative picture of how actual users understand the systems they use […]
I feel like there's a real need for database user research--building a quantitative and qualitative picture of how actual users understand the systems they use […]
A new #Jepsen report! We worked with Buf to analyze the safety of Bufstream, a Kafka-compatible streaming system. We found three safety and two liveness issues in Bufstream 0.1.0, including the loss of acknowledged writes in healthy clusters. These problems were resolved by version 0.1.3 […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
November 12, 2024 at 2:15 PM
A new #Jepsen report! We worked with Buf to analyze the safety of Bufstream, a Kafka-compatible streaming system. We found three safety and two liveness issues in Bufstream 0.1.0, including the loss of acknowledged writes in healthy clusters. These problems were resolved by version 0.1.3 […]
Last year we reported that MySQL and MariaDB's "Repeatable Read" was badly broken. The MariaDB team has been hard at work, and they've added a new flag, `--innodb-snapshot-isolation=true`, which causes "Repeatable Read" to prevent Lost Update, Non-repeatable Read, and violations of Monotonic […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
November 7, 2024 at 3:43 PM
Last year we reported that MySQL and MariaDB's "Repeatable Read" was badly broken. The MariaDB team has been hard at work, and they've added a new flag, `--innodb-snapshot-isolation=true`, which causes "Repeatable Read" to prevent Lost Update, Non-repeatable Read, and violations of Monotonic […]
If you've been waiting for a chance to take Jepsen's distributed systems class, we've got great news. We're running a special session December 16-19 that anyone can register for.
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
October 25, 2024 at 11:20 PM
If you've been waiting for a chance to take Jepsen's distributed systems class, we've got great news. We're running a special session December 16-19 that anyone can register for.
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
https://www.eventbrite.com/e/distributed-systems-fundamentals-registration-1060426286569?aff=mastodon
Just want to give a shout out to Irene Kannyo, who's edited every Jepsen analysis over the last four years. She brings her background as scientist and writer to our industry-focused reports, maintains an evolving style guide, streamlines prose, ensures diagrams match descriptions, and so much […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
October 24, 2024 at 3:50 PM
Just want to give a shout out to Irene Kannyo, who's edited every Jepsen analysis over the last four years. She brings her background as scientist and writer to our industry-focused reports, maintains an evolving style guide, streamlines prose, ensures diagrams match descriptions, and so much […]
#Jepsen 0.3.6 is now available on GitHub and Clojars. This is a sizeable release. It includes a significant correctness bugfix for a rare bug that could make operations in the history print with the wrong data. It also adds a new namespace for composing databases, nemeses, and generators when […]
Original post on mastodon.jepsen.io
mastodon.jepsen.io
October 18, 2024 at 7:16 PM
#Jepsen 0.3.6 is now available on GitHub and Clojars. This is a sizeable release. It includes a significant correctness bugfix for a rare bug that could make operations in the history print with the wrong data. It also adds a new namespace for composing databases, nemeses, and generators when […]