Tom Cooper
@tomcooper.dev
Senior Principal Engineer at IBM, Ex-Red Hat, Ex-Reddit, Ex-Twitter, Distributed Systems PhD, interested in all things stream processing, also cycling
Reposted by Tom Cooper
Today is the 30th anniversary of Hackers
September 15, 2025 at 11:56 PM
Today is the 30th anniversary of Hackers
A bit late, but the recording of my #StrimziCon2025 talk, Phased Upgrades of Strimzi Managed Kafka Fleets (hows that for a snappy title) is now up on Youtube: youtu.be/pEuYFFkQCUU?...
Phased upgrades of Strimzi managed Kafka fleets
YouTube video by Strimzi
youtu.be
July 7, 2025 at 10:34 AM
A bit late, but the recording of my #StrimziCon2025 talk, Phased Upgrades of Strimzi Managed Kafka Fleets (hows that for a snappy title) is now up on Youtube: youtu.be/pEuYFFkQCUU?...
It's my last day at Red Hat. Tomorrow I & the rest of middleware move to IBM. I wanted to work at RH since I heard about Fedora on the Linux Action Show (RIP) decades ago & I've loved developing in the open with RH. That'll continue after the move & after all, no one got fired for buying IBM...
June 30, 2025 at 4:49 PM
It's my last day at Red Hat. Tomorrow I & the rest of middleware move to IBM. I wanted to work at RH since I heard about Fedora on the Linux Action Show (RIP) decades ago & I've loved developing in the open with RH. That'll continue after the move & after all, no one got fired for buying IBM...
Reposted by Tom Cooper
🐊 Kroxylicious Proposal: "Routing API"
This looks like a very useful addition to @kroxylicious.io (wire-protocol proxy for Apache Kafka): dynamic routing of requests to different upstream clusters, enabling use cases such as union clusters and client-based routing.
👉 github.com/kroxylicious...
This looks like a very useful addition to @kroxylicious.io (wire-protocol proxy for Apache Kafka): dynamic routing of requests to different upstream clusters, enabling use cases such as union clusters and client-based routing.
👉 github.com/kroxylicious...
June 14, 2025 at 9:26 AM
🐊 Kroxylicious Proposal: "Routing API"
This looks like a very useful addition to @kroxylicious.io (wire-protocol proxy for Apache Kafka): dynamic routing of requests to different upstream clusters, enabling use cases such as union clusters and client-based routing.
👉 github.com/kroxylicious...
This looks like a very useful addition to @kroxylicious.io (wire-protocol proxy for Apache Kafka): dynamic routing of requests to different upstream clusters, enabling use cases such as union clusters and client-based routing.
👉 github.com/kroxylicious...
Going from 1 to 13 PB/day 🤯 thanks to moving from a self-managed to a Strimzi-managed Kafka cluster! Great to hear these stories of how Strimzi not only made life easier for infra teams but improved the service they provide!
strimzi.io/blog/2025/06...
strimzi.io/blog/2025/06...
Taming the Kafka Chaos: How Strimzi Helped Us Scale with Confidence
Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations.
strimzi.io
June 9, 2025 at 1:50 PM
Going from 1 to 13 PB/day 🤯 thanks to moving from a self-managed to a Strimzi-managed Kafka cluster! Great to hear these stories of how Strimzi not only made life easier for infra teams but improved the service they provide!
strimzi.io/blog/2025/06...
strimzi.io/blog/2025/06...
Ending #Current25 with a talk on Polymorphic Table Functions in Flink🤘
May 21, 2025 at 1:57 PM
Ending #Current25 with a talk on Polymorphic Table Functions in Flink🤘
Great hallway track at #Current25, lots of ideas around Kafka connect, mirror maker, geo-replication, cluster balancing. Lots to mull over on the train home...
May 21, 2025 at 1:14 PM
Great hallway track at #Current25, lots of ideas around Kafka connect, mirror maker, geo-replication, cluster balancing. Lots to mull over on the train home...
"Vibe-coded database schemas" 😱
May 21, 2025 at 1:11 PM
"Vibe-coded database schemas" 😱
Woooo 🎉 great to see Strimzi maintainers Kate Stanley and Paolo Patierno on stage at #Current25 talking about how Strimzi helps you migrate your K8s Kafka clusters from Zookeeper to KRaft
May 21, 2025 at 12:04 PM
Woooo 🎉 great to see Strimzi maintainers Kate Stanley and Paolo Patierno on stage at #Current25 talking about how Strimzi helps you migrate your K8s Kafka clusters from Zookeeper to KRaft
After a very interesting lunch time at #Current25 speaking to folks from OpenAI about Mirror Maker and what we could do in Strimzi to make things easier, I'm watching the Flink AMA MC'd by @rmetzger.bsky.social
May 21, 2025 at 11:28 AM
After a very interesting lunch time at #Current25 speaking to folks from OpenAI about Mirror Maker and what we could do in Strimzi to make things easier, I'm watching the Flink AMA MC'd by @rmetzger.bsky.social
My ⚡ talk choice at #Current25 is @hpgrahsl.bsky.social and vector stores in Flink
May 21, 2025 at 10:01 AM
My ⚡ talk choice at #Current25 is @hpgrahsl.bsky.social and vector stores in Flink
Onto something a bit more concrete now at #Current25. Improving CDC performance using Kafka, Flink and Iceberg
May 21, 2025 at 9:06 AM
Onto something a bit more concrete now at #Current25. Improving CDC performance using Kafka, Flink and Iceberg
Kicking off day 2 of #Current25 by boarding the AI agent train🤖🚂 with MongoDB 🌿
May 21, 2025 at 8:04 AM
Kicking off day 2 of #Current25 by boarding the AI agent train🤖🚂 with MongoDB 🌿
Final talk of the day at #Current25, Async processing with Flink SQL
May 20, 2025 at 4:31 PM
Final talk of the day at #Current25, Async processing with Flink SQL
Learning about how OpenAI built its stream processing system using Kafka and Flink at #Current25
May 20, 2025 at 3:34 PM
Learning about how OpenAI built its stream processing system using Kafka and Flink at #Current25
Next up at #Current25 Flink 🐿️ but with robots 🤖 (AI agents)!
May 20, 2025 at 2:33 PM
Next up at #Current25 Flink 🐿️ but with robots 🤖 (AI agents)!
Next at #Current25 a panel discussion on the past, present and future of @kafkastreams.bsky.social. I love the background...the likeness is uncanny...
May 20, 2025 at 1:17 PM
Next at #Current25 a panel discussion on the past, present and future of @kafkastreams.bsky.social. I love the background...the likeness is uncanny...
Starting #Current25 with an overview of anomaly detection in Flink SQL. This is from Confluent and AFAICS uses their proprietary ML functionality. But there is work in the upstream to support the same that builds on FLIP-437 which added the model syntax to Flink SQL
cwiki.apache.org/confluence/d...
cwiki.apache.org/confluence/d...
May 20, 2025 at 12:06 PM
Starting #Current25 with an overview of anomaly detection in Flink SQL. This is from Confluent and AFAICS uses their proprietary ML functionality. But there is work in the upstream to support the same that builds on FLIP-437 which added the model syntax to Flink SQL
cwiki.apache.org/confluence/d...
cwiki.apache.org/confluence/d...
Ready to start another Kafka and Flink filled couple of days in London at #Current25
Looking forward to hearing about stream processing with Flink, integrating with open table formats and this newfangled AI thing...
Looking forward to hearing about stream processing with Flink, integrating with open table formats and this newfangled AI thing...
May 20, 2025 at 8:59 AM
Ready to start another Kafka and Flink filled couple of days in London at #Current25
Looking forward to hearing about stream processing with Flink, integrating with open table formats and this newfangled AI thing...
Looking forward to hearing about stream processing with Flink, integrating with open table formats and this newfangled AI thing...
The agenda for StrimziCon 2025 is out and yours-truly is 2nd on the bill! I'll be presenting my recent work on phased upgrades of Strimzi managed Kafka fleets.
There are many other great talks on their so check out the full agenda: strimzi.io/blog/2025/04...
There are many other great talks on their so check out the full agenda: strimzi.io/blog/2025/04...
StrimziCon 2025: Schedule announced!
Strimzi provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations.
strimzi.io
April 16, 2025 at 12:25 PM
The agenda for StrimziCon 2025 is out and yours-truly is 2nd on the bill! I'll be presenting my recent work on phased upgrades of Strimzi managed Kafka fleets.
There are many other great talks on their so check out the full agenda: strimzi.io/blog/2025/04...
There are many other great talks on their so check out the full agenda: strimzi.io/blog/2025/04...
If you're running Strimzi in production, you'll probably come up against the issue of wanting to upgrade the operator but not wanting to roll all its Kafka clusters right away. As this comes up a lot, I wrote a blog post with an example of how you could achieve this: strimzi.io/blog/2025/04...
strimzi.io
April 10, 2025 at 12:23 PM
If you're running Strimzi in production, you'll probably come up against the issue of wanting to upgrade the operator but not wanting to roll all its Kafka clusters right away. As this comes up a lot, I wrote a blog post with an example of how you could achieve this: strimzi.io/blog/2025/04...
Apache Flink 2.0.0 is out, featuring disaggregated state management (object storage), materialized tables unifying stream/batch processing, enhanced batch execution, deeper Paimon (Open table format) integration for creating streaming lakehouses and AI model support: flink.apache.org/2025/03/24/a...
Apache Flink 2.0.0: A new Era of Real-Time Data Processing
Today, the Flink PMC is proud to announce the official release of Apache Flink 2.0.0! This marks the first release in the Flink 2.x series and is the first major release since Flink 1.0 launched nine ...
flink.apache.org
March 24, 2025 at 9:26 AM
Apache Flink 2.0.0 is out, featuring disaggregated state management (object storage), materialized tables unifying stream/batch processing, enhanced batch execution, deeper Paimon (Open table format) integration for creating streaming lakehouses and AI model support: flink.apache.org/2025/03/24/a...
This got me right in the feels...
It's always nice to return to sc2k, as it was one of the very first games I owned and was the subject of the first post I made on the original version of this account (Management Games Aesthetics) over on the old website...
March 5, 2025 at 10:41 PM
This got me right in the feels...
Got a bit trigger happy in @vinylguru.co.uk. Finally invested in some John Coltrane and filling in the gaps in the Fleetwood Mac collection. Also very happy to have a copy of the Cowboy Bebop soundtrack all the way from Japan "three, two, one, let's jam..."
February 21, 2025 at 3:45 PM
Got a bit trigger happy in @vinylguru.co.uk. Finally invested in some John Coltrane and filling in the gaps in the Fleetwood Mac collection. Also very happy to have a copy of the Cowboy Bebop soundtrack all the way from Japan "three, two, one, let's jam..."
This man is talking my language!
Diary entry 2 of formally verifying Kafka transactions. This entry covers my initial design decisions, and the nuances of liveness properties, with this first iteration of the TLA+ specification. Only read if you want to get down and dirty with TLA+!
jack-vanlightly.com/...
jack-vanlightly.com/...
Verifying Kafka transactions - Diary entry 2 - Writing an initial TLA+ spec — Jack Vanlightly
Strap in and fire up your nerd engines—this one's gonna be intense!
In my previous diary entry, I explained I would begin the formal modelling
of Kafka transactions by implementing a tiny initial piece of the protocol
- how a producer obtains a producer id. In this post I will discuss some
aspects of the TLA+ specification I have written of this initial model.
jack-vanlightly.com
December 3, 2024 at 1:46 PM
This man is talking my language!