Lightnews — Scholar-powered news

Igfasouza

@igfasouza.bsky.social

How OpenAI Uses Kubernetes And Apache Kafka for GenAI

#apachekafka

blog.bytebytego.com/p/how-openai...

How OpenAI Uses Kubernetes And Apache Kafka for GenAI

In this article, we will understand how OpenAI built such a system and the challenges they faced.

blog.bytebytego.com

November 11, 2025 at 10:19 AM

Igfasouza

@igfasouza.bsky.social

Why Diskless Is a Game-Changer for Running Kafka at Scale

#apachekafka

thenewstack.io/why-diskless...

Why Inkless Kafka Is a Game-Changer for Running Kafka at Scale

High replication costs are a serious problem when running Kafka in the cloud. Learn how writing data directly in object storage can help.

thenewstack.io

November 10, 2025 at 10:07 PM

Igfasouza

@igfasouza.bsky.social

From Tower of Babel to Babel Fish | Voxxed Days Thessaloniki 2025

#apachekafka

youtu.be/QOSoh_Q8pxU?...

From Tower of Babel to Babel Fish | Voxxed Days Thessaloniki 2025

YouTube video by Viktor Gamov

youtu.be

November 10, 2025 at 10:15 AM

Igfasouza

@igfasouza.bsky.social

You Don't Need Kafka, Just Use Postgres" Considered Harmful

#apachekafka

www.morling.dev/blog/you-don...

"You Don't Need Kafka, Just Use Postgres" Considered Harmful

Looking to make it to the front page of HackerNews? Then writing a post arguing that "Postgres is enough", or why "you don’t need Kafka at your scale" is a pretty failsafe way of achieving exactly tha...

www.morling.dev

November 9, 2025 at 7:15 PM

Igfasouza

@igfasouza.bsky.social

Building a 3 node Kafka cluster

#apachekafka

medium.com/@ranjanabhab...

Building a 3 node Kafka cluster

In the world of real-time data streaming, scalability and reliability are non-negotiable. Apache Kafka has become the backbone of modern…

medium.com

November 9, 2025 at 8:53 AM

Igfasouza

@igfasouza.bsky.social

The Real-Time Data Journey: Connecting Flink, Airflow and StarRocks

#apacheflink

medium.com/fresha-data-...

The Real-Time Data Journey: Connecting Flink, Airflow and StarRocks

Exploring how modern streaming tools power the next generation of analytics with StarRocks.

medium.com

November 8, 2025 at 9:39 PM

Igfasouza

@igfasouza.bsky.social

Event Streaming is Topping Out

#apachekafka

There's no money left in real-time event streaming. A major consolidation wave is beginning.

bigdata.2minutestreaming.com/p/event-stre...

Event Streaming is Topping Out

There's no money left in real-time event streaming. A major consolidation wave is beginning.

bigdata.2minutestreaming.com

November 8, 2025 at 10:19 AM

Igfasouza

@igfasouza.bsky.social

How Would You Like Your Iceberg Sir? Stream or Batch Ordered?

#apachekafka

jack-vanlightly.com/blog/2025/11...

How Would You Like Your Iceberg Sir? Stream or Batch Ordered? — Jack Vanlightly

Today I want to talk about stream analytics, batch analytics and Apache Iceberg. Stream and batch analytics work differently but both can be built on top of Iceberg, but due to their differences there...

jack-vanlightly.com

November 7, 2025 at 7:33 PM

Igfasouza

@igfasouza.bsky.social

Old-School Kafka: A Terminal-First Approach to Learning Distributed Streaming

#apachekafka

medium.com/@priyankaguh...

Old-School Kafka: A Terminal-First Approach to Learning Distributed Streaming

Imagine a bustling city. Every second, millions of things are happening: someone orders a coffee, a taxi picks up a passenger, a tweet goes…

medium.com

November 7, 2025 at 9:30 AM

Igfasouza

@igfasouza.bsky.social

Exploring Seamless Kafka-to-Data Lake Integration: Evaluating Tableflow as a Robust Streaming Analytics Solution

#apachekafka

medium.com/@jp.mantuano...

Exploring Seamless Kafka-to-Data Lake Integration: Evaluating Tableflow as a Robust Streaming…

In today’s rapidly evolving data landscape, organizations are continuously exploring innovative solutions to better manage data stored in…

medium.com

November 6, 2025 at 7:36 PM

Igfasouza

@igfasouza.bsky.social

The Stream Processing Showdown: Kafka Streams vs. Flink vs. Spark

#apachekafka
#apacheflink
#apachespark

medium.com/@balaji.raja...

The Stream Processing Showdown: Kafka Streams vs. Flink vs. Spark

The Stream Processing Showdown: Kafka Streams vs. Flink vs. Spark Read the full story for free …

medium.com

November 6, 2025 at 9:09 AM

Igfasouza

@igfasouza.bsky.social

Kafka Log Cleanup: Delete vs Compact

#apachekafka

www.geekish.fun/blog/kafka-d...

Geekish.Fun | Kafka Log Cleanup: Delete vs Compact

When to use cleanup.policy=delete vs compact, with trade-offs, configs, and recipes

www.geekish.fun

November 5, 2025 at 6:06 PM

Igfasouza

@igfasouza.bsky.social

Managing Kafka at Scale: Common Pitfalls and the Conduktor Fix

#apachekafka

medium.com/@rajesh.sgr/...

Managing Kafka at Scale: Common Pitfalls and the Conduktor Fix

Apache Kafka is the lifeblood of modern enterprises. From retail payments to IoT pipelines, it keeps data flowing in real time. But anyone…

medium.com

November 5, 2025 at 9:11 AM

Igfasouza

@igfasouza.bsky.social

The Good, The Bad and The AutoMQ

#apachekafka

medium.com/fresha-data-...

The Good, The Bad and The AutoMQ

Slack’s KIP-1176, Aiven’s Diskless 2.0, and AutoMQ’s KIP-1183 and how Kafka’s storage frontier shifts from disks to the cloud.

medium.com

November 4, 2025 at 7:48 PM

Igfasouza

@igfasouza.bsky.social

Kafka’s biggest contribution to enterprises isn’t what you think.

#apachekafka

medium.com/@chanaud.jul...

Kafka’s biggest contribution to enterprises isn’t what you think.

Kafka’s biggest contribution to the “Traditional World” [1] isn’t what you think. These organizations don’t use Kafka at LinkedIn scale and…

medium.com

November 4, 2025 at 9:13 AM

Igfasouza

@igfasouza.bsky.social

A collection of demos showcasing how stream processing can be used to solve real-world problems

#apachekafka

github.com/risingwavela...

GitHub - risingwavelabs/awesome-stream-processing: A collection of demos showcasing how stream processing can be used to solve real-world problems.

A collection of demos showcasing how stream processing can be used to solve real-world problems. - risingwavelabs/awesome-stream-processing

github.com

November 3, 2025 at 5:27 PM

Igfasouza

@igfasouza.bsky.social

Apache Kafka (Kafka Connect) vs. Apache Flink vs. Apache Spark: Choosing the Right Ingestion Framework

#apachekafka
#apacheflink
#apachespark

www.onehouse.ai/blog/kafka-c...

Apache Kafka® (Kafka Connect) vs. Apache Flink® vs. Apache Spark™: Choosing the Right Ingestion Framework

This article compares three data ingestion frameworks—Kafka, Flink, and Spark—highlighting their unique strengths, use cases, and performance capabilities.

www.onehouse.ai

November 3, 2025 at 9:11 AM

Igfasouza

@igfasouza.bsky.social

Examples of Kafka Headers

#apachekafka

www.superstream.ai/blog/kafka-h...

Kafka Headers Explained: Examples, Best Practices, and Metadata Governance | Superstream Blog

kafka-headers-explained-examples

www.superstream.ai

November 2, 2025 at 6:05 PM

Igfasouza

@igfasouza.bsky.social

Flink's 95% problem

#apacheflink

Flink might sound like the holy grail of streaming data processing, but for 95% of us, it's just a complex headache we don't need.

www.tinybird.co/blog/flink-i...

Flink's 95% problem

Flink might sound like the holy grail of streaming data processing, but for 95% of us, it's just a complex headache we don't need.

www.tinybird.co

November 2, 2025 at 1:00 PM

Igfasouza

@igfasouza.bsky.social

The ZooKeeper Eviction Notice: A Senior Architect’s Playbook for Migrating to KRaft

#apachekafka

medium.com/@chathuskadi...

The ZooKeeper Eviction Notice: A Senior Architect’s Playbook for Migrating to KRaft

For over a decade, Apache ZooKeeper has been the faithful, if somewhat complex, companion to every Kafka cluster. It’s the distributed…

medium.com

November 2, 2025 at 9:23 AM

Igfasouza

@igfasouza.bsky.social

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.

#apachekafka

github.com/linkedin/cru...