Elad Leev
@leevs.dev
Staff Eng | Stream all the things | @confluentinc Community Catalyst 🦸| #DistributedSystems #DataStreams #Databases #Scalability #kafka #flink #golang
Curious to know what connectors are mostly in use, and how many of them are in-house/forks?
February 20, 2025 at 11:33 PM
Curious to know what connectors are mostly in use, and how many of them are in-house/forks?
What’s your take on this implementation? I have love-hate relationship with it. Totally get why it was built that way (consumer offsets, Schema Registry topic, kStreams state, etc etc), but does it stand the test of time?
December 13, 2024 at 9:19 PM
What’s your take on this implementation? I have love-hate relationship with it. Totally get why it was built that way (consumer offsets, Schema Registry topic, kStreams state, etc etc), but does it stand the test of time?
Unfortunately, not really. We have a very (very) raw version of it open sourced, but it’s missing 7 months of advancement.
I do know there is an implementation which about to get merged:
github.com/apache/flink...
Worth giving it a try!
I do know there is an implementation which about to get merged:
github.com/apache/flink...
Worth giving it a try!
[FLINK-34440][formats][protobuf-confluent] Protobuf confluent dynamic format by dmariassy · Pull Request #25114 · apache/flink
What is the purpose of the change
Add support for deserializing protobuf messages using the Confluent wire format and whose schemas can be fetched from Confluent Schema Registry
Add support for se...
github.com
October 30, 2024 at 9:57 PM
Unfortunately, not really. We have a very (very) raw version of it open sourced, but it’s missing 7 months of advancement.
I do know there is an implementation which about to get merged:
github.com/apache/flink...
Worth giving it a try!
I do know there is an implementation which about to get merged:
github.com/apache/flink...
Worth giving it a try!
Reposted by Elad Leev
Having said that, everything is a trade off, and although it put more load (in terms of eng time) on the Data Platform teams, our clients are extremely happy with it. Proto tooling (code gen, grpc, protovalidte) are just superior to Avro
October 25, 2024 at 9:46 PM
Having said that, everything is a trade off, and although it put more load (in terms of eng time) on the Data Platform teams, our clients are extremely happy with it. Proto tooling (code gen, grpc, protovalidte) are just superior to Avro
We’ve made the decision to go full Proto this time (always used Avro as main format in past companies). The amount of forks, bugs, and code changes we had to make to support it is insane. Most notably, we had to build our own Flink format to pull from registry, as Flink currently doesn’t support it
October 25, 2024 at 9:35 PM
We’ve made the decision to go full Proto this time (always used Avro as main format in past companies). The amount of forks, bugs, and code changes we had to make to support it is insane. Most notably, we had to build our own Flink format to pull from registry, as Flink currently doesn’t support it