Karafka

Inside Kafka: Enhancing Data Reliability Through Transactional Offsets with Karafka 1

Inside Kafka: Enhancing Data Reliability Through Transactional Offsets with Karafka

Leave a Comment / apache kafka, Karafka, Ruby, Software, Ubuntu, Ubuntu Feed / By / June 22, 2023

Karafka is a Ruby and Rails framework that simplifies the development of Apache Kafka-based applications. Among its varied features, the Filtering API provides enhanced control over the data flow. The crux of this article is about managing offsets – unique identifiers for messages within Kafka’s partitions. Often, there’s a need to manage offsets alongside database …

Inside Kafka: Enhancing Data Reliability Through Transactional Offsets with Karafka Read More »

Karafka framework 2.1 announcement 5

Karafka framework 2.1 announcement

Leave a Comment / Karafka, karafka framework, kubernetes, Ruby, Software, Ubuntu, Ubuntu Feed / By / June 2, 2023

I’m happy to announce that Karafka 2.1 has just been released. For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework. The Karafka 2.1 release builds upon the foundation set by its predecessor, 2.0, making it a seamless continuation rather than a major rewrite. This means that …

Karafka framework 2.1 announcement Read More »

Delaying Kafka Messages Processing with Karafka: A Deep Dive into Delayed Topics 9

Delaying Kafka Messages Processing with Karafka: A Deep Dive into Delayed Topics

Leave a Comment / kafka, Karafka, Ruby, Software, Ubuntu, Ubuntu Feed / By / April 16, 2023

Kafka is a popular distributed streaming platform that is commonly used for building real-time data pipelines and streaming applications. One of the core features of Kafka is its ability to handle high-volume, real-time data streams and reliably process and distribute them to multiple consumers. However, in some cases, it may be necessary to postpone the …

Delaying Kafka Messages Processing with Karafka: A Deep Dive into Delayed Topics Read More »

Kafka topics as code – declarative Kafka topics management in Ruby 15

Kafka topics as code – declarative Kafka topics management in Ruby

Leave a Comment / apache kafka, kafka, Karafka, karafka framework, Ruby, Software, Ubuntu, Ubuntu Feed / By / March 15, 2023

Kafka topics are a fundamental concept in Apache Kafka. Topics are logical names or labels representing a stream of messages that Kafka clients can produce and consume. What makes them interesting is the variety of settings that can be applied to them. These settings, amongst others include: Partition count: The number of partitions that a …

Kafka topics as code – declarative Kafka topics management in Ruby Read More »

Karafka Web UI – Your Ruby and Rails out-of-the-box Kafka UI 17

Karafka Web UI – Your Ruby and Rails out-of-the-box Kafka UI

Leave a Comment / apache kafka, kafka, Karafka, karafka framework, Rails, Ruby, Software, Ubuntu, Ubuntu Feed, waterdrop / By / February 15, 2023

I’m thrilled to announce the new and shiny addition to the Karafka ecosystem: Karafka Web. For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework. Karafka has always been a convenient framework, and I’ve abstracted or hidden many complexities related to working with Apache Kafka. However, the …

Karafka Web UI – Your Ruby and Rails out-of-the-box Kafka UI Read More »

Ruby concurrency is hard: how I became a Ruby on Rails contributor 20

Ruby concurrency is hard: how I became a Ruby on Rails contributor

Leave a Comment / bug, concurrency, concurrent-ruby, Education, Karafka, mutex, parallel, Rails, Ruby, Software, Ubuntu, Ubuntu Feed / By / November 23, 2022

For the past several weeks, I’ve been trying to fix a cranky spec in Karafka integrations suite, which in the end, lead me to become a Ruby on Rails micro-contributor and submitting similar fix to several other high-popularity projects from the Ruby ecosystem. Here’s my story of trying to make sense of my specs and …

Ruby concurrency is hard: how I became a Ruby on Rails contributor Read More »

Karafka framework 2.0 announcement 24

Karafka framework 2.0 announcement

Leave a Comment / apache kafka, kafka, Karafka, karafka framework, Rails, Ruby, Ubuntu, Ubuntu Feed / By / August 9, 2022

I’m thrilled to announce the new and shiny Karafka 2.0. It is an effect of my work of almost four years. For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework. Karafka 2.0 is a major rewrite that brings many new things to the table but removes …

Karafka framework 2.0 announcement Read More »

Reduce your method calls by 99.9% by replacing Thread#pass with Queue#pop 31

Reduce your method calls by 99.9% by replacing Thread#pass with Queue#pop

Leave a Comment / benchmark, Karafka, karafka framework, Performance, Ruby, Software, Ubuntu, Ubuntu Feed / By / January 20, 2022

When doing multi-threaded work in Ruby, there are a couple of ways to control the execution flow within a given thread. In this article, I will be looking at Thread#pass and Queue#pop and how understanding each of them can help you drastically optimize your applications. Thread#pass – what it is and how does it work …

Reduce your method calls by 99.9% by replacing Thread#pass with Queue#pop Read More »

The hidden cost of a Ruby threads leakage 42

The hidden cost of a Ruby threads leakage

Leave a Comment / Karafka, karafka framework, Performance, Ruby, Ruby 3, Software, Ubuntu, Ubuntu Feed / By / December 31, 2020

Bug hunting Recently I’ve been working with one small application that would gradually become slower and slower. While there were many reasons for it to happen, I found one of them interesting. To give you a bit of context: the application was a simple single topic legacy Kafka consumer. I rewrote it to Karafka, and …

The hidden cost of a Ruby threads leakage Read More »

Karafka framework 1.4.0 Release Notes (Ruby + Kafka)

Leave a Comment / apache kafka, kafka, Karafka, karafka framework, Rails, Ruby, Software, Ubuntu, Ubuntu Feed / By / September 5, 2020

This release mostly solves problems related to message deserialization and normalizes some of the naming conventions to ease during the upgrade to the upcoming 2.0 version. Note: This release is the last release with ruby-kafka under the hood. We’ve already started the process of moving to rdkafka-ruby. Note: If you are using Sidekiq-Backend plugin, please …

Karafka framework 1.4.0 Release Notes (Ruby + Kafka) Read More »