Introduction Now and then, I like to go on a performance improvement hunt because life isn’t just about adding new features. Recently, I have been focusing on enhancing efficiency, particularly regarding CPU and memory usage in Karafka. Three of my recent pull requests (PR 117, PR 118, PR 123), have made some minor improvements, and […]
Under the Hood: Enhancing Karafka’s CPU and Memory Efficiency Read More »
Opening Note We all make mistakes, and fundamentally, the havoc caused by this incident was due to a flaw in the design of rdkafka-ruby. While the disappearance of librdkafka from GitHub was unexpected, this article aims to clarify and explain how rdkafka-ruby should have prevented it and what was poorly designed. By examining this incident,
The librdkafka Supply Chain Breakdown: rdkafka-ruby’s Darkest Hour Read More »
As an open-source developer, I constantly seek performance gains in the code I maintain. Since I took over rdkafka from AppSignal in November 2023, I promised not only to maintain the gem but to provide a stream of feature improvements and performance enhancements. One key area where performance can often be improved is how synchronization
From Sleep to Speed: Making Rdkafka Sync Operations 16 Times Faster Read More »
Introduction I am thrilled to announce the release of Karafka 2.4, a significant milestone in the evolution of my Ruby and Rails multi-threaded efficient Kafka processing framework. This release builds upon the solid foundation laid by its predecessor, Karafka 2.3. It introduces many new features, enhancements, and improvements across the entire Karafka ecosystem, including Karafka,
Karafka 2.4 Release Announcement: Advancing Kafka Processing for Ruby and Rails Read More »
Introduction In the world of Kafka and its applications, utilizing regular expressions for topic subscriptions is a common strategy. This approach is particularly beneficial for dynamically managing data. For example it can be used to handle information from various zones without necessitating application redeployment for each new topic. For instance, businesses operating across multiple zones
From Oniguruma to POSIX: The Regex Rift Between Ruby and C Read More »
Introduction The journey towards efficient parallelization in library development has often been based on using threads. As Karafka celebrates its eighth anniversary, it’s become clear that while threads have served us well for many tasks, there’s room to explore further. That’s why I’ve decided to introduce forking capabilities into Karafka, aiming to offer another dimension
The Art of Forking: Unlocking Scalability in Ruby Read More »
Introduction Karafka is a Ruby and Rails framework designed to simplify processing messages consumed from Apache Kafka. One of Karafka’s components is the https://karafka.io/docs/Web-UI-Getting-Started/. It provides a convenient way for developers to monitor and manage their Karafka-based applications without using the command line or third-party software. The interface, amongst others, includes: historical metrics, real-time aggregated
Monitoring Karafka Jobs Progress Using Web UI Read More »
Introduction Managing frontend assets in gems serving Web UI can be tricky. Why? Because while you want assets to expire across versions, you don’t want them fetched repeatedly, slowing things down. This challenge popped up when working on the Karafka Web UI, which runs on the Roda framework. In this article, I’ll explore how I
Asset Handling in Roda: Cache Forever, Refresh When Needed Read More »
Karafka is a Ruby and Rails framework that simplifies the development of Apache Kafka-based applications. Among its varied features, the Filtering API provides enhanced control over the data flow. The crux of this article is about managing offsets – unique identifiers for messages within Kafka’s partitions. Often, there’s a need to manage offsets alongside database
Inside Kafka: Enhancing Data Reliability Through Transactional Offsets with Karafka Read More »
I’m happy to announce that Karafka 2.1 has just been released. For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework. The Karafka 2.1 release builds upon the foundation set by its predecessor, 2.0, making it a seamless continuation rather than a major rewrite. This means that
Karafka framework 2.1 announcement Read More »
