Rdkafka is a modern and efficient Kafka client library specifically designed for Ruby, offering a powerful interface based on the robust librdkafka C client. Developed by AppSignal, this library is engineered for high performance and reliability in high-traffic environments.

Key Features:

• Version Compatibility: Supports Kafka 1.0+ and actively maintained Ruby versions to ensure security and functionality.
• Production-Ready: Proven in production by AppSignal, making it a trusted choice for developers working with Kafka in Ruby.
• Core Functionalities: Implements essential consumer, producer, and admin APIs while maintaining a focus on the direct functionalities provided by librdkafka.

Project Scope:

Rdkafka aims to simplify Kafka integration in Ruby applications without overcomplicating its use:

• Does not include complex producer/consumer functionalities but focuses closely on the capabilities provided by librdkafka.
• Existing functionalities such as producer metadata caching and a simple consumer are included to enhance usability without deviating from the primary focus.

Example Usage:

Consuming Messages:

Use the library to subscribe to a Kafka topic and receive messages seamlessly:

config = {
  :"bootstrap.servers" => "localhost:9092",
  :"group.id" => "ruby-test"
}
consumer = Rdkafka::Config.new(config).consumer
consumer.subscribe("ruby-test-topic")

consumer.each do |message|
  puts "Message received: #{message}"
end

Producing Messages:

Efficiently produce messages to a Kafka topic, ensuring messages are batched before sending:

config = {:"bootstrap.servers" => "localhost:9092"}
producer = Rdkafka::Config.new(config).producer
delivery_handles = []

100.times do |i|
  puts "Producing message #{i}"
  delivery_handles << producer.produce(
      topic:   "ruby-test-topic",
      payload: "Payload #{i}",
      key:     "Key #{i}"
  )
end

delivery_handles.each(&:wait)

Additional Libraries:

For developers seeking higher-level abstractions, consider using:

• Karafka: An efficient framework for processing Kafka messages in Ruby and Rails.
• WaterDrop: A standalone library for producing messages to Kafka.

Forking Considerations:

Due to known limitations in the underlying librdkafka regarding fork-safe operations, it’s crucial to manage producer and consumer initialization carefully in Ruby applications. Avoid initializing these clients before forking to prevent state corruption.

Development Environment:

A Docker Compose file is available for easy Kafka setup:

docker-compose up

To run tests and see debug outputs, you can easily manage your development workflow with RSpec and Docker.

Stay Updated:

Rdkafka-ruby is continuously enhanced to integrate with the latest librdkafka versions, ensuring you have access to the most reliable and modern Kafka capabilities available for Ruby.