I’ve spent the last several months working with event streaming systems in production environments, and I keep seeing the same patterns emerge. Teams struggle with scaling their Kafka implementations beyond basic examples, especially when combining Kafka with Go’s concurrency model. That’s why I want to share a practical approach to building production-ready event streaming applications.

Let me walk you through what actually works in real systems. First, let’s talk about configuration management - it’s more critical than you might think.

type ProducerConfig struct {
    Acks              string `mapstructure:"acks"`
    RetryBackoffMs    int    `mapstructure:"retry_backoff_ms"`
    MessageTimeoutMs  int    `mapstructure:"message_timeout_ms"`
    CompressionType   string `mapstructure:"compression_type"`
    EnableIdempotence bool   `mapstructure:"enable_idempotence"`
}

Setting acks to “all” and enabling idempotence gives you exactly-once semantics in many scenarios. But what happens when your producer faces network partitions or broker failures?

Here’s a production-ready producer implementation that handles these edge cases:

func (p *KafkaProducer) SendEvent(ctx context.Context, topic string, event models.BaseEvent) error {
    data, err := json.Marshal(event)
    if err != nil {
        return fmt.Errorf("failed to marshal event: %w", err)
    }

    deliveryChan := make(chan kafka.Event)
    err = p.producer.Produce(&kafka.Message{
        TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
        Value:          data,
        Headers:        []kafka.Header{{Key: "event-type", Value: []byte(event.Metadata.EventType)}},
    }, deliveryChan)

    if err != nil {
        return fmt.Errorf("failed to produce message: %w", err)
    }

    select {
    case e := <-deliveryChan:
        msg := e.(*kafka.Message)
        if msg.TopicPartition.Error != nil {
            return fmt.Errorf("delivery failed: %w", msg.TopicPartition.Error)
        }
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

Notice how we’re using context for cancellation and proper channel handling? This prevents goroutine leaks and ensures clean shutdowns.

Now, let’s talk about consumers. Have you ever wondered why some consumer groups rebalance constantly while others run smoothly for months?

The secret often lies in proper configuration and offset management. Here’s a consumer that handles rebalances gracefully:

func (c *KafkaConsumer) Consume(ctx context.Context, handler EventHandler) error {
    for {
        select {
        case <-ctx.Done():
            return c.close()
        default:
            msg, err := c.consumer.ReadMessage(100 * time.Millisecond)
            if err != nil {
                if err.(kafka.Error).Code() == kafka.ErrTimedOut {
                    continue
                }
                log.Errorf("Consumer error: %v", err)
                continue
            }

            var event models.BaseEvent
            if err := json.Unmarshal(msg.Value, &event); err != nil {
                log.Errorf("Failed to unmarshal event: %v", err)
                continue
            }

            if err := handler.Handle(ctx, event); err != nil {
                log.Errorf("Failed to handle event: %v", err)
                // Send to dead letter queue
                c.sendToDLQ(msg, err)
            }
        }
    }
}

Error handling in event streaming systems deserves special attention. Most tutorials show happy paths, but production systems need to handle failures gracefully. What do you do when a message processing fails repeatedly?

Dead letter queues are your friend. Here’s how I implement them:

func (c *KafkaConsumer) sendToDLQ(originalMsg *kafka.Message, processingError error) {
    dlqEvent := DLQEvent{
        OriginalMessage: originalMsg.Value,
        Error:           processingError.Error(),
        Timestamp:       time.Now(),
        Topic:           *originalMsg.TopicPartition.Topic,
        Partition:       originalMsg.TopicPartition.Partition,
        Offset:          originalMsg.TopicPartition.Offset,
    }

    data, _ := json.Marshal(dlqEvent)
    c.dlqProducer.Produce(&kafka.Message{
        TopicPartition: kafka.TopicPartition{Topic: &c.dlqTopic, Partition: kafka.PartitionAny},
        Value:          data,
    }, nil)
}

Monitoring and observability are non-negotiable in production systems. I instrument everything - from message rates to processing latency. But here’s a question: how do you distinguish between a temporary spike and a real problem in your streaming pipeline?

The answer lies in setting proper SLOs and using structured logging:

func (c *KafkaConsumer) processWithMetrics(ctx context.Context, event models.BaseEvent) error {
    start := time.Now()
    labels := prometheus.Labels{"event_type": event.Metadata.EventType}

    c.metrics.EventsProcessed.With(labels).Inc()
    defer func() {
        c.metrics.ProcessingDuration.With(labels).Observe(time.Since(start).Seconds())
    }()

    err := c.handler.Handle(ctx, event)
    if err != nil {
        c.metrics.ProcessingErrors.With(labels).Inc()
        log.WithFields(log.Fields{
            "event_id": event.Metadata.EventID,
            "duration": time.Since(start),
            "error":    err.Error(),
        }).Error("Event processing failed")
    }

    return err
}

Deployment considerations often get overlooked until it’s too late. When running in Kubernetes, you need to think about resource limits, liveness probes, and proper shutdown handling. Here’s a graceful shutdown pattern I’ve found invaluable:

func (c *KafkaConsumer) Start() error {
    ctx, cancel := context.WithCancel(context.Background())
    
    // Handle OS signals
    sigChan := make(chan os.Signal, 1)
    signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
    
    go func() {
        <-sigChan
        log.Info("Received shutdown signal")
        cancel()
        
        // Wait for in-flight messages to complete
        time.Sleep(30 * time.Second)
        c.close()
    }()
    
    return c.Consume(ctx, c.handler)
}

Testing event streaming applications requires a different mindset. You’re not just testing functions - you’re testing data flows and failure scenarios. I use Docker Compose to spin up a test Kafka cluster and run integration tests that simulate network partitions, broker failures, and message ordering issues.

One final thought: building production-ready event streaming systems is as much about operations as it is about code. You need proper alerting, capacity planning, and disaster recovery procedures. The code patterns I’ve shared will get you started, but remember that every production environment has its own unique challenges.

If you found these patterns helpful or have your own experiences to share, I’d love to hear from you in the comments. Feel free to share this with your team if you’re planning your next event-driven system. What challenges have you faced with Kafka in production?