I’ve been working with distributed systems for years, and few technologies have transformed how we build scalable applications as much as Apache Kafka. Pairing it with Go—a language built for concurrency and performance—creates a powerhouse for event-driven architectures. Whether you’re handling millions of messages or ensuring fault tolerance in microservices, this combination delivers.

Let’s start with the basics. Kafka organizes data into topics, which are split into partitions for parallelism. Producers write events to these partitions, and consumers read them. But how do you ensure messages are delivered exactly once, even when things go wrong?

Here’s a simple producer setup using Shopify’s Sarama library:

config := sarama.NewConfig()
config.Producer.Return.Successes = true
config.Producer.Idempotent = true
config.Producer.RequiredAcks = sarama.WaitForAll

producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, config)
if err != nil {
    log.Fatalf("Failed to create producer: %v", err)
}

msg := &sarama.ProducerMessage{
    Topic: "user-events",
    Value: sarama.StringEncoder(`{"user_id": 123, "action": "login"}`),
}

partition, offset, err := producer.SendMessage(msg)
if err != nil {
    log.Printf("Failed to send message: %v", err)
} else {
    log.Printf("Message sent to partition %d at offset %d", partition, offset)
}

Notice the Idempotent and RequiredAcks settings—these are crucial for production reliability. But what happens if a consumer fails mid-process?

Consumers need to handle offsets carefully to avoid missing or reprocessing messages. Consumer groups allow multiple instances to share the load, each taking responsibility for specific partitions. Here’s a basic consumer:

config := sarama.NewConfig()
config.Consumer.Group.Rebalance.GroupStrategies = []sarama.BalanceStrategy{sarama.NewBalanceStrategyRange()}
config.Consumer.Offsets.Initial = sarama.OffsetOldest

consumer, err := sarama.NewConsumerGroup([]string{"localhost:9092"}, "user-group", config)
if err != nil {
    log.Fatalf("Error creating consumer group: %v", err)
}

handler := &ConsumerHandler{}
ctx := context.Background()
for {
    err := consumer.Consume(ctx, []string{"user-events"}, handler)
    if err != nil {
        log.Printf("Consume error: %v", err)
    }
}

Handling errors gracefully is where many applications stumble. Consider implementing a dead-letter queue for messages that repeatedly fail processing:

func (h *ConsumerHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    for msg := range claim.Messages() {
        err := h.processMessage(msg)
        if err != nil {
            h.sendToDLQ(msg, err)
        }
        session.MarkMessage(msg, "")
    }
    return nil
}

What metrics would you track to know if your system is healthy? Latency, throughput, error rates, and consumer lag are essential. Integrating Prometheus gives you real-time visibility:

func init() {
    prometheus.MustRegister(processedMessagesCounter)
    prometheus.MustRegister(processingDurationHistogram)
}

func recordMetrics(start time.Time, success bool) {
    processedMessagesCounter.WithLabelValues(strconv.FormatBool(success)).Inc()
    processingDurationHistogram.Observe(time.Since(start).Seconds())
}

Testing event-driven applications requires simulating real-world conditions. Use Docker Compose to spin up a local Kafka cluster for integration tests. This ensures your application behaves correctly in a production-like environment.

Deploying to production? Consider using circuit breakers to prevent cascading failures. The gobreaker library works well here:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "kafka-producer",
    Timeout:     30 * time.Second,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

_, err := cb.Execute(func() (interface{}, error) {
    return producer.SendMessage(msg)
})

Building with Kafka and Go means designing for failure. Networks partition, brokers restart, consumers crash—your application should handle these gracefully. Always implement proper shutdown handling to avoid message loss:

func (h *ConsumerHandler) Setup(sarama.ConsumerGroupSession) error {
    return nil
}

func (h *ConsumerHandler) Cleanup(sarama.ConsumerGroupSession) error {
    return nil
}

func (h *ConsumerHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    for {
        select {
        case msg := <-claim.Messages():
            h.processMessage(msg)
            session.MarkMessage(msg, "")
        case <-session.Context().Done():
            return nil
        }
    }
}

I hope this gives you a solid foundation. Building production-ready systems is challenging but immensely rewarding. What strategies do you use for monitoring event-driven applications? Share your thoughts in the comments—I’d love to hear your experiences. If this was helpful, please like and share!