I’ve been thinking a lot about event streaming lately. In my work, I’ve seen how critical it is to build systems that can handle massive data flows reliably. That’s why I want to share what I’ve learned about creating production-ready applications with Apache Kafka and Go.

When you’re building event-driven systems, every decision matters. How do you ensure messages aren’t lost? What happens when things go wrong? These questions kept me up at night until I developed patterns that actually work in production environments.

Let me show you how I structure a Kafka producer that can handle real-world demands. The key is proper configuration and error handling from the start.

type SafeProducer struct {
    producer *kafka.Producer
    logger   *zap.Logger
    metrics  *ProducerMetrics
    mu       sync.Mutex
}

func NewSafeProducer(config *kafka.ConfigMap, logger *zap.Logger) (*SafeProducer, error) {
    p, err := kafka.NewProducer(config)
    if err != nil {
        return nil, fmt.Errorf("failed to create producer: %w", err)
    }
    
    sp := &SafeProducer{
        producer: p,
        logger:   logger,
        metrics:  newProducerMetrics(),
    }
    
    go sp.handleDeliveryReports()
    return sp, nil
}

Notice how I separate concerns here? The producer handles message delivery, while dedicated goroutines manage reporting and error handling. This separation is crucial for maintaining system stability under load.

But what happens when your consumer needs to process thousands of events per second? Traditional approaches often fall short. That’s where Go’s concurrency model shines.

func (c *ConcurrentConsumer) Start(ctx context.Context) {
    for i := 0; i < c.workerCount; i++ {
        go c.worker(ctx, i)
    }
    
    c.logger.Info("started consumer workers", zap.Int("count", c.workerCount))
}

func (c *ConcurrentConsumer) worker(ctx context.Context, id int) {
    for {
        select {
        case <-ctx.Done():
            return
        case msg, ok := <-c.messages:
            if !ok {
                return
            }
            c.processMessage(msg)
        }
    }
}

Error handling is where many systems break down. Have you considered what happens when processing fails repeatedly? I implemented a dead letter queue pattern that’s saved countless production issues.

func (h *MessageHandler) ProcessWithDLQ(msg *kafka.Message) error {
    err := h.process(msg)
    if err != nil {
        if h.shouldSendToDLQ(err) {
            return h.sendToDLQ(msg, err)
        }
        return fmt.Errorf("processing failed: %w", err)
    }
    return nil
}

Schema evolution is another critical aspect. As your system grows, your data structures will change. Using Avro with Schema Registry ensures backward and forward compatibility.

type AvroSerializer struct {
    client    *srclient.SchemaRegistryClient
    cache     map[int]*goavro.Codec
    cacheLock sync.RWMutex
}

func (s *AvroSerializer) Serialize(topic string, data interface{}) ([]byte, error) {
    schemaID, err := s.getSchemaID(topic)
    if err != nil {
        return nil, err
    }
    
    // Serialization logic with schema validation
    return s.encodeWithSchema(schemaID, data)
}

Monitoring and observability aren’t afterthoughts—they’re fundamental to production readiness. I instrument everything with metrics and tracing from day one.

func instrumentedProduce(producer *kafka.Producer, topic string, value []byte) error {
    start := time.Now()
    err := producer.Produce(&kafka.Message{
        TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
        Value:          value,
    }, nil)
    
    // Record metrics
    produceDuration.Observe(time.Since(start).Seconds())
    if err != nil {
        produceErrors.Inc()
    }
    return err
}

Testing event streaming applications requires a different approach. I’ve found that using Docker containers for integration testing provides the most realistic environment.

func TestOrderProcessing(t *testing.T) {
    compose := testutil.StartKafkaCluster(t)
    defer compose.Stop()
    
    producer := testutil.NewTestProducer(t)
    consumer := testutil.NewTestConsumer(t)
    
    // Test scenario implementation
    testOrder := createTestOrder()
    err := producer.Send("orders", testOrder)
    require.NoError(t, err)
    
    // Verify processing
    received := consumer.WaitForMessage(5*time.Second)
    assert.Equal(t, testOrder.ID, received.OrderID)
}

Deployment considerations often get overlooked until it’s too late. Containerization and proper resource management are non-negotiable for production systems.

# docker-compose.prod.yml
version: '3.8'
services:
  kafka-producer:
    build: .
    command: ["./producer"]
    deploy:
      resources:
        limits:
          memory: 512M
          cpus: '0.5'
    environment:
      - KAFKA_BROKERS=kafka1:9092,kafka2:9092,kafka3:9092
      - SCHEMA_REGISTRY_URL=http://schema-registry:8081

Building production-ready event streaming applications requires attention to detail at every level. From proper error handling and monitoring to deployment strategies, each component plays a vital role in system reliability.

I’d love to hear about your experiences with Kafka and Go. What challenges have you faced? What patterns have worked well for you? Share your thoughts in the comments below, and if you found this helpful, please like and share with others who might benefit from these insights.