I’ve been thinking a lot lately about how we build systems that can handle real-world complexity. Not just academic examples, but services that actually survive in production. The combination of NATS, Go, and Kubernetes keeps coming up as a powerful trio for creating resilient, scalable event-driven architectures. Let me share what I’ve learned about making this work effectively.

Event-driven architecture changes how services communicate. Instead of direct calls, services publish events that others can react to. This creates loose coupling and better scalability. But how do we ensure these events don’t get lost when things go wrong?

NATS provides a solid foundation for this approach. Its lightweight nature and JetStream persistence make it ideal for microservices. The real challenge comes in building services that handle failures gracefully while maintaining performance.

Let me show you how I structure a base service in Go:

type BaseService struct {
    Name     string
    Logger   zerolog.Logger
    NATS     *nats.Conn
    Router   *gin.Engine
    Metrics  *ServiceMetrics
}

func NewBaseService(name string) (*BaseService, error) {
    logger := zerolog.New(os.Stdout).With().
        Timestamp().
        Str("service", name).
        Logger()
    
    nc, err := nats.Connect(os.Getenv("NATS_URL"),
        nats.MaxReconnects(-1),
        nats.ReconnectWait(2*time.Second))
    if err != nil {
        return nil, err
    }
    
    return &BaseService{
        Name:    name,
        Logger:  logger,
        NATS:    nc,
        Router:  gin.New(),
        Metrics: newMetrics(name),
    }, nil
}

This foundation handles logging, metrics, and NATS connectivity with proper retry logic. But what happens when we need to publish events reliably?

Event publishing requires careful consideration. We need to ensure messages aren’t lost during failures while maintaining ordering where necessary. Here’s how I handle event publication:

func (s *BaseService) PublishEvent(subject string, data []byte) error {
    ack, err := s.JetStream.PublishAsync(subject, data)
    if err != nil {
        s.Logger.Error().Err(err).Msg("Failed to publish event")
        return err
    }
    
    select {
    case <-ack.Ok():
        s.Metrics.EventsPublished.Inc()
        return nil
    case err := <-ack.Err():
        s.Logger.Error().Err(err).Msg("Event publish failed")
        return err
    case <-time.After(5 * time.Second):
        return errors.New("publish acknowledgement timeout")
    }
}

Error handling becomes critical in distributed systems. How do we ensure services can recover from temporary failures without manual intervention?

Retry patterns with exponential backoff help handle transient issues. I implement this using Go’s context and time packages:

func withRetry(ctx context.Context, maxAttempts int, fn func() error) error {
    for attempt := 1; attempt <= maxAttempts; attempt++ {
        err := fn()
        if err == nil {
            return nil
        }
        
        if attempt == maxAttempts {
            return err
        }
        
        backoff := time.Duration(math.Pow(2, float64(attempt))) * time.Second
        select {
        case <-time.After(backoff):
            continue
        case <-ctx.Done():
            return ctx.Err()
        }
    }
    return nil
}

Deploying these services to Kubernetes requires proper health checks and resource management. Liveness and readiness probes ensure containers restart when unhealthy and receive traffic only when ready:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

Monitoring distributed systems presents unique challenges. How do we trace requests across service boundaries while maintaining performance?

Structured logging combined with distributed tracing provides visibility. Each log entry includes correlation IDs that connect related events across services:

func (s *BaseService) LogWithContext(ctx context.Context) zerolog.Logger {
    if correlationID := GetCorrelationID(ctx); correlationID != "" {
        return s.Logger.With().Str("correlation_id", correlationID).Logger()
    }
    return s.Logger
}

Testing event-driven systems requires simulating the entire ecosystem. Docker Compose helps create integration test environments that mirror production:

services:
  nats:
    image: nats:latest
    ports:
      - "4222:4222"
  
  postgres:
    image: postgres:14
    environment:
      POSTGRES_DB: testdb
  
  service-under-test:
    build: .
    environment:
      NATS_URL: nats://nats:4222
      DB_URL: postgres://postgres@postgres/testdb

Building production-ready systems means anticipating failure at every level. Circuit breakers prevent cascading failures, while proper shutdown handling ensures clean termination:

func (s *BaseService) Start() error {
    go func() {
        if err := s.Server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            s.Logger.Fatal().Err(err).Msg("Server failed")
        }
    }()
    
    // Wait for shutdown signal
    <-s.ctx.Done()
    
    // Graceful shutdown
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    
    if err := s.Server.Shutdown(ctx); err != nil {
        return err
    }
    
    s.NATS.Close()
    return nil
}

The journey to production-ready microservices involves many considerations, but the payoff is systems that scale gracefully and handle failures without drama. Each service becomes an independent unit that can evolve separately while contributing to the whole system’s resilience.

I’d love to hear about your experiences with event-driven architectures. What challenges have you faced, and how did you solve them? Share your thoughts in the comments below, and if you found this useful, please consider sharing it with others who might benefit from these approaches.