I’ve been thinking about microservices a lot lately, especially how to build them for real-world production use. It’s not just about writing code; it’s about creating systems that are resilient, observable, and scalable. That’s why I want to share my approach to building event-driven microservices using Go, NATS JetStream, and OpenTelemetry.

Event-driven architectures have become essential for modern applications. They allow services to communicate asynchronously, making systems more decoupled and scalable. But how do you ensure these systems are reliable when things go wrong?

Let me show you how I structure a production-ready service. The foundation starts with a solid base service that handles everything from HTTP servers to graceful shutdowns.

type BaseService struct {
    Name        string
    Port        int
    Logger      *zap.Logger
    EventBus    *messaging.EventBus
    HTTPServer  *http.Server
    Router      *gin.Engine
    Tracer      trace.Tracer
    shutdownCh  chan struct{}
    wg          sync.WaitGroup
}

NATS JetStream provides the messaging backbone. It’s not just about sending messages—it’s about ensuring they’re persisted and delivered reliably. Have you considered what happens when your messaging system goes down?

func NewEventBus(config NATSConfig) (*EventBus, error) {
    opts := []nats.Option{
        nats.Name(config.ConnectionName),
        nats.MaxReconnects(config.MaxReconnects),
        nats.ReconnectWait(config.ReconnectWait),
        nats.DisconnectErrHandler(func(nc *nats.Conn, err error) {
            fmt.Printf("NATS disconnected: %v\n", err)
        }),
    }
    
    conn, err := nats.Connect(config.URL, opts...)
    if err != nil {
        return nil, fmt.Errorf("failed to connect to NATS: %w", err)
    }
    
    js, err := conn.JetStream()
    if err != nil {
        return nil, fmt.Errorf("failed to create JetStream context: %w", err)
    }
    
    return &EventBus{conn: conn, js: js}, nil
}

Observability is where OpenTelemetry shines. It’s not just about collecting data—it’s about understanding your system’s behavior. How do you trace a request across multiple services?

func (eb *EventBus) Publish(ctx context.Context, subject string, event events.Event) error {
    span := trace.SpanFromContext(ctx)
    event.TraceID = span.SpanContext().TraceID().String()
    event.SpanID = span.SpanContext().SpanID().String()
    
    data, err := json.Marshal(event)
    if err != nil {
        return fmt.Errorf("failed to marshal event: %w", err)
    }
    
    _, err = eb.js.Publish(subject, data)
    if err != nil {
        return fmt.Errorf("failed to publish event: %w", err)
    }
    
    return nil
}

Error handling is crucial in distributed systems. What strategies do you use when a service becomes unavailable? I implement retries with exponential backoff and circuit breakers to prevent cascading failures.

func (s *OrderService) processOrder(ctx context.Context, order events.OrderCreated) error {
    const maxRetries = 3
    var lastErr error
    
    for i := 0; i < maxRetries; i++ {
        err := s.inventoryClient.ReserveInventory(ctx, order.Items)
        if err == nil {
            return nil
        }
        
        lastErr = err
        time.Sleep(time.Duration(math.Pow(2, float64(i))) * time.Second)
    }
    
    return fmt.Errorf("failed after %d retries: %w", maxRetries, lastErr)
}

Testing event-driven systems requires a different approach. How do you verify that events are being produced and consumed correctly? I use containerized testing with real NATS instances to simulate production behavior.

Deployment and monitoring complete the picture. Containerizing services with Docker ensures consistency across environments, while proper monitoring helps catch issues before they affect users.

Building production-ready microservices is challenging but rewarding. The combination of Go’s performance, NATS JetStream’s reliability, and OpenTelemetry’s observability creates a powerful foundation for modern applications.

What challenges have you faced with microservices? I’d love to hear your experiences—feel free to share your thoughts in the comments below, and if you found this helpful, please like and share!