I’ve been thinking a lot about microservices lately—not just as isolated components, but as interconnected systems that need to communicate reliably, scale gracefully, and remain observable under real-world conditions. That’s why I decided to build a production-ready event-driven architecture using Go, NATS JetStream, and OpenTelemetry. If you’re working on distributed systems, you know how critical it is to get the foundations right from day one.

Let me show you how I approached this.

The core of our system is NATS JetStream, which provides persistent, fault-tolerant messaging. Here’s how I set up the connection with proper error handling and reconnection logic:

conn, err := nats.Connect(natsURL,
    nats.ReconnectWait(2*time.Second),
    nats.MaxReconnects(10),
    nats.DisconnectHandler(func(nc *nats.Conn) {
        log.Warn().Msg("NATS connection lost")
    }),
    nats.ReconnectHandler(func(nc *nats.Conn) {
        log.Info().Msg("NATS connection restored")
    }))
if err != nil {
    return nil, fmt.Errorf("NATS connection failed: %w", err)
}

But what happens when services need to understand the context of a request across multiple events? That’s where distributed tracing comes in.

OpenTelemetry gives us visibility across service boundaries. Here’s how I instrument event publishing to propagate trace context:

func (eb *EventBus) PublishWithTrace(ctx context.Context, subject string, event Event) error {
    carrier := propagation.HeaderCarrier{}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
    
    event.Metadata["traceparent"] = carrier.Get("traceparent")
    return eb.js.Publish(subject, event.Bytes())
}

When building event-driven systems, have you considered how you’ll handle partial failures or retries?

Circuit breakers prevent cascading failures. I use gobreaker to wrap external calls:

breaker := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "payment-service",
    Timeout:     10 * time.Second,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

result, err := breaker.Execute(func() (interface{}, error) {
    return paymentClient.Process(ctx, request)
})

Event schemas evolve over time. How do you ensure backward compatibility?

I use versioned event types and careful schema design:

type OrderCreatedV1 struct {
    OrderID    string       `json:"order_id"`
    CustomerID string       `json:"customer_id"`
    Items      []OrderItem  `json:"items"`
    Total      float64      `json:"total"`
}

type OrderCreatedV2 struct {
    OrderID    string       `json:"order_id"`
    CustomerID string       `json:"customer_id"`
    Items      []OrderItem  `json:"items"`
    Total      MonetaryAmount `json:"total"` // Now supports multiple currencies
}

Monitoring is not optional in production. I export metrics for every critical operation:

ordersProcessed := prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: "orders_processed_total",
    Help: "Total number of orders processed",
}, []string{"service", "status"})

prometheus.MustRegister(ordersProcessed)

Deployment matters too. Here’s a snippet from my Docker setup ensuring proper shutdown:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), 
        syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    app := fx.New(
        fx.Provide(NewEventBus),
        fx.Provide(NewOrderService),
        fx.Invoke(RegisterHandlers),
    )

    go func() {
        <-ctx.Done()
        stop()
        app.Stop(context.Background())
    }()

    app.Run()
}

Building event-driven microservices requires attention to reliability, observability, and resilience. Each service must handle its own state while cooperating through well-defined events. The patterns I’ve shared—proper tracing, circuit breaking, versioned events, and graceful shutdown—form a solid foundation for production systems.

What challenges have you faced with event-driven architectures? I’d love to hear your experiences—share your thoughts in the comments below, and if you found this useful, please like and share with others who might benefit from these patterns.