I’ve been thinking a lot about distributed systems lately—how we can build applications that remain responsive even when components fail. Last week, a payment service outage at work caused cascading failures across our platform. That experience convinced me: we need better patterns for resilient systems. Today, I’ll share how to build production-ready event-driven microservices using Go, NATS JetStream, and OpenTelemetry. Stick with me—this approach could prevent those midnight outage calls.

Let’s start with why event-driven architecture makes sense. When services communicate through events rather than direct calls, failures become isolated. If payment processing goes down, orders can still queue up for later processing. But how do we ensure messages aren’t lost during failures? That’s where NATS JetStream shines—it provides persistent, fault-tolerant message streaming.

Our example is an e-commerce order system with four services: order processing, payment handling, inventory management, and notifications. They’ll communicate through events like OrderCreated or PaymentCompleted. We’ll use Protocol Buffers for efficient serialization—here’s a snippet defining our events:

message OrderCreated {
  string order_id = 1;
  string customer_id = 2;
  double total_amount = 3;
}

message PaymentCompleted {
  string order_id = 1;
  string payment_id = 2;
}

Notice how we’re separating event metadata from payloads? This structure helps with versioning and tracing. Speaking of tracing, have you ever debugged a request that crossed five services? Distributed tracing becomes essential. We’ll integrate OpenTelemetry directly into our event bus:

func (eb *EventBus) PublishEvent(ctx context.Context, subject string, event proto.Message) error {
    ctx, span := eb.tracer.Start(ctx, "EventBus.PublishEvent")
    defer span.End()
    // Inject trace context into event metadata
    carrier := propagation.MapCarrier{}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
    // ... serialization and publishing logic
}

This automatically propagates trace IDs across services. When a payment fails, we can see the entire journey from order creation to the failed API call. But observability alone isn’t enough—we need resilience. What happens when external APIs timeout? We’ll implement circuit breakers using the gobreaker library:

func ProcessPayment(order events.OrderCreated) error {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name: "PaymentProcessor",
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            return counts.ConsecutiveFailures > 5
        },
    })
    
    _, err := cb.Execute(func() (interface{}, error) {
        return nil, callPaymentGateway(order) // External call
    })
    return err
}

When failures exceed a threshold, the circuit opens, giving downstream services time to recover. Failed events stay in JetStream until we’re back online. But how do we handle load spikes efficiently? Go’s concurrency primitives are perfect for this. We’ll use worker pools to process events:

func StartWorkers(ctx context.Context, handler EventHandler, workers int) {
    for i := 0; i < workers; i++ {
        go func(id int) {
            for msg := range workQueue {
                ctx, span := tracer.Start(ctx, fmt.Sprintf("worker-%d", id))
                handler(ctx, msg) // Process event
                span.End()
                msg.Ack() // Confirm processing
            }
        }(i)
    }
}

Each service runs multiple workers pulling from shared channels. If one worker blocks, others keep processing. We balance throughput and resource usage. Now, what about deployment? Our Docker Compose brings up the entire stack—NATS, Jaeger for tracing, and our services—with health checks built in:

// Health endpoint
router.GET("/health", func(c *gin.Context) {
    if natsConnected && dbConnected {
        c.Status(http.StatusOK)
    } else {
        c.Status(http.StatusServiceUnavailable)
    }
})

This lets Kubernetes know when to restart pods. We also expose Prometheus metrics for queue lengths and processing times. Remember that payment outage I mentioned? With this architecture, even if the payment service restarts, JetStream replays all missed events in order. Customers might experience delays, but no data is lost.

The real beauty emerges when we need to add new capabilities. Recently, we added fraud detection by subscribing to OrderCreated events—zero changes to existing services. How many times have you delayed adding features because of dependencies?

Building this changed how I view distributed systems. Events become your source of truth. Tracing provides visibility. Resilience patterns handle real-world chaos. And Go’s simplicity keeps the code maintainable. Try implementing just the circuit breaker pattern next week—you’ll immediately notice fewer cascading failures.

If this resonates with you, give it a like. Share it with that colleague who’s always firefighting outages. Got questions or war stories? Drop them in the comments—let’s learn from each other’s battles.