I’ve been thinking about microservices a lot recently. Specifically, how we can build systems that handle real-world chaos - network failures, overloaded components, and unpredictable traffic spikes. That’s what led me to explore event-driven architectures using Go, NATS JetStream, and OpenTelemetry. Why these tools? Go’s concurrency model fits distributed systems like a glove, NATS JetStream provides durable messaging, and OpenTelemetry gives us visibility into complex interactions. Let me show you how these pieces come together to create resilient systems.

When building our e-commerce order processing system, we started with clear boundaries between services. Each service - orders, payments, inventory, notifications, and auditing - owns its domain logic. They communicate purely through events published to NATS JetStream streams. This separation prevents cascading failures; if the notification service goes down, orders still get processed. Have you considered how your services would behave if one component stopped responding?

Our foundation begins with defining event schemas. Clear contracts prevent integration headaches down the line:

type OrderCreated struct {
    OrderID    string     `json:"order_id"`
    CustomerID string     `json:"customer_id"`
    Items      []Item     `json:"items"`
    Total      float64    `json:"total_amount"`
    CreatedAt  time.Time  `json:"created_at"`
}

type PaymentProcessed struct {
    OrderID    string     `json:"order_id"`
    PaymentID  string     `json:"payment_id"`
    Amount     float64    `json:"amount"`
    ProcessedAt time.Time `json:"processed_at"`
}

Setting up the infrastructure is straightforward with Docker. Our docker-compose brings up NATS with JetStream enabled, plus Jaeger for tracing and Prometheus for metrics:

services:
  nats:
    image: nats:2.10-alpine
    command: ["--jetstream", "--store_dir=/data"]
    ports: ["4222:4222"]
  
  jaeger:
    image: jaegertracing/all-in-one:1.50
    ports: ["16686:16686"]

The event bus implementation handles tracing propagation and durable publishing. Notice how we attach OpenTelemetry context to events:

func (b *JetStreamBus) Publish(ctx context.Context, event events.Event, opts PublishOptions) error {
    span := trace.SpanFromContext(ctx)
    event.TraceID = span.SpanContext().TraceID().String()
    event.SpanID = span.SpanContext().SpanID().String()

    msg := nats.NewMsg(opts.Subject)
    msg.Data, _ = json.Marshal(event)
    
    if opts.Dedupe {
        msg.Header.Set("Nats-Msg-Id", opts.DedupeID)
    }
    
    _, err := b.js.PublishMsg(msg, nats.MaxWait(opts.Timeout))
    return err
}

For consumers, we implement pull-based subscribers with configurable error handling. This snippet shows how we process messages with automatic retries:

func (s *PaymentService) processPayments() {
    sub, _ := s.js.PullSubscribe("ORDERS.created", "payments-group", 
        nats.MaxDeliver(5),
        nats.AckWait(30*time.Second),
    )
    
    for {
        msgs, _ := sub.Fetch(10, nats.MaxWait(5*time.Second))
        for _, msg := range msgs {
            var event events.Event
            json.Unmarshal(msg.Data, &event)
            
            if err := s.handlePayment(event); err != nil {
                msg.Nak() // Trigger redelivery
            } else {
                msg.Ack()
            }
        }
    }
}

What happens when downstream services fail repeatedly? We use circuit breakers to prevent overwhelming struggling systems. The gobreaker package provides this protection:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:     "InventoryService",
    Timeout:  15 * time.Second,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

_, err := cb.Execute(func() (interface{}, error) {
    return s.inventoryClient.ReserveItems(order.Items)
})

For observability, we instrument everything with OpenTelemetry. This trace shows an order’s journey through our system:

func (s *OrderService) CreateOrder(ctx context.Context, order Order) {
    ctx, span := tracer.Start(ctx, "OrderService.CreateOrder")
    defer span.End()
    
    event := events.NewOrderCreatedEvent(order)
    if err := s.bus.Publish(ctx, event); err != nil {
        span.RecordError(err)
    }
}

Monitoring comes alive with Prometheus metrics. We track everything from event delivery latency to processing errors:

var (
    eventsPublished = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "events_published_total",
        Help: "Total published events",
    }, []string{"event_type"})
    
    processingTime = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name: "event_processing_seconds",
        Help: "Event processing time",
    }, []string{"handler"})
)

func recordPublish(eventType string) {
    eventsPublished.WithLabelValues(eventType).Inc()
}

Testing resilience reveals interesting behaviors. We use chaos techniques like:

• Injecting network partitions between services
• Randomly delaying message delivery
• Forcing NATS server restarts
• Simulating downstream timeouts

These experiments validate our failure handling. How would your system hold up under similar stress?

Deployment follows immutable infrastructure principles. Each service runs in its own container, with coordinated releases through CI/CD pipelines. Our monitoring stack alerts on:

• Event backlog growth
• Circuit breaker trips
• Trace error rates
• Resource saturation

Common pitfalls we’ve encountered include:

• Forgetting to set message deduplication IDs
• Missing context propagation in async operations
• Underestimating JetStream storage requirements
• Overlooking consumer group rebalancing

The combination of Go’s efficiency, JetStream’s persistence, and OpenTelemetry’s visibility creates a powerful foundation. We’ve handled peak loads exceeding 10,000 events per second with predictable latency. The true value emerges when production issues arise - we can trace problems across service boundaries and quickly resolve them.

What challenges have you faced with microservices? Share your experiences in the comments below. If you found this useful, consider sharing it with others building distributed systems.