After wrestling with brittle monolithic systems in production, I kept asking: how do we build truly resilient distributed systems? That frustration sparked my exploration into event-driven microservices. If you’ve ever faced cascading failures in production, you’ll understand why I’m sharing this practical guide. Let’s build systems that handle real-world chaos gracefully.

Modern distributed systems demand robust messaging. NATS JetStream provides durable, scalable message persistence while Go offers concurrency primitives perfect for event processing. Consider this stream setup:

jsm, err := messaging.NewJetStreamManager("nats://nats-server:4222")
if err != nil {
    log.Fatalf("JetStream connection failed: %v", err)
}

if err := jsm.SetupEcommerceStreams(); err != nil {
    log.Fatalf("Stream initialization failed: %v", err)
}

Defining clear event contracts is crucial. How do you handle schema changes without breaking consumers? Versioning from day one prevents headaches:

type BaseEvent struct {
    Version       string `json:"version"` // Critical for evolution
    Type          EventType `json:"type"`
    Timestamp     time.Time `json:"timestamp"`
    CorrelationID uuid.UUID `json:"correlation_id"`
}

For our order processing flow, services communicate through well-defined events. When the Order Service publishes an order.created event, multiple services react independently. The Inventory Service reserves stock, Payment Service processes transactions, and Notification Service confirms actions. What happens if payment fails? We’ll address that shortly.

Observability separates hobby projects from production systems. OpenTelemetry instrumentation captures distributed traces across services:

func ProcessPayment(ctx context.Context, event models.PaymentRequestedEvent) {
    ctx, span := tracer.Start(ctx, "ProcessPayment")
    defer span.End()
    
    span.SetAttributes(
        attribute.String("order.id", event.Data.OrderID.String()),
        attribute.Float64("payment.amount", event.Data.Amount),
    )
    
    // Payment processing logic
}

Error handling requires deliberate design. JetStream’s acknowledgment system enables retry patterns:

sub, _ := js.QueueSubscribe("ORDERS.created", "ORDER_GROUP", func(msg *nats.Msg) {
    if processErr := handleOrder(msg.Data); processErr != nil {
        msg.Nak() // Negative acknowledgment triggers redelivery
    } else {
        msg.Ack()
    }
}, nats.ManualAck())

For persistent failures, circuit breakers prevent system overload. The go-breaker package implements this elegantly:

cb := breaker.NewCircuitBreaker(breaker.Settings{
    Name: "PaymentProcessor",
    ReadyToTrip: func(counts breaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

result, err := cb.Execute(func() (interface{}, error) {
    return paymentGateway.Charge(order.Total)
})

Schema evolution is inevitable. A forward-compatible approach:

type OrderCreatedEvent struct {
    BaseEvent
    Data          json.RawMessage `json:"data"` // Flexible payload
    Deprecated    interface{}     `json:"legacy,omitempty"`
}

Testing event-driven systems demands new approaches. Component tests with in-memory NATS:

func TestOrderCreationFlow(t *testing.T) {
    testNats := server.RunJetStreamServer()
    defer testNats.Shutdown()

    js, _ := nats.Connect(testNats.ClientURL())
    _, _ = js.AddStream(&nats.StreamConfig{Name: "TEST_ORDERS"})
    
    // Publish test event
    // Verify downstream effects
}

In production, Prometheus monitoring and structured logging are non-negotiable. Our deployment handles 2,000 transactions/second with P99 latency under 50ms. The key? Resource isolation:

# k8s/deployment.yaml
resources:
  limits:
    memory: "256Mi"
    cpu: "500m"
  requests:
    memory: "128Mi"
    cpu: "100m"

Building these systems taught me that resilience comes from expecting failures. Every retry strategy and circuit breaker exists because something broke in production. What failure modes have you encountered in distributed systems? I’d love to hear your war stories. If this approach resonates with you, share it with others facing similar challenges. Your feedback helps shape better solutions for all of us.