I’ve been building distributed systems for over a decade, and recently faced a critical challenge while designing an e-commerce platform. How do you ensure order processing remains reliable when payment systems fail or inventory services become unavailable? This pushed me toward event-driven architecture with NATS JetStream, Go, and Kubernetes - a combination that handles real-world chaos exceptionally well. Let’s explore how these technologies create resilient systems.

First, our NATS JetStream setup provides the backbone. We run a clustered configuration for fault tolerance. Here’s our Docker Compose snippet for a 3-node cluster:

services:
  nats-1:
    image: nats:2.10-alpine
    command: --jetstream --cluster_name=NATS --routes=nats-route://nats-2:6222,nats-route://nats-3:6222
    ports: ["4222:4222"]
  nats-2:
    image: nats:2.10-alpine
    command: --jetstream --cluster_name=NATS --routes=nats-route://nats-1:6222,nats-route://nats-3:6222
  nats-3:
    image: nats:2.10-alpine
    command: --jetstream --cluster_name=NATS --routes=nats-route://nats-1:6222,nats-route://nats-2:6222

Event schemas form our communication contract. We define strict types in Go:

type BaseEvent struct {
    ID          string    `json:"id"`
    Type        string    `json:"type"` // e.g., "order.created"
    Timestamp   time.Time `json:"timestamp"`
}

type OrderCreatedEvent struct {
    BaseEvent
    OrderID string  `json:"orderId"`
    Items   []Item  `json:"items"`
}

type Item struct {
    ProductID string  `json:"productId"`
    Quantity  int     `json:"quantity"`
}

The Order Service initiates workflows by publishing events. Notice how we include correlation IDs for tracing:

func (s *OrderService) CreateOrder(ctx context.Context, order Order) error {
    event := events.OrderCreatedEvent{
        BaseEvent: events.BaseEvent{
            ID:        uuid.NewString(),
            Type:      "order.created",
            Timestamp: time.Now().UTC(),
        },
        OrderID: order.ID,
        Items:   order.Items,
    }
    data, _ := json.Marshal(event)
    return s.js.Publish("ORDERS.created", data)
}

For the Payment Service, we implement circuit breakers using Sony’s gobreaker. What happens when payment gateways start failing repeatedly? The breaker prevents cascading failures:

var cb = gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:    "PaymentProcessor",
    Timeout: 30 * time.Second,
})

func ProcessPayment(orderID string) error {
    result, err := cb.Execute(func() (interface{}, error) {
        return gateway.Charge(orderID) // External call
    })
    if err != nil {
        return fmt.Errorf("payment failed: %w", err)
    }
    return nil
}

The Inventory Service uses event sourcing for accuracy. By replaying events, we rebuild state after outages:

func RebuildInventoryState(streamName string) (map[string]int, error) {
    inventory := make(map[string]int)
    msgs, _ := js.GetLastMsg(streamName, "inventory.*")
    for _, msg := range msgs {
        switch msg.Header.Get("EventType") {
        case "inventory.reserved":
            // Deduct from inventory
        case "inventory.released":
            // Add back to inventory
        }
    }
    return inventory, nil
}

Dead letter queues handle poison messages. When processing fails persistently, we move messages to isolation:

func processMsg(msg *nats.Msg) {
    if err := handle(msg.Data); err != nil {
        if attempts, _ := msg.Metadata(); attempts > 3 {
            s.js.Publish("DLQ.orders", msg.Data) // Quarantine
            msg.Ack()
            return
        }
        msg.Nak() // Retry later
    }
    msg.Ack()
}

Testing event-driven systems requires simulating real-world failures. We use Testcontainers for integration tests:

func TestOrderFlow(t *testing.T) {
    ctx := context.Background()
    natsContainer, _ := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
        ContainerRequest: testcontainers.ContainerRequest{
            Image: "nats:2.10",
            Cmd:   []string{"-js"},
        },
    })
    defer natsContainer.Terminate(ctx)
    // Run tests against real NATS instance
}

Kubernetes deployments leverage JetStream’s horizontal scaling. Our StatefulSet configuration ensures message durability:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nats
spec:
  serviceName: nats
  replicas: 3
  template:
    spec:
      containers:
      - name: nats
        image: nats:2.10-alpine
        args: ["-js", "-sd", "/data"]
        volumeMounts:
        - name: data
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

Monitoring combines Prometheus metrics and OpenTelemetry traces. This snippet exposes custom business metrics:

func recordOrderMetrics() {
    prometheus.Register(orderCounter)
    http.Handle("/metrics", promhttp.Handler())
    go http.ListenAndServe(":2112", nil)
}

var orderCounter = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "orders_created_total",
        Help: "Total created orders",
    },
    []string{"status"},
)

Performance tuning involves careful configuration. These JetStream settings balance throughput and durability:

_, err := js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"ORDERS.*"},
    Retention: nats.WorkQueuePolicy,
    Replicas: 3,
    MaxMsgs:  1_000_000,
})

Common pitfalls? Message ordering guarantees trip up many developers. While JetStream maintains per-subject order, partitioned streams need special handling. Have you considered how consumer groups affect your delivery semantics?

This architecture shines in production. During a recent payment gateway outage, our system processed 14,000 orders offline, automatically reconciling when services recovered. The true test comes when components fail - that’s where event-driven designs prove their worth.

What challenges have you faced with microservices? Share your experiences below! If this approach resonates with you, like and share this article to help others build more resilient systems. Comments and questions are always welcome - let’s learn together.