Ever faced the challenge of scaling systems while maintaining reliability? That exact puzzle occupied my thoughts last month when our order system buckled under peak traffic. This pushed me to design a resilient architecture using Go, NATS JetStream, and Kubernetes—tools I’ll demonstrate today. Follow along as I share practical insights for building robust event-driven systems.

Go excels in microservices due to its concurrency model and minimal resource footprint. Consider this worker pool pattern for parallel event processing:

// internal/events/worker_pool.go
func StartEventWorkers(ctx context.Context, js jetstream.JetStream, subject string, handler func(msg *nats.Msg), workerCount int) {
    wg := &sync.WaitGroup{}
    for i := 0; i < workerCount; i++ {
        wg.Add(1)
        go func(workerID int) {
            defer wg.Done()
            cons, _ := js.CreateOrUpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{
                Durable:   fmt.Sprintf("worker-%d", workerID),
                AckPolicy: jetstream.AckExplicitPolicy,
            })
            iter, _ := cons.Messages()
            for {
                select {
                case <-ctx.Done():
                    return
                default:
                    msg, _ := iter.Next()
                    handler(msg) // Your custom logic
                    msg.Ack()
                }
            }
        }(i)
    }
    wg.Wait()
}

Notice how each worker maintains its own durable subscription? This prevents head-of-line blocking. But what happens during deployment restarts? Kubernetes liveness probes ensure recovery:

# deployments/k8s/order-service.yaml
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
readinessProbe:
  exec:
    command: ["sh", "-c", "nats-rply -s nats://$NATS_URL REQ health.check && exit 0 || exit 1"]

For event persistence, JetStream’s retention policies shine. How do we handle poison pill messages? Dead-letter queues with exponential backoff:

// internal/events/delivery.go
func processOrder(msg *nats.Msg) {
    var order models.Order
    if err := json.Unmarshal(msg.Data, &order); err != nil {
        // Move to DLQ after 3 retries
        if msg.Header.Get("Nats-Msg-Retries") > "2" {
            msg.NakWithDelay(time.Minute * 10) // Park for manual inspection
        } else {
            msg.NakWithDelay(time.Second * time.Duration(math.Pow(2, retryCount)))
        }
        return
    }
    // Processing logic
    msg.Ack()
}

Observability becomes critical in distributed systems. OpenTelemetry traces connected through context propagation:

// internal/observability/tracing.go
func PublishEventWithTrace(ctx context.Context, js jetstream.JetStream, subject string, data []byte) {
    carrier := propagation.MapCarrier{}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
    
    headers := nats.Header{}
    for k, v := range carrier {
        headers.Set(k, v[0])
    }
    
    js.PublishMsg(&nats.Msg{
        Subject: subject,
        Data:    data,
        Header:  headers,
    })
}

When deploying to Kubernetes, ConfigMaps manage environment-specific settings:

kubectl create configmap nats-config \
  --from-literal=NATS_URL=nats://nats:4222 \
  --from-literal=JETSTREAM_BUCKET=orders-prod

Circuit breakers prevent cascading failures. The gobreaker library integrates cleanly:

// internal/services/inventory.go
cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:    "inventory-db",
    Timeout: 15 * time.Second,
})
_, err := cb.Execute(func() (interface{}, error) {
    return db.ReserveInventory(order)
})

Why use JetStream over traditional queues? Its stream persistence allows replaying events after outages—essential for audit trails.

For database migrations, embed schema updates in Docker builds:

# docker/order-service/Dockerfile
COPY migrations /migrations
RUN migrate -path /migrations -database "$DB_URL" up

Testing event flows requires simulating failures. Try this chaos pattern:

// internal/testing/chaos.go
func RandomFailureMiddleware(next HandlerFunc) HandlerFunc {
    return func(msg *nats.Msg) {
        if rand.Intn(10) == 0 { // 10% failure rate
            time.Sleep(5 * time.Second) // Simulate timeout
            msg.Nak()
            return
        }
        next(msg)
    }
}

Encountered unexpected message ordering? Use JetStream’s consumer sequence tracking:

// internal/events/sequence.go
metadata, _ := msg.Metadata()
fmt.Printf("Processed seq %d in stream %s", metadata.Sequence.Stream, metadata.Stream)

This architecture processed 12,000 orders/minute during our last stress test. I encourage you to implement these patterns—they transformed our system’s resilience. Share your implementation challenges in the comments! If this helped, consider liking or sharing with others facing similar scaling hurdles. What reliability patterns have worked best for your team?