I’ve been thinking about modern e-commerce systems and how they handle thousands of transactions without breaking. What makes them so resilient? That question led me down the path of event-driven architectures. Today I’ll share how to build a robust system using Go, NATS JetStream, and OpenTelemetry – the same patterns powering high-scale platforms you use daily.

Setting up our environment begins with thoughtful organization. Our project structure separates concerns cleanly:

// go.mod
module github.com/yourorg/event-driven-ecommerce

go 1.21

require (
    github.com/nats-io/nats.go v1.31.0
    go.opentelemetry.io/otel v1.20.0
    // ... other critical dependencies
)

Messaging forms the backbone of our architecture. Here’s how we configure NATS JetStream for reliable event streaming:

func NewJetStreamManager(natsURL string) (*JetStreamManager, error) {
    nc, err := nats.Connect(natsURL,
        nats.ReconnectWait(time.Second*2),
        nats.MaxReconnects(10)
    // ... connection handling
}

func (jsm *JetStreamManager) CreateStream(ctx context.Context, config StreamConfig) error {
    _, err := jsm.js.AddStream(&nats.StreamConfig{
        Name:        config.Name,
        Subjects:    config.Subjects,
        MaxAge:      config.MaxAge,
        Replicas:    config.Replicas,
        Storage:     nats.FileStorage,
    })
    // ... error handling
}

Notice how we’ve built in automatic reconnection? That’s crucial for real-world resilience. But what happens when services need to communicate complex state changes? That’s where OpenTelemetry shines. Consider this instrumentation example:

func (s *OrderService) CreateOrder(ctx context.Context, order Order) {
    ctx, span := s.tracer.Start(ctx, "create_order")
    defer span.End()
    
    // Attach metadata to our trace
    span.SetAttributes(
        attribute.String("order.id", order.ID),
        attribute.Int("order.items", len(order.Items)),
    )
    
    // Business logic here
}

When implementing our order service, we focus on atomic operations. Each order creation emits an event that downstream services consume. But how do we prevent inventory overselling? Our inventory service uses Go’s concurrency primitives:

func (inv *Inventory) ReserveItem(itemID string, qty int) error {
    inv.mu.Lock()
    defer inv.mu.Unlock()
    
    if inv.stock[itemID] < qty {
        return errors.New("insufficient stock")
    }
    inv.stock[itemID] -= qty
    return nil
}

Payment processing demands special care. We implement the circuit breaker pattern to handle third-party failures:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name: "PaymentProcessor",
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 3
    },
})

result, err := cb.Execute(func() (interface{}, error) {
    return processor.Charge(order.Total)
})

For notifications, we use worker pools to manage throughput spikes. Notice how we limit concurrent email sends:

func (ns *NotificationService) StartWorkers(poolSize int) {
    sem := make(chan struct{}, poolSize)
    
    for msg := range ns.msgChannel {
        sem <- struct{}{}
        go func(m Message) {
            defer func() { <-sem }()
            ns.sendEmail(m)
        }(msg)
    }
}

Deploying to Kubernetes requires careful health check configuration. Our readiness probe ensures dependencies are live:

# deployment.yml
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

Throughout development, we validate behavior with contract tests. This snippet verifies event schemas:

func TestOrderCreatedEvent(t *testing.T) {
    event := OrderCreatedEvent{
        OrderID: "test-123",
        Amount:  99.99,
    }
    
    err := js.Publish("ORDERS.created", event)
    require.NoError(t, err)
    
    // Assert event structure
    assert.Equal(t, "test-123", event.OrderID)
}

Ever wonder how large systems maintain performance during traffic surges? Our approach combines horizontal scaling and careful backpressure management. The notification service’s worker pool demonstrates this – it won’t accept new messages when overloaded, preventing cascading failures.

What separates good systems from great ones? Visibility. Our OpenTelemetry integration provides distributed tracing across services. When an order fails, we see the entire journey: from HTTP request through payment processing to inventory updates. No more debugging blind spots.

I’ve shared concrete patterns from real implementations – from concurrency control to failure management. These principles apply far beyond e-commerce. Try adapting them to your next project. If this helped you understand resilient system design, share it with a colleague facing similar challenges. I’d love to hear about your implementation experiences in the comments!