I’ve been thinking about how modern systems handle scale and complexity. Traditional approaches often struggle under real-world pressures. That’s why I want to share a robust approach using event-driven architecture. Let’s explore building production-grade microservices with Go, NATS JetStream, and OpenTelemetry. This combination delivers performance and observability you can rely on.

We’ll create an e-commerce order system. Three services will work together: order processing, inventory management, and notifications. They’ll communicate through events rather than direct calls. This keeps services independent and resilient. How might this change your system’s failure tolerance?

# Project structure
mkdir -p order-service/{cmd,internal}
mkdir -p shared/{events,nats,telemetry}

Our shared events define the contract between services. Clear event schemas prevent integration issues:

// shared/events/events.go
type EventType string

const (
    OrderCreated EventType = "order.created"
    InventoryReserved EventType = "inventory.reserved"
)

type OrderCreatedEvent struct {
    OrderID    string
    Items      []OrderItem
    Total      float64
}

type OrderItem struct {
    ProductID string
    Quantity  int
}

The NATS client handles messaging with built-in tracing. Notice how we propagate context through headers:

// shared/nats/client.go
func (c *Client) PublishEvent(ctx context.Context, subject string, event interface{}) error {
    // Start tracing span
    ctx, span := c.tracer.Start(ctx, "nats.publish")
    defer span.End()

    // Marshal event
    data, _ := json.Marshal(event)
    
    // Inject tracing headers
    headers := make(nats.Header)
    otel.GetTextMapPropagator().Inject(ctx, &NATSHeaderCarrier{headers})
    
    // Publish
    _, err := c.js.PublishMsg(&nats.Msg{
        Subject: subject,
        Data:    data,
        Header:  headers,
    })
    return err
}

For the order service, we use goroutines wisely. A worker pool pattern handles concurrency safely. What happens if a message processing fails? Our system handles it gracefully:

// order-service/internal/worker.go
func StartWorkers(ctx context.Context, numWorkers int) {
    jobs := make(chan *nats.Msg, 100)
    
    for i := 0; i < numWorkers; i++ {
        go func(id int) {
            for msg := range jobs {
                processOrder(ctx, msg)
                msg.Ack()
            }
        }(i)
    }
    
    // Message receiving logic
    sub, _ := js.ChanSubscribe("orders.*", jobs)
    <-ctx.Done()
    sub.Unsubscribe()
}

Observability is built-in, not bolted on. OpenTelemetry gives us distributed tracing across services:

// shared/telemetry/setup.go
func InitTracing() (func(), error) {
    exporter, _ := stdouttrace.New()
    
    provider := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            attribute.String("service.name", "order-service"),
        )),
    )
    
    otel.SetTracerProvider(provider)
    return func() { provider.Shutdown(context.Background()) }, nil
}

Error handling follows three key principles: retry with backoff, dead-letter queues, and circuit breakers. This snippet shows a resilient message processor:

func processOrder(ctx context.Context, msg *nats.Msg) {
    retries := 0
    maxRetries := 3
    
    for {
        err := decodeAndProcess(msg.Data)
        if err == nil {
            return
        }
        
        if retries >= maxRetries {
            sendToDLQ(msg)
            return
        }
        
        select {
        case <-time.After(time.Second * time.Duration(math.Pow(2, float64(retries)))):
            retries++
        case <-ctx.Done():
            return
        }
    }
}

Deployment readiness includes health checks and graceful shutdown. Kubernetes-friendly endpoints ensure smooth operations:

// order-service/cmd/server.go
func main() {
    // Setup server
    srv := &http.Server{Addr: ":8080"}
    
    // Health endpoint
    http.HandleFunc("/health", func(w http.ResponseWriter, _ *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    
    // Graceful shutdown
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGTERM, os.Interrupt)
    
    go func() {
        <-quit
        ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
        defer cancel()
        
        if err := srv.Shutdown(ctx); err != nil {
            log.Fatal("Forced shutdown")
        }
    }()
    
    // Start server
    if err := srv.ListenAndServe(); err != http.ErrServerClosed {
        log.Fatalf("Server failed: %v", err)
    }
}

Containerization ensures consistency across environments. This Dockerfile builds efficient Go containers:

# order-service/Dockerfile
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o /order-service ./cmd

FROM gcr.io/distroless/static-debian11
COPY --from=builder /order-service /
CMD ["/order-service"]

Message ordering matters in financial systems. JetStream’s ordered consumers prevent inventory inconsistencies:

// inventory-service/main.go
_, err := js.AddConsumer("ORDERS", &nats.ConsumerConfig{
    Durable: "inventory-processor",
    AckPolicy: nats.AckExplicitPolicy,
    DeliverPolicy: nats.DeliverAllPolicy,
    FilterSubject: "orders.created",
    MaxDeliver: 5,
})

Finally, bulkheads isolate failures. This pattern prevents one failing component from crashing the whole system:

func main() {
    // Create separate connection pools
    orderDB := createDBPool(5)
    inventoryDB := createDBPool(3)
    
    // Use different goroutine pools
    go orderProcessor(orderDB)
    go inventoryManager(inventoryDB)
}

This approach gives you scalable, observable microservices ready for production demands. The patterns shown here handle real-world challenges like partial failures and traffic spikes. What problems could this solve in your current systems? If you found this useful, share it with your team and leave a comment about your implementation experiences.