I’ve been thinking about how modern systems handle complex transactions while staying reliable and observable. This led me to explore event-driven architectures using Go’s efficiency. Why settle for fragile systems when we can build robust ones?

Our setup uses NATS JetStream for persistent messaging and OpenTelemetry for visibility. Here’s how we implement it:

Shared Event Definitions
We start by defining events in a shared package. This ensures all services speak the same language:

// shared/events/events.go
type OrderCreatedEvent struct {
    BaseEvent
    Data struct {
        OrderID    string 
        CustomerID string
        Items      []OrderItem
    }
}

func NewOrderCreated(orderID string) OrderCreatedEvent {
    return OrderCreatedEvent{
        BaseEvent: NewBaseEvent("order.created"),
        Data: struct{ OrderID string }{OrderID: orderID},
    }
}

Observability Setup
OpenTelemetry gives us tracing and metrics. Notice how we instrument handlers:

// shared/observability/telemetry.go
func InitTracer(serviceName string) (func(), error) {
    exporter, _ := jaeger.New(jaeger.WithCollectorEndpoint())
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )
    otel.SetTracerProvider(tp)
    
    return func() { tp.Shutdown(context.Background()) }, nil
}

// In service handler
func (s *OrderService) CreateOrder(ctx context.Context) {
    ctx, span := tracer.Start(ctx, "CreateOrder")
    defer span.End()
    // ... business logic
}

JetStream Configuration
Reliable messaging requires proper stream setup. We define this during initialization:

// shared/messaging/jetstream.go
js, _ := nc.JetStream()
js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"order.>"},
    Retention: nats.WorkQueuePolicy,
})

Service Implementation
Services follow a clear pattern. The payment service demonstrates error handling:

// services/payment/handler.go
func (p *PaymentProcessor) HandlePaymentRequested(ctx context.Context, msg *nats.Msg) {
    event := events.PaymentRequestedEvent{}
    if err := events.Unmarshal(msg.Data, &event); err != nil {
        p.logger.Error("Unmarshal failed", zap.Error(err))
        return
    }
    
    if err := p.ProcessPayment(ctx, event); err != nil {
        p.logger.Warn("Payment failed", zap.String("order", event.Data.OrderID))
        // Publish PaymentFailedEvent
    }
}

Resilience Patterns
What happens when services fail? We implement retries with exponential backoff:

// services/order/saga.go
func (s *SagaCoordinator) Compensate(orderID string) {
    retryPolicy := backoff.NewExponentialBackOff()
    err := backoff.Retry(func() error {
        return s.undoInventoryReservation(orderID)
    }, retryPolicy)
    if err != nil {
        s.logger.Error("Compensation failed", zap.String("order", orderID))
    }
}

Deployment Setup
Docker Compose ties everything together. Notice the health checks:

# docker-compose.yml
services:
  order-service:
    image: order-service:v1
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
  nats:
    image: nats:jetstream
    command: "-js"

Observability in Action
When an order flows through services, we see the entire journey in Jaeger. How much easier does debugging become when you follow a transaction across 5 services?

Performance Optimizations
We batch inventory updates to reduce database load:

// services/inventory/service.go
func (i *InventoryManager) batchUpdate() {
    ticker := time.NewTicker(5 * time.Second)
    for {
        select {
        case <-ticker.C:
            i.flushReservations()
        }
    }
}

Building this revealed interesting insights. Go’s lightweight goroutines handle event processing efficiently. JetStream’s persistence ensures no message loss during restarts. OpenTelemetry’s auto-instrumentation captures critical metrics without boilerplate.

The real test came when simulating network partitions. Our compensating transactions kicked in, reversing incomplete operations. Health checks quickly identified troubled containers.

What separates this from a POC? Production concerns: resource limits in Docker, connection pooling for NATS, and rigorous tracing. We included structured logging with Zap - crucial when debugging distributed systems.

Try running docker-compose up then creating an order. Watch services communicate through events. Notice how the audit service captures every state change. See tracing IDs flow between services in logs.

This approach scales well. Add new services by subscribing to events. Need to handle peak loads? Scale consumers horizontally. JetStream’s load balancing distributes work automatically.

I’m curious - how would you modify this for your use case? Share your thoughts below. If this helped you, consider sharing it with others facing similar challenges.