Recently, I found myself building a complex order processing system. The challenge wasn’t just about writing code—it was about creating services that could communicate reliably at scale, handle failures gracefully, and provide clear visibility into their operations. This led me to explore a powerful combination: Go for performance, NATS for messaging, and OpenTelemetry for observability.

Why choose this stack? Go’s lightweight concurrency model makes it ideal for handling thousands of simultaneous events. NATS JetStream provides durable messaging with exactly-once delivery semantics. OpenTelemetry offers standardized instrumentation for tracing and metrics across all services. Together, they form a robust foundation for production systems.

Let me show you how to set up the core event system. We start by defining our event types and structures:

type EventType string

const (
    OrderCreated EventType = "order.created"
    OrderValidated EventType = "order.validated"
)

type Event struct {
    ID        string
    Type      EventType
    Source    string
    Timestamp time.Time
    Data      json.RawMessage
}

Have you ever wondered how to ensure events are properly traced across service boundaries? OpenTelemetry solves this by propagating trace context through event metadata. Here’s how we instrument our event publisher:

func (p *NATSPublisher) Publish(ctx context.Context, subject string, event *Event) error {
    ctx, span := tracer.Start(ctx, "event.publish")
    defer span.End()
    
    // Propagate trace context
    event.Metadata["trace_id"] = span.SpanContext().TraceID().String()
    
    data, _ := json.Marshal(event)
    _, err := p.js.Publish(subject, data)
    return err
}

Building resilient services requires handling failures gracefully. What happens when a downstream service is temporarily unavailable? Circuit breakers prevent cascading failures:

func NewPaymentService() *PaymentService {
    return &PaymentService{
        breaker: gobreaker.NewCircuitBreaker(gobreaker.Settings{
            Name:     "payment-service",
            Timeout:  30 * time.Second,
            ReadyToTrip: func(counts gobreaker.Counts) bool {
                return counts.ConsecutiveFailures > 5
            },
        }),
    }
}

For event processing, Go’s concurrency patterns shine. Here’s a robust event handler using worker pools:

func (s *OrderService) StartWorkers(ctx context.Context, subject string) {
    for i := 0; i < s.workerCount; i++ {
        go func(workerID int) {
            for {
                select {
                case <-ctx.Done():
                    return
                default:
                    msg, _ := s.js.SubscribeSync(subject)
                    s.processMessage(ctx, msg)
                }
            }
        }(i)
    }
}

Deployment and monitoring complete the picture. Docker containers ensure consistency, while Prometheus collects metrics exposed by each service. The key is instrumenting everything—from HTTP handlers to database queries:

func InstrumentHTTPHandler(handler http.Handler) http.Handler {
    return promhttp.InstrumentHandlerCounter(
        httpRequestsTotal,
        promhttp.InstrumentHandlerDuration(
            httpRequestDuration,
            handler,
        ),
    )
}

This architecture has served me well in production environments. The combination of Go’s efficiency, NATS’ reliability, and OpenTelemetry’s visibility creates systems that are both performant and maintainable. The patterns shown here can scale from small projects to enterprise systems handling millions of events daily.

I’d love to hear about your experiences with event-driven architectures. What challenges have you faced? Share your thoughts in the comments below, and if you found this useful, please like and share with others who might benefit from these patterns.