Lately, I’ve been thinking about how modern applications need to handle massive scale while staying resilient and observable. That’s why I want to share my approach to building production-ready event-driven microservices using Go, NATS JetStream, and OpenTelemetry. This isn’t just theory—it’s a practical guide based on real-world experience.

Let me show you how to build systems that can handle high loads while providing clear visibility into their operations. Have you ever wondered how services stay in sync without direct communication?

We start by defining our events. Clear event definitions are crucial for maintaining consistency across services.

type OrderCreatedEvent struct {
    BaseEvent
    OrderID     string
    CustomerID  string
    Items       []OrderItem
    TotalAmount float64
}

Each event carries its own context and metadata, making them self-contained units of work. How do we ensure these events reach their destinations reliably?

NATS JetStream provides persistent messaging with exactly-once delivery semantics. Here’s how we set up our connection:

func NewEventBus(config NATSConfig) (*EventBus, error) {
    conn, err := nats.Connect(config.URL, nats.MaxReconnects(5))
    if err != nil {
        return nil, fmt.Errorf("connection failed: %w", err)
    }
    
    js, err := conn.JetStream()
    return &EventBus{conn: conn, js: js}, nil
}

But what happens when things go wrong? How do we track requests across service boundaries?

OpenTelemetry gives us distributed tracing capabilities. We instrument our services to provide full visibility:

func StartSpan(ctx context.Context, name string) (context.Context, trace.Span) {
    tracer := otel.Tracer("order-service")
    return tracer.Start(ctx, name)
}

Each service processes events independently while maintaining transaction context through trace propagation. This separation allows individual services to scale based on their specific load requirements.

Consider this message handler pattern:

func (s *OrderService) handleOrderCreated(ctx context.Context, msg *nats.Msg) error {
    ctx, span := StartSpan(ctx, "process_order")
    defer span.End()
    
    var event OrderCreatedEvent
    if err := json.Unmarshal(msg.Data, &event); err != nil {
        span.RecordError(err)
        return fmt.Errorf("failed to unmarshal: %w", err)
    }
    
    // Process order logic
    span.SetStatus(codes.Ok, "order processed")
    return nil
}

What about service resilience? We implement retries and circuit breakers to handle temporary failures:

func WithRetry(operation func() error, maxAttempts int) error {
    backoff := backoff.NewExponentialBackOff()
    return backoff.Retry(operation, backoff.WithMaxRetries(maxAttempts))
}

Monitoring is essential for production systems. We expose metrics through Prometheus:

func NewMetrics() *prometheus.Registry {
    registry := prometheus.NewRegistry()
    registry.MustRegister(prometheus.NewGoCollector())
    return registry
}

Deployment becomes straightforward with Docker Compose. Each service runs in its own container, connected through the NATS network.

The beauty of this architecture lies in its simplicity and power. Services communicate through events rather than direct API calls, reducing coupling and increasing flexibility. Can you see how this approach simplifies scaling and maintenance?

Error handling becomes more manageable when each service focuses on its specific domain. Failed messages can be replayed, and new services can join the ecosystem without disrupting existing ones.

Observability isn’t an afterthought—it’s built into every component. Traces, metrics, and logs work together to provide a complete picture of system behavior.

I encourage you to try this approach in your next project. The combination of Go’s efficiency, NATS JetStream’s reliability, and OpenTelemetry’s observability creates a powerful foundation for modern applications.

What challenges have you faced with microservices communication? I’d love to hear your thoughts and experiences. Please share this article if you found it helpful, and let me know in the comments how you approach building resilient systems.