I’ve been building distributed systems for years, and I keep seeing the same patterns emerge. Teams start with simple REST APIs, then gradually add queues, caches, and services until they’re wrestling with a complex web of dependencies. That’s why I’m excited to share a better approach. Event-driven microservices with Go, NATS, and OpenTelemetry have transformed how I design resilient systems. This architecture handles scale gracefully while providing the observability needed for production environments.

Have you ever wondered how modern systems process thousands of orders without dropping a single one?

Let me show you how to build a production-ready e-commerce order processing system. We’ll use Go for its excellent concurrency support, NATS JetStream for reliable messaging, and OpenTelemetry for comprehensive observability. The result will be a system that handles failures gracefully and provides clear visibility into every transaction.

First, let’s talk about project structure. A clean foundation prevents technical debt from accumulating. Here’s how I organize my event-driven services:

// internal/common/events/types.go
type Event struct {
    ID          string                 `json:"id"`
    Type        EventType              `json:"type"`
    AggregateID string                 `json:"aggregate_id"`
    Version     int                    `json:"version"`
    Data        map[string]interface{} `json:"data"`
    Timestamp   time.Time              `json:"timestamp"`
    TraceID     string                 `json:"trace_id,omitempty"`
}

This event structure includes tracing information from the start. Notice how we embed OpenTelemetry trace IDs directly in events? This creates a seamless flow across service boundaries.

Why do we need multiple event types instead of a generic message format?

Different event types allow services to subscribe only to what they care about. The order service publishes “order.created” events, while payment and inventory services listen for their specific triggers. This separation keeps concerns clean and prevents unnecessary processing.

Here’s how I implement the event bus using NATS JetStream:

// pkg/eventbus/nats.go
func (b *NATSEventBus) Publish(ctx context.Context, event *events.Event) error {
    ctx, span := b.tracer.Start(ctx, "eventbus.publish")
    defer span.End()

    data, err := json.Marshal(event)
    if err != nil {
        return fmt.Errorf("failed to marshal event: %w", err)
    }

    subject := string(event.Type)
    _, err = b.js.Publish(subject, data, nats.MsgId(event.ID))
    if err != nil {
        span.RecordError(err)
        return fmt.Errorf("failed to publish event: %w", err)
    }

    b.logger.Info("event published", zap.String("event_id", event.ID))
    return nil
}

This code includes automatic tracing and logging. Every published event gets a span that follows it through the system. If something goes wrong, we can trace the entire flow from order creation to notification.

What happens when a payment service goes offline temporarily?

That’s where JetStream’s persistence shines. Messages wait in streams until consumers come back online. Combined with circuit breakers, this prevents cascading failures. Here’s how I add resilience to service consumers:

// internal/common/patterns/circuit_breaker.go
func WithCircuitBreaker(handler events.EventHandler, name string) events.EventHandler {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:        name,
        MaxRequests: 5,
        Interval:    30 * time.Second,
        Timeout:     60 * time.Second,
    })

    return func(ctx context.Context, event *events.Event) error {
        _, err := cb.Execute(func() (interface{}, error) {
            return nil, handler(ctx, event)
        })
        return err
    }
}

This wrapper protects services from being overwhelmed during outages. When failures exceed thresholds, the circuit opens and gives the system time to recover.

How do we ensure events are processed exactly once?

JetStream provides durable consumers with acknowledgments. Each service tracks its position in the stream, and messages are only marked as processed after successful handling. For critical operations, I include idempotency keys in event metadata.

Observability isn’t just about debugging—it’s about understanding system behavior. OpenTelemetry gives us both tracing and metrics in a single framework:

// internal/common/telemetry/setup.go
func InitTracing(serviceName string) (func(), error) {
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint())
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
    )
    otel.SetTracerProvider(tp)

    return func() { tp.Shutdown(context.Background()) }, nil
}

This setup sends traces to Jaeger for visualization. I can see exactly how long each service takes to process events and identify bottlenecks.

Deployment becomes straightforward with Docker Compose. All services connect to the same NATS and observability backend. Prometheus scrapes metrics from each service, while Jaeger collects traces. The entire system can run on a single server or scale across clusters.

Building event-driven microservices requires shifting from request-response to event-based thinking. Once you experience the resilience and scalability, you’ll wonder how you managed without it. The combination of Go’s performance, NATS’s reliability, and OpenTelemetry’s visibility creates systems that handle real-world loads gracefully.

I’d love to hear about your experiences with event-driven architectures. What challenges have you faced? Share your thoughts in the comments below, and if this approach resonates with you, please like and share this with your team. Let’s build more reliable systems together.