Recently, I’ve been thinking a lot about how modern applications handle complexity at scale. Traditional request-response architectures often struggle with the demands of today’s distributed systems. That’s why I want to share my approach to building production-ready event-driven microservices using Go, NATS, and OpenTelemetry.

Why choose this stack? Go provides the performance and simplicity needed for high-throughput services. NATS JetStream offers reliable messaging without the overhead of heavier alternatives. OpenTelemetry gives us the observability we need to understand our distributed system’s behavior.

Have you ever wondered how to ensure your microservices can handle failures gracefully while maintaining data consistency? Event-driven architectures provide an elegant solution by decoupling services and enabling asynchronous communication.

Let me show you how we structure our events using Protocol Buffers. This approach ensures type safety and efficient serialization across our services:

message OrderCreated {
  EventMetadata metadata = 1;
  string order_id = 2;
  string customer_id = 3;
  repeated OrderItem items = 4;
  double total_amount = 5;
}

Implementing distributed tracing is crucial for understanding the flow of events through our system. Here’s how we set up OpenTelemetry with Jaeger:

func InitTracer(config Config) func() {
    exp, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint(config.JaegerEndpoint),
    ))
    if err != nil {
        log.Fatal("Failed to create Jaeger exporter:", err)
    }
    
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exp),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(config.ServiceName),
        )),
    )
    return tp.Shutdown
}

What happens when a service goes down during message processing? NATS JetStream’s persistence and acknowledgment mechanisms ensure we don’t lose critical events. Here’s how we handle message publishing with proper error handling:

func (p *Publisher) PublishEvent(ctx context.Context, subject string, event proto.Message) error {
    span := trace.SpanFromContext(ctx)
    defer span.End()

    data, err := proto.Marshal(event)
    if err != nil {
        span.RecordError(err)
        return fmt.Errorf("failed to marshal event: %w", err)
    }

    msg := nats.NewMsg(subject)
    msg.Data = data
    msg.Header = telemetry.TraceHeaders(ctx)

    _, err = p.js.PublishMsg(msg)
    if err != nil {
        span.RecordError(err)
        return fmt.Errorf("failed to publish message: %w", err)
    }

    return nil
}

Graceful shutdown is another critical aspect of production readiness. Our services need to handle termination signals properly:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), 
        syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    // Initialize services
    tracerShutdown := telemetry.InitTracer(telemetry.Config{
        ServiceName: "order-service",
        JaegerEndpoint: "http://jaeger:14268/api/traces",
    })
    defer tracerShutdown()

    natsConn, js := setupNATS()
    defer natsConn.Close()

    // Wait for shutdown signal
    <-ctx.Done()
    log.Println("Shutting down gracefully...")
}

How do we ensure our services remain healthy under load? Implementing proper health checks and metrics collection gives us the insights we need:

func healthCheckHandler(w http.ResponseWriter, r *http.Request) {
    if natsConn.Status() != nats.CONNECTED {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

The beauty of this architecture lies in its resilience. Services can fail and restart without affecting the overall system. Messages persist until they’re successfully processed, and we maintain full visibility into every event’s journey through our services.

Building with these patterns has transformed how I think about distributed systems. The combination of Go’s efficiency, NATS’ reliability, and OpenTelemetry’s observability creates a foundation that can scale with your business needs.

I’d love to hear about your experiences with event-driven architectures. What challenges have you faced, and how have you solved them? Share your thoughts in the comments below, and if you found this useful, please consider sharing it with others who might benefit from these patterns.