I’ve been working with microservices for years, and one question keeps coming up: how do we build systems that are not just functional but truly production-ready? Recently, I found myself debugging a complex issue across multiple services, and that experience solidified my belief in event-driven architectures with proper observability. Today, I want to share how we can create robust systems using NATS, Go, and distributed tracing.

Have you ever struggled to trace a request through multiple services?

Let me walk you through building an e-commerce order processing system. We’ll use NATS for messaging because its simplicity and performance make it ideal for event-driven patterns. Go provides the concurrency features and efficiency we need for high-throughput services.

First, we need to define our event schemas. Why use Protocol Buffers instead of JSON? Protocol Buffers offer better performance and type safety. Here’s how we define our core events:

message OrderCreated {
  string order_id = 1;
  string customer_id = 2;
  repeated OrderItem items = 3;
  double total_amount = 4;
  google.protobuf.Timestamp created_at = 5;
  string trace_id = 6;
}

Generating Go code from this is straightforward with the protoc compiler. This gives us strongly-typed events that we can serialize efficiently across our services.

Now, let’s talk about distributed tracing. Without proper tracing, debugging distributed systems becomes a nightmare. We’ll use OpenTelemetry with Jaeger to track requests across service boundaries. Here’s how I initialize tracing in my services:

func InitTracer(config TracingConfig) (func(context.Context) error, error) {
    exp, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint(config.JaegerEndpoint),
    ))
    if err != nil {
        return nil, fmt.Errorf("failed to create jaeger exporter: %w", err)
    }

    tp := tracesdk.NewTracerProvider(
        tracesdk.WithBatcher(exp),
        tracesdk.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(config.ServiceName),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp.Shutdown, nil
}

What happens when a service becomes unavailable? Circuit breakers prevent cascading failures. I’ve integrated gobreaker with our NATS client to handle such scenarios gracefully.

Our NATS client needs to be resilient. Here’s how we handle connections and implement circuit breaking:

type NATSClient struct {
    conn          *nats.Conn
    js            nats.JetStreamContext
    circuitBreaker *gobreaker.CircuitBreaker
}

func (nc *NATSClient) PublishWithContext(ctx context.Context, subject string, data []byte) error {
    _, span := tracing.StartSpan(ctx, "nats.publish")
    defer span.End()

    operation := func() (interface{}, error) {
        return nil, nc.js.Publish(subject, data)
    }
    
    _, err := nc.circuitBreaker.Execute(operation)
    if err != nil {
        span.SetStatus(codes.Error, "publish failed")
        span.RecordError(err)
    }
    return err
}

When building the order service, we need to consider what happens if payment processing fails. Saga patterns help manage distributed transactions across services. We publish events like OrderCreated, which triggers PaymentRequested, and so on.

Testing event-driven systems requires a different approach. How do you verify that events are published and processed correctly? I use integration tests that spin up NATS and verify event flows.

Deployment becomes simpler with Docker Compose. Our docker-compose.yml includes NATS, Jaeger, and Prometheus. Each service exports metrics that Prometheus scrapes, and Grafana dashboards give us real-time visibility.

Monitoring production services means watching error rates and latency. I’ve configured alerts in Prometheus when circuit breakers open or when tracing shows high latency spans.

Building this system taught me valuable lessons about fault tolerance. Services must handle partial failures and retry gracefully. Dead letter queues in NATS help manage failed messages.

What patterns have you found effective for handling failures in distributed systems?

As we wrap up, remember that production readiness isn’t just about code. It’s about monitoring, resilience, and maintainability. The combination of NATS, Go, and distributed tracing gives us a solid foundation for building systems that can scale and recover from failures.

If you found this helpful, please like and share this article. I’d love to hear about your experiences in the comments—what challenges have you faced with event-driven architectures?