I’ve been thinking a lot lately about how we build systems that don’t just work, but work reliably under pressure. You know that feeling when your service goes down at 2 AM because some component failed silently? That’s exactly what led me to explore production-ready event-driven architectures. The combination of NATS, Go-Kit, and distributed tracing creates something truly resilient.

Let me show you how we can build systems that handle failures gracefully while maintaining perfect visibility into what’s happening.

First, consider the core of our architecture: NATS JetStream. It’s not just a message broker—it’s the foundation for reliable event delivery. Here’s how we set up a persistent connection:

nc, err := nats.Connect("nats://localhost:4222",
    nats.RetryOnFailedConnect(true),
    nats.MaxReconnects(-1),
    nats.ReconnectWait(2*time.Second))
if err != nil {
    return fmt.Errorf("NATS connection failed: %w", err)
}

But what happens when messages start backing up? That’s where JetStream’s persistence comes in. We configure streams with retention policies that match our business needs:

_, err = js.AddStream(&nats.StreamConfig{
    Name:      "ORDERS",
    Subjects:  []string{"orders.>"},
    Storage:   nats.FileStorage,
    Retention: nats.InterestPolicy,
    MaxAge:    7 * 24 * time.Hour,
})

Now, here’s a question: how do we ensure our services can handle both success and failure scenarios? This is where Go-Kit shines. It provides the scaffolding for building robust services with clear separation of concerns.

Let me show you a typical service structure:

type orderService struct {
    repo    OrderRepository
    events  EventPublisher
    tracer  trace.Tracer
}

func (s *orderService) CreateOrder(ctx context.Context, order Order) (Order, error) {
    ctx, span := s.tracer.Start(ctx, "CreateOrder")
    defer span.End()
    
    // Business logic here
    if err := s.repo.Save(ctx, order); err != nil {
        return Order{}, fmt.Errorf("failed to save order: %w", err)
    }
    
    // Publish event
    if err := s.events.Publish(ctx, "orders.created", order); err != nil {
        // Handle event publishing failure
        log.Printf("Failed to publish event: %v", err)
    }
    
    return order, nil
}

But what good is all this if we can’t see what’s happening across services? That’s where distributed tracing transforms our debugging experience. With OpenTelemetry, we get a complete picture of request flows:

func InstrumentHTTPClient(client *http.Client, serviceName string) *http.Client {
    return otelhttp.NewClient(client,
        otelhttp.WithPropagators(propagation.TraceContext{}),
        otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string {
            return fmt.Sprintf("%s %s", r.Method, r.URL.Path)
        }))
}

Have you ever wondered how to handle partial failures without bringing down the entire system? Circuit breakers are your answer. Here’s how we implement them:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "payment-service",
    Timeout:     30 * time.Second,
    MaxRequests: 5,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

result, err := cb.Execute(func() (interface{}, error) {
    return paymentClient.Process(ctx, payment)
})

The real magic happens when we combine all these pieces. Our services become resilient, observable, and maintainable. They handle traffic spikes, network partitions, and dependent service failures without breaking a sweat.

But here’s the most important part: testing. How do we verify our system behaves correctly under various failure conditions? We create comprehensive test scenarios that simulate real-world problems:

func TestOrderService_InventoryUnavailable(t *testing.T) {
    mockInventory := &MockInventoryService{}
    mockInventory.ReserveFunc = func(ctx context.Context, productID string, quantity int) error {
        return errors.New("insufficient inventory")
    }
    
    service := NewOrderService(mockInventory, nil, nil)
    _, err := service.CreateOrder(context.Background(), testOrder)
    
    if err == nil {
        t.Error("Expected error when inventory is unavailable")
    }
}

Deployment is the final piece of the puzzle. With Docker and Kubernetes, we can ensure our services are always available and scalable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: order-service
        image: order-service:latest
        ports:
        - containerPort: 8080
        env:
        - name: NATS_URL
          value: "nats://nats:4222"
        - name: JAEGER_ENDPOINT
          value: "http://jaeger:14268/api/traces"

The beauty of this approach is that it scales from small projects to enterprise systems. Each service focuses on its domain while maintaining loose coupling through events. When something goes wrong—and it will—we have the tools to understand why and fix it quickly.

What if I told you that building such systems doesn’t have to be complicated? With the right patterns and tools, we can create architectures that are both robust and understandable.

I’d love to hear your thoughts on this approach. Have you implemented similar patterns in your projects? What challenges did you face? Share your experiences in the comments below—let’s learn from each other’s journeys in building better systems.

If you found this useful, please like and share it with others who might benefit. Your feedback helps me create better content for our community.