I’ve been thinking a lot about how modern applications handle scale and complexity lately. After building several distributed systems that struggled under load, I realized event-driven architectures with proper tooling could solve many of these challenges. Today, I want to share my approach to creating production-ready microservices using NATS, Go, and Kubernetes.

Have you ever faced a situation where a single service failure brought down your entire system?

Let me show you how we can build something more resilient. I’ll walk through creating an e-commerce order processing system where services communicate through events rather than direct calls. This approach keeps our system loosely coupled and scalable.

First, we need a reliable event bus. NATS JetStream provides persistent messaging with exactly-once delivery semantics. Here’s how I set up the core event system in Go:

// Event definition
type Event struct {
    ID          string                 `json:"id"`
    Type        string                 `json:"type"`
    AggregateID string                 `json:"aggregate_id"`
    Data        map[string]interface{} `json:"data"`
    Timestamp   time.Time              `json:"timestamp"`
}

func NewOrderCreatedEvent(orderID string, items []Item) *Event {
    return &Event{
        ID:          uuid.New().String(),
        Type:        "order.created",
        AggregateID: orderID,
        Data:        map[string]interface{}{"items": items},
        Timestamp:   time.Now().UTC(),
    }
}

What happens when multiple services need to react to the same event?

We use NATS JetStream to ensure events are processed reliably. Here’s how I configure the connection:

// NATS connection with resilience
nc, err := nats.Connect("nats://localhost:4222",
    nats.ReconnectWait(2*time.Second),
    nats.MaxReconnects(10),
    nats.DisconnectErrHandler(func(nc *nats.Conn, err error) {
        log.Printf("Disconnected: %v", err)
    }),
)
if err != nil {
    return nil, fmt.Errorf("failed to connect: %w", err)
}

Building the order service taught me important lessons about error handling. Services must handle temporary failures gracefully. I implement retry logic with exponential backoff:

// Retry with backoff
func ProcessWithRetry(ctx context.Context, operation func() error) error {
    backoff := backoff.NewExponentialBackOff()
    backoff.MaxElapsedTime = 2 * time.Minute
    
    return backoff.Retry(operation, backoff)
}

Have you considered what happens when downstream services are unavailable?

Circuit breakers prevent cascading failures. I use the gobreaker package to implement this pattern:

// Circuit breaker for payment service
var paymentCircuit = gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:    "PaymentService",
    Timeout: 30 * time.Second,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

func ProcessPayment(ctx context.Context, payment Payment) error {
    _, err := paymentCircuit.Execute(func() (interface{}, error) {
        return nil, paymentService.Process(ctx, payment)
    })
    return err
}

Observability is crucial in distributed systems. I integrate OpenTelemetry to trace requests across services:

// Tracing setup
func InitTracing(serviceName string) (*trace.TracerProvider, error) {
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint())
    if err != nil {
        return nil, err
    }
    
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
    )
    
    otel.SetTracerProvider(tp)
    return tp, nil
}

What about testing these distributed interactions?

I use testcontainers to spin up real dependencies during tests. This approach gives me confidence that services work together correctly:

// Integration test setup
func TestOrderCreation(t *testing.T) {
    ctx := context.Background()
    
    natsContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
        ContainerRequest: testcontainers.ContainerRequest{
            Image: "nats:jetstream",
            ExposedPorts: []string{"4222/tcp"},
        },
    })
    // Test implementation continues...
}

Deploying to Kubernetes requires careful configuration. I use Helm charts to manage the NATS cluster and ensure proper resource allocation:

# Kubernetes deployment for order service
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: order-service:latest
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "500m"

In production, I’ve found that proper monitoring saves countless hours. I expose metrics using Prometheus and set up alerts for key business indicators:

// Metrics collection
func RecordOrderMetrics(orderValue float64) {
    orderCounter.Inc()
    orderValueHistogram.Observe(orderValue)
    revenueGauge.Add(orderValue)
}

Have you ever wondered how to handle distributed transactions across microservices?

Saga patterns help maintain data consistency without distributed locks. I implement compensation actions for rollback scenarios:

// Saga step with compensation
type CreateOrderSaga struct {
    steps []SagaStep
}

func (s *CreateOrderSaga) Execute(ctx context.Context) error {
    for _, step := range s.steps {
        if err := step.Execute(ctx); err != nil {
            return s.Compensate(ctx)
        }
    }
    return nil
}

Performance optimization comes from experience. I’ve learned to tune NATS stream configurations based on traffic patterns:

// Stream configuration for high throughput
streamConfig := jetstream.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"orders.>"},
    Retention: jetstream.WorkQueuePolicy,
    MaxMsgs:  1000000,
    Replicas: 3,
}

Remember that building production systems involves trade-offs. I prioritize observability over premature optimization and test failure scenarios rigorously.

What challenges have you faced with microservices communication?

I hope this guide helps you build more robust systems. If you found these insights valuable, please like and share this article. I’d love to hear about your experiences in the comments—what patterns have worked well in your projects?