I’ve been thinking about microservices a lot lately. Not the simple kind that just handle HTTP requests, but the kind that actually talk to each other, that form complex systems capable of handling real business workflows. The kind that don’t just work in development but thrive in production. That’s why I want to share what I’ve learned about building event-driven microservices with Go, NATS, and OpenTelemetry.

Have you ever wondered how to make services communicate reliably without creating tight coupling? Event-driven architecture provides an answer. Services publish events when something meaningful happens, and other services react to those events. This creates systems that are both loosely coupled and highly cohesive.

Let’s start with the basics. NATS provides the messaging backbone. It’s incredibly fast and simple to use. Here’s how I typically set up a connection:

nc, err := nats.Connect("nats://localhost:4222",
    nats.Name("order-service"),
    nats.MaxReconnects(10),
    nats.ReconnectWait(5*time.Second))
if err != nil {
    log.Fatal("Connection failed:", err)
}
defer nc.Close()

But what happens when messages get lost or services go down? That’s where JetStream comes in. It adds persistence and reliability to NATS. I configure streams to ensure messages aren’t lost:

js, _ := nc.JetStream()
js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"orders.*"},
    Retention: nats.WorkQueuePolicy,
})

Now, let’s talk about events. I define them as simple Go structs with JSON tags:

type OrderCreated struct {
    OrderID    string    `json:"order_id"`
    UserID     string    `json:"user_id"`
    Amount     float64   `json:"amount"`
    CreatedAt  time.Time `json:"created_at"`
}

Publishing events becomes straightforward:

event := OrderCreated{
    OrderID:   "123",
    UserID:    "user-456",
    Amount:    99.99,
    CreatedAt: time.Now(),
}

msg, _ := json.Marshal(event)
js.Publish("orders.created", msg)

But how do we know what’s happening across our distributed system? This is where OpenTelemetry shines. I instrument everything to get complete visibility:

func ProcessOrder(ctx context.Context, order Order) {
    ctx, span := tracer.Start(ctx, "ProcessOrder")
    defer span.End()
    
    span.SetAttributes(
        attribute.String("order.id", order.ID),
        attribute.Float64("order.amount", order.Amount),
    )
    
    // Business logic here
}

What about errors and retries? I use circuit breakers to prevent cascading failures:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "payment-service",
    Timeout:     30 * time.Second,
    MaxRequests: 100,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

result, err := cb.Execute(func() (interface{}, error) {
    return processPayment(order)
})

The real power comes when we combine these patterns. Services become resilient, observable, and capable of handling complex workflows. They can process orders, update inventory, handle payments, and send notifications - all while maintaining clear boundaries and the ability to scale independently.

Monitoring becomes crucial in production. I export metrics to Prometheus:

http.Handle("/metrics", promhttp.Handler())
go http.ListenAndServe(":9090", nil)

And set up structured logging:

logger, _ := zap.NewProduction()
defer logger.Sync()

logger.Info("Order processed",
    zap.String("order_id", orderID),
    zap.Duration("processing_time", duration),
)

The result is a system that’s not just functional but production-ready. It handles failures gracefully, provides clear visibility into operations, and scales to meet demand. Each service focuses on its specific responsibility while contributing to the overall business workflow.

What patterns have you found most effective in your microservices journey? I’d love to hear about your experiences and challenges. If you found this helpful, please share it with others who might benefit from these approaches. Your thoughts and comments are always welcome!