I’ve been thinking a lot about how modern systems handle massive scale while staying resilient. In my work building distributed systems, I’ve seen firsthand how event-driven architectures can transform applications from fragile monoliths into robust, scalable solutions. That’s why I want to share my approach to creating production-ready microservices using Go, NATS JetStream, and OpenTelemetry. These technologies have become my go-to stack for building systems that not only perform well but are also observable and maintainable in real-world conditions.

When I first started with event-driven systems, I struggled with message reliability and observability. Traditional messaging queues often left me guessing about message delivery, and debugging distributed transactions felt like searching for needles in haystacks. That’s when I discovered the power combination of Go’s concurrency model, NATS JetStream’s persistence guarantees, and OpenTelemetry’s comprehensive instrumentation.

Let me walk you through building an e-commerce order processing system. We’ll create three microservices: order service for handling orders, payment service for processing transactions, and inventory service for managing stock. Each service will communicate asynchronously through events, ensuring loose coupling and independent scalability.

Setting up the development environment is straightforward with Docker Compose. Here’s a basic setup that includes NATS with JetStream enabled, Jaeger for tracing, Prometheus for metrics, and PostgreSQL for data persistence:

services:
  nats:
    image: nats:2.10-alpine
    ports: ["4222:4222", "8222:8222"]
    command: --jetstream --store_dir=/data
    volumes: [nats-data:/data]

Have you ever wondered how to structure microservices so they’re both independent and cohesive? I’ve found that a base service structure in Go provides consistency across all services while handling common concerns like connection management and graceful shutdown. Here’s a simplified version:

type BaseService struct {
    Name    string
    Logger  *zap.Logger
    NC      *nats.Conn
    JS      nats.JetStreamContext
    ctx     context.Context
    cancel  context.CancelFunc
}

func (s *BaseService) Start() error {
    ctx, cancel := context.WithCancel(context.Background())
    s.ctx, s.cancel = ctx, cancel
    // Initialize components and start processing
    return nil
}

Configuring NATS JetStream properly is crucial for production reliability. I always create streams with explicit retention policies and durability settings. For our order system, we might configure a stream like this:

stream, err := js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"orders.>"},
    Retention: nats.WorkQueuePolicy,
    MaxMsgs:  1000000,
})

What happens when messages need to be processed in order or without duplication? JetStream’s consumer configurations help here. I typically set up pull consumers with acknowledgment wait times and max delivery limits to handle various failure scenarios gracefully.

Implementing event patterns requires careful consideration of delivery semantics. For payment processing, we might use a request-reply pattern where the order service publishes an event and waits for a response from the payment service. Here’s how that looks in code:

msg, err := js.Request("payment.process", orderData, 10*time.Second)
if err != nil {
    return fmt.Errorf("payment request failed: %w", err)
}

Observability isn’t just about collecting data—it’s about making systems understandable. I integrate OpenTelemetry from day one, instrumenting both NATS interactions and business logic. This tracing setup has saved me countless hours during incidents:

func processOrder(ctx context.Context, order Order) error {
    ctx, span := tracer.Start(ctx, "processOrder")
    defer span.End()
    // Business logic here
    span.SetAttributes(attribute.String("order.id", order.ID))
}

Testing event-driven systems presents unique challenges. How do you verify that services react correctly to events? I combine unit tests with integration tests that run against real NATS instances in Docker. For critical paths, I use contract testing to ensure services agree on event schemas.

Deploying to production requires attention to health checks and resource management. Each service exposes a health endpoint that checks NATS connectivity and database status. I configure liveness and readiness probes in Kubernetes to ensure smooth deployments and failovers.

Performance optimization often comes down to tuning JetStream configurations and Go runtime parameters. I monitor memory usage carefully, especially when dealing with large message volumes. Proper connection pooling and goroutine management in Go prevent resource exhaustion under load.

When things go wrong—and they will—having good troubleshooting practices makes all the difference. I rely on correlated logs and traces across services to pinpoint issues quickly. Structured logging with request IDs helps track events through the entire system.

Building event-driven microservices has transformed how I approach system design. The combination of Go’s efficiency, NATS JetStream’s reliability, and OpenTelemetry’s visibility creates a foundation that scales with business needs while remaining manageable.

I’d love to hear about your experiences with event-driven architectures! What challenges have you faced, and how did you overcome them? If you found this helpful, please share it with others who might benefit, and leave a comment with your thoughts or questions.