Over the past year, I’ve noticed many teams struggle to build truly resilient microservices. When my own project faced cascading failures during a peak sales event, I knew we needed a better approach. This journey led me to combine Go’s efficiency with NATS JetStream’s reliability and OpenTelemetry’s observability – a combination I’ll share with you today. Stick around to discover how these technologies solve real production challenges.

Setting up our project begins with a clear structure. We organize services in a cmd directory while sharing common components like event definitions internally. Our Go dependencies include critical packages:

// go.mod
module github.com/yourname/event-driven-microservices

go 1.21

require (
    github.com/nats-io/nats.go v1.31.0
    go.opentelemetry.io/otel v1.21.0
    github.com/sony/gobreaker v0.5.0
    github.com/prometheus/client_golang v1.17.0
)

Why spend hours debugging when you can define events properly from day one? Our event schema includes versioning and metadata for future evolution:

// internal/common/events/events.go
type BaseEvent struct {
    ID          string    `json:"id"`
    Version     int       `json:"version"`
    Timestamp   time.Time `json:"timestamp"`
}

type OrderCreated struct {
    BaseEvent
    CustomerID string  `json:"customer_id"`
    Items      []Item  `json:"items"`
}

Observability isn’t optional in distributed systems. Our OpenTelemetry setup captures traces and metrics in one initialization:

// internal/common/observability/tracing.go
func NewTracer(serviceName string) (trace.Tracer, error) {
    exporter, _ := jaeger.New(jaeger.WithCollectorEndpoint())
    provider := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        ),
    )
    return provider.Tracer(serviceName), nil
}

When implementing the order service, how do we ensure events survive failures? JetStream’s persistent streams provide the answer. Notice how we attach the trace context to messages:

// internal/order/service.go
func (s *OrderService) CreateOrder(ctx context.Context, order Order) error {
    _, span := tracer.Start(ctx, "CreateOrder")
    defer span.End()

    event := events.NewOrderCreated(order)
    msg := nats.NewMsg("ORDERS.created")
    msg.Data, _ = json.Marshal(event)
    msg.Header.Set("TraceID", span.SpanContext().TraceID().String())
    
    if _, err := js.PublishMsg(msg); err != nil {
        span.RecordError(err)
        return err
    }
    return nil
}

For payment processing, resilience becomes critical. We combine retries with circuit breakers:

// internal/payment/processor.go
func ProcessPayment(amount float64) error {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:    "PaymentProcessor",
        Timeout: 30 * time.Second,
    })

    _, err := cb.Execute(func() (interface{}, error) {
        if err := bankClient.Charge(amount); err != nil {
            return nil, err
        }
        return nil, nil
    })
    
    return err
}

Ever wonder how to track requests across services? OpenTelemetry’s Gin middleware auto-instruments HTTP routes:

// cmd/order-service/main.go
r := gin.Default()
r.Use(otelgin.Middleware("order-service"))
r.POST("/orders", orderHandler.CreateOrder)

In production, we run everything in Docker with this compose snippet:

# deployments/docker-compose.yml
services:
  nats:
    image: nats:latest
    command: "-js"
  
  jaeger:
    image: jaegertracing/all-in-one:latest

  order-service:
    build: ./cmd/order-service
    environment:
      NATS_URL: nats://nats:4222

What separates hobby projects from production systems? Instrumentation. We expose Prometheus metrics with just five lines:

// internal/common/observability/metrics.go
func NewMetrics() {
    exporter, _ := prometheus.New()
    provider := metric.NewMeterProvider(metric.WithReader(exporter))
    meter := provider.Meter("service_metrics")
    eventsCounter, _ = meter.Int64Counter("events_processed")
}

When processing events, we update metrics within our consumer logic:

// internal/inventory/consumer.go
func (c *Consumer) HandleReservation(msg *nats.Msg) {
    eventsCounter.Add(ctx, 1)
    // Inventory logic here
    msg.Ack()
}

Deploying this stack gives us Jaeger traces showing event flow between services, Prometheus dashboards tracking throughput, and alerting when circuit breakers trip. We gain confidence that orders won’t vanish during network blips.

This approach transformed our system’s reliability – no more midnight calls about lost orders. What challenges could it solve for you? If this resonates with your experiences, share it with a colleague who’s battled microservice complexity. I’d love to hear your implementation stories in the comments below!