I’ve been thinking about resilient microservices lately. Why? Because last month, a minor database hiccup cascaded through our payment system, causing hours of downtime. That painful experience led me to build robust event-driven systems using Go, NATS JetStream, and OpenTelemetry. Let me share what I’ve learned about creating production-ready systems that handle failure gracefully.

Go’s concurrency model makes it ideal for event processing. When combined with NATS JetStream’s persistence features, you get reliable message delivery without sacrificing speed. Did you know JetStream can handle millions of messages per second while guaranteeing delivery? That’s why I chose it over Kafka for this implementation.

First, let’s set up our environment. We’ll organize our project with clear separation of concerns:

# Project structure
order-system/
├── cmd/          # Service entry points
├── internal/     # Domain logic
├── pkg/          # Shared packages
└── deployments/  # Infrastructure configs

Install essential dependencies:

go get github.com/nats-io/nats.go
go get go.opentelemetry.io/otel
go get github.com/sony/gobreaker

Our event schema forms the communication backbone. Notice how we version events and include metadata for tracing:

// pkg/events/types.go
type BaseEvent struct {
    ID        string            `json:"id"`
    Type      string            `json:"type"` // e.g., "order.created"
    Timestamp time.Time         `json:"timestamp"`
    Version   string            `json:"version"`
    Metadata  map[string]string `json:"metadata,omitempty"` 
}

type OrderCreatedEvent struct {
    BaseEvent
    Data struct {
        OrderID    string  `json:"order_id"`
        ProductID  string  `json:"product_id"`
        Quantity   int     `json:"quantity"`
    } `json:"data"`
}

For the messaging layer, we create a robust NATS client. How do we handle network disruptions? Automatic reconnections with exponential backoff:

// pkg/messaging/nats.go
func NewNATSClient() (*nats.Conn, error) {
    nc, err := nats.Connect("nats://localhost:4222",
        nats.ReconnectWait(2*time.Second),
        nats.MaxReconnects(10),
        nats.DisconnectErrHandler(func(_ *nats.Conn, err error) {
            log.Printf("Disconnected: %v", err)
        }),
        nats.ReconnectHandler(func(nc *nats.Conn) {
            log.Printf("Reconnected to %s", nc.ConnectedUrl())
        }))
    if err != nil {
        return nil, err
    }
    return nc, nil
}

Now let’s implement a consumer with circuit breaking. Why use circuit breakers? They prevent overwhelmed services from being bombarded during outages:

// internal/order/processor.go
func StartConsumer(ctx context.Context) {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:    "OrderProcessor",
        Timeout: 30 * time.Second,
    })

    _, _ = nc.JetStream().Subscribe("order.created", func(msg *nats.Msg) {
        result, err := cb.Execute(func() (interface{}, error) {
            return processOrder(msg)
        })

        if err != nil {
            // Handle open circuit state
            msg.Nak() // Negative acknowledgment
            return
        }
        msg.Ack() // Acknowledge message
    })
}

For observability, we integrate OpenTelemetry. Notice how we propagate traces through NATS headers:

// pkg/telemetry/tracing.go
func InjectTrace(ctx context.Context, headers nats.Header) {
    propagator := otel.GetTextMapPropagator()
    propagator.Inject(ctx, propagation.HeaderCarrier(headers))
}

func StartSpan(ctx context.Context, name string) (context.Context, trace.Span) {
    return otel.Tracer("order-service").Start(ctx, name)
}

In the order service, we publish events with tracing context:

// internal/order/service.go
func CreateOrder(order Order) error {
    ctx, span := StartSpan(context.Background(), "CreateOrder")
    defer span.End()

    event := OrderCreatedEvent{
        BaseEvent: BaseEvent{ID: uuid.NewString()},
        Data:      order,
    }

    msg := nats.NewMsg("order.created")
    InjectTrace(ctx, msg.Header)
    msg.Data, _ = json.Marshal(event)

    return nc.PublishMsg(msg)
}

Deployment matters too. Our docker-compose ensures all components talk to each other:

# deployments/docker-compose.yml
services:
  order-service:
    image: orders:latest
    environment:
      NATS_URL: nats://nats:4222
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  
  nats:
    image: nats:latest
    command: "-js" # Enable JetStream

For message consumers, we implement graceful shutdown. What happens when Kubernetes terminates a pod? We stop accepting new messages but finish processing current ones:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
    defer stop()

    consumer := StartConsumer(ctx)

    <-ctx.Done()
    log.Println("Shutting down...")
    consumer.Drain() // Finish processing in-flight messages
}

We’ve covered the essentials: event design, resilient messaging, distributed tracing, and deployment patterns. But remember, production readiness requires testing failure scenarios. Can your system handle a 10x traffic spike? What happens when NATS restarts?

I’d love to hear your experiences with event-driven architectures. What challenges have you faced? Share your thoughts in the comments below, and if this helped you, consider sharing with your team!