Lately, I’ve been thinking a lot about how we build systems that can handle real-world chaos. In production, services fail, networks partition, and users expect everything to work seamlessly. That’s why I decided to explore event-driven microservices using Go, NATS JetStream, and OpenTelemetry. This combination isn’t just about writing code; it’s about crafting systems that are resilient, observable, and ready for the demands of a live environment. If you’ve ever spent nights debugging a cascading failure across services, you’ll understand why this matters.

Event-driven architecture shifts how services communicate. Instead of direct HTTP calls that can time out or create tight coupling, services emit events when something important happens. Other services listen and react. This approach improves scalability and fault tolerance. But how do we ensure these events are processed reliably, even when parts of the system are down?

Go’s concurrency model makes it a natural fit for handling multiple event streams. Goroutines and channels allow us to process messages efficiently without blocking. When I started using generics in Go 1.21, it simplified how I define and handle different event types. Here’s a snippet from a shared events package:

type EventType string

const (
    OrderCreated EventType = "order.created"
)

type BaseEvent struct {
    ID        string    `json:"id"`
    Type      EventType `json:"type"`
    Timestamp time.Time `json:"timestamp"`
}

func NewBaseEvent(eventType EventType) BaseEvent {
    return BaseEvent{
        ID:        uuid.New().String(),
        Type:      eventType,
        Timestamp: time.Now().UTC(),
    }
}

NATS JetStream adds persistence to messaging. It stores events in streams, so if a service restarts, it can pick up where it left off. Setting up a JetStream client involves creating a connection and ensuring the stream exists. What happens if the connection drops? The client reconnects automatically, which I’ve configured with backoff strategies to avoid overwhelming the server.

func NewNATSClient(url string) (*NATSClient, error) {
    nc, err := nats.Connect(url, nats.MaxReconnects(5))
    if err != nil {
        return nil, err
    }
    js, err := nc.JetStream()
    if err != nil {
        return nil, err
    }
    // Ensure stream configuration
    _, err = js.AddStream(&nats.StreamConfig{
        Name:     "EVENTS",
        Subjects: []string{"events.>"},
    })
    return &NATSClient{nc: nc, js: js}, err
}

OpenTelemetry provides the visibility needed to understand what’s happening across services. Without it, tracing a request through multiple services feels like searching for a needle in a haystack. I instrumented my services to propagate trace contexts, making it easy to follow an event from creation to processing. Have you ever wondered why a particular order took too long to process? Distributed tracing answers that.

In one project, I used the Chi router for HTTP endpoints and integrated OpenTelemetry middleware. This automatically captures spans for incoming requests. Here’s how I set up tracing in a service:

func setupTracing(serviceName string) (func(), error) {
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint())
    if err != nil {
        return nil, err
    }
    tp := trace.NewTracerProvider(trace.WithBatcher(exporter))
    otel.SetTracerProvider(tp)
    return func() { tp.Shutdown(context.Background()) }, nil
}

Error handling in distributed systems requires careful thought. I implemented retry mechanisms with exponential backoff for message processing. If a service fails to handle an event, it’s retried later. This prevents temporary issues from causing data loss. How do you decide when to give up on a message? I use dead-letter queues for events that fail repeatedly after several attempts.

Deploying with Docker Compose simplifies running the entire stack. I define services for each microservice, NATS, Prometheus, and Jaeger. Prometheus scrapes metrics from the services, and I set up alerts for unusual patterns. Observability isn’t just about debugging; it’s about proactively understanding system health.

Building this system taught me the importance of idempotency in event handlers. Services must handle duplicate events gracefully. In the inventory service, I check if an order has already been processed before reserving stock. This prevents double deductions when events are retried.

I encourage you to try building something similar. Start small, add observability early, and test failure scenarios. What challenges have you faced with microservices? Share your experiences in the comments below. If this resonates with you, please like and share this article to help others in the community.