I’ve been thinking a lot about how modern systems handle complexity while staying reliable. In my work with distributed systems, I’ve seen teams struggle with tangled communication between services. That’s why I’m excited to share this approach using event-driven architecture. It changes how services interact, making systems more resilient and scalable. Have you ever wondered what happens when a service goes down mid-operation?

Let me show you how to build production-ready microservices with Go, NATS JetStream, and OpenTelemetry. This combination creates systems that handle failures gracefully while providing clear visibility into operations. We’ll start with the foundation—designing events that carry meaning across service boundaries.

Consider this basic event structure in Go:

type UserCreatedEvent struct {
    UserID    string `json:"user_id"`
    Email     string `json:"email"`
    Timestamp int64  `json:"timestamp"`
}

func NewUserEvent(userID, email string) (*Event, error) {
    data := UserCreatedEvent{
        UserID:    userID,
        Email:     email,
        Timestamp: time.Now().Unix(),
    }
    return NewEvent("user.created", "user-service", userID, data)
}

Events become the language your services speak. But how do we ensure messages aren’t lost when systems fail? That’s where NATS JetStream adds persistence to event streaming. It provides durable storage and exactly-once delivery semantics.

Here’s how you might publish an event:

func (p *EventPublisher) Publish(ctx context.Context, event *Event) error {
    span := trace.SpanFromContext(ctx)
    event.AddTraceContext(ctx)
    
    data, err := json.Marshal(event)
    if err != nil {
        span.RecordError(err)
        return fmt.Errorf("event marshaling failed: %w", err)
    }
    
    ack, err := p.jetStream.Publish(event.Subject, data)
    if err != nil {
        span.RecordError(err)
        return fmt.Errorf("event publishing failed: %w", err)
    }
    
    span.SetAttributes(attribute.String("event.id", event.ID))
    return nil
}

What separates production code from prototypes? Comprehensive observability. OpenTelemetry gives us distributed tracing and metrics out of the box. When an event travels through multiple services, you can follow its entire journey.

Implementing tracing in your handlers looks like this:

func (h *UserHandler) CreateUser(ctx context.Context, req *CreateUserRequest) error {
    ctx, span := h.tracer.Start(ctx, "user.create")
    defer span.End()
    
    user, err := h.repo.CreateUser(ctx, req)
    if err != nil {
        span.RecordError(err)
        return err
    }
    
    event, err := NewUserEvent(user.ID, user.Email)
    if err != nil {
        span.RecordError(err)
        return err
    }
    
    event.AddTraceContext(ctx)
    return h.publisher.Publish(ctx, event)
}

Services need to handle partial failures without bringing down the entire system. Circuit breakers and retry mechanisms prevent cascading failures. How do you know when to retry versus when to fail fast?

Here’s a simple circuit breaker pattern:

type CircuitBreaker struct {
    failureThreshold int
    failureCount     int
    state            State
    mutex            sync.RWMutex
}

func (cb *CircuitBreaker) Execute(fn func() error) error {
    cb.mutex.RLock()
    state := cb.state
    cb.mutex.RUnlock()
    
    if state == StateOpen {
        return ErrCircuitOpen
    }
    
    err := fn()
    if err != nil {
        cb.recordFailure()
        return err
    }
    
    cb.recordSuccess()
    return nil
}

Deployment brings everything together. Containerizing services with Docker and orchestrating with Kubernetes ensures consistent environments. Monitoring with Prometheus and visualizing traces in Jaeger completes the picture.

The real test comes when things go wrong—network partitions, database outages, or resource exhaustion. Proper error handling and graceful shutdowns make the difference between a hiccup and a catastrophe.

func (s *Service) Start() error {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    
    go s.handleSignals(cancel)
    
    group, ctx := errgroup.WithContext(ctx)
    group.Go(func() error { return s.startHTTP(ctx) })
    group.Go(func() error { return s.startEventConsumer(ctx) })
    
    return group.Wait()
}

Building systems this way transforms how you think about reliability. Events create loose coupling, observability provides clarity, and resilience patterns handle the unexpected. What patterns have you found most effective in your projects?

I’d love to hear about your experiences with event-driven systems. If this approach resonates with you, please share your thoughts in the comments. Your feedback helps all of us build better systems together. Don’t forget to like and share if you found this useful!