Recently, I found myself needing to design a system that could handle thousands of concurrent orders without breaking under pressure. The challenge wasn’t just about writing code that worked, but building something resilient, observable, and truly production-ready. This led me down the path of combining Go’s concurrency strengths with NATS JetStream’s reliable messaging and comprehensive observability tooling.

Why build an event-driven system? Think about what happens when you place an order online. Multiple services need to coordinate: processing payments, updating inventory, sending notifications. If one service fails, the entire system shouldn’t collapse. This architecture allows services to work independently while staying coordinated through events.

Let me show you how we can structure our event bus to handle this gracefully. Here’s a simplified version of our event publishing mechanism:

func (eb *EventBus) Publish(ctx context.Context, subject string, event *Event) error {
    data, err := json.Marshal(event)
    if err != nil {
        return fmt.Errorf("failed to marshal event: %w", err)
    }

    _, err = eb.js.Publish(subject, data, nats.Context(ctx))
    if err != nil {
        return fmt.Errorf("failed to publish event: %w", err)
    }
    
    eb.logger.Info("Event published successfully", 
        zap.String("subject", subject),
        zap.String("event_id", event.ID))
    return nil
}

But what happens when things go wrong? How do we ensure messages aren’t lost during network issues or service restarts? JetStream’s persistent storage and acknowledgment mechanisms provide the reliability we need. Messages are stored until they’re successfully processed and acknowledged.

Now consider this: how do we know our system is actually working correctly in production? This is where observability becomes crucial. We need to track not just whether services are running, but how they’re performing and interacting:

func instrumentedHandler(handler EventHandler, metricName string) EventHandler {
    return func(ctx context.Context, event *Event) error {
        start := time.Now()
        err := handler(ctx, event)
        
        // Record metrics
        metrics.ProcessingTime.WithLabelValues(metricName).Observe(time.Since(start).Seconds())
        if err != nil {
            metrics.ProcessingErrors.WithLabelValues(metricName).Inc()
        }
        
        return err
    }
}

Building this system taught me that error handling isn’t just about catching exceptions—it’s about designing for failure. We implement circuit breakers to prevent cascading failures and use dead-letter queues for messages that repeatedly fail processing. The goal isn’t to prevent all errors, but to ensure the system degrades gracefully and recovers automatically.

What does it take to make a service truly production-ready? It’s the combination of reliability patterns, comprehensive monitoring, and the ability to understand system behavior under load. We deploy each service in containers with health checks and use distributed tracing to follow requests across service boundaries.

The beauty of this approach is how it scales. As order volume grows, we can add more instances of payment processors or notification services without changing the core architecture. Each service focuses on its specific responsibility while communicating through well-defined events.

I’d love to hear about your experiences with building resilient systems. What challenges have you faced with event-driven architectures? Share your thoughts in the comments below, and if you found this useful, please like and share with others who might benefit from these patterns.