I’ve been thinking a lot about how modern applications handle scale and complexity. The shift toward distributed systems is undeniable, and event-driven architectures offer a powerful way to build resilient, scalable services. But how do we ensure these systems remain observable and maintainable in production? That’s what led me to explore combining Go’s efficiency with NATS JetStream’s reliability and OpenTelemetry’s observability capabilities.

Go’s concurrency model makes it ideal for handling high-throughput event processing. The language’s simplicity and performance characteristics align perfectly with the demands of microservices. When you combine this with NATS JetStream’s persistent streaming capabilities, you create a foundation that can handle real-world workloads without dropping messages or losing state.

Setting up NATS JetStream provides durable message storage and exactly-once delivery semantics. Here’s how you might configure a basic JetStream connection:

nc, err := nats.Connect("nats://localhost:4222")
if err != nil {
    log.Fatal("Connection failed:", err)
}

js, err := nc.JetStream(nats.PublishAsyncMaxPending(256))
if err != nil {
    log.Fatal("JetStream context failed:", err)
}

// Create stream with retention policy
_, err = js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"orders.*"},
    MaxAge:   24 * time.Hour,
})

What happens when messages start flowing between services at scale? That’s where OpenTelemetry becomes invaluable. Distributed tracing gives you visibility across service boundaries, helping you understand performance bottlenecks and failure points.

Implementing tracing in Go services is straightforward:

func initTracer() (*sdktrace.TracerProvider, error) {
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://localhost:14268/api/traces"),
    ))
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("order-service"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

Building resilient services means anticipating failure. Circuit breakers and retry mechanisms prevent cascading failures. The gobreaker package provides an excellent implementation:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "payment-service",
    Timeout:     30 * time.Second,
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

result, err := cb.Execute(func() (interface{}, error) {
    return processPayment(order)
})

Testing event-driven systems requires simulating different scenarios. How do you verify that services react correctly to various event types? Mock NATS servers and structured test cases help ensure reliability:

func TestOrderProcessing(t *testing.T) {
    s := RunBasicJetStreamServer()
    defer s.Shutdown()

    nc, js := jsClient(t, s)
    defer nc.Close()

    // Test order creation and event publication
    order := createTestOrder()
    err := js.Publish("orders.created", order.Marshal())
    require.NoError(t, err)

    // Verify downstream effects
    // ... test assertions
}

Deployment considerations include health checks, metrics collection, and proper resource allocation. Prometheus metrics give you insight into system behavior:

func initMetrics() {
    http.Handle("/metrics", promhttp.Handler())
    go http.ListenAndServe(":9090", nil)

    ordersProcessed = promauto.NewCounter(prometheus.CounterOpts{
        Name: "orders_processed_total",
        Help: "Total number of processed orders",
    })
}

Monitoring production systems requires alerting on key metrics like message backlog, processing latency, and error rates. These indicators help you maintain system health and respond to issues before they affect users.

Building production-ready event-driven systems involves balancing performance, reliability, and observability. The combination of Go, NATS JetStream, and OpenTelemetry provides a solid foundation for systems that can scale while remaining maintainable.

What challenges have you faced with event-driven architectures? I’d love to hear your experiences and solutions. If you found this useful, please share it with others who might benefit from these patterns.