I’ve been thinking a lot lately about how modern applications need to handle massive scale and remain resilient under pressure. That’s what led me to explore event-driven microservices—a way to build systems that are not only scalable but also robust and observable. In this article, I’ll guide you through creating production-ready event-driven microservices using Go, NATS JetStream, and OpenTelemetry. Let’s get started.

Event-driven architectures allow services to communicate asynchronously, which means they can handle high loads without blocking. Go, with its simplicity and powerful concurrency model, is an excellent choice for building these services. Combine that with NATS JetStream for durable messaging and OpenTelemetry for observability, and you have a solid foundation for any distributed system.

Setting up the project structure is the first step. Here’s a simple way to organize your code:

order-system/
├── cmd/
│   ├── order-service/
│   ├── inventory-service/
│   └── ...
├── internal/
│   ├── events/
│   └── shared/
└── pkg/
    └── messaging/

For dependencies, your go.mod might look like this:

module github.com/example/order-system

go 1.21

require (
    github.com/nats-io/nats.go v1.31.0
    go.opentelemetry.io/otel v1.21.0
    // other dependencies...
)

NATS JetStream provides persistent messaging, which is crucial for ensuring no events are lost. Have you ever wondered how to set up a reliable stream that survives service restarts? Here’s a snippet to create a JetStream connection in Go:

nc, err := nats.Connect("nats://localhost:4222", nats.MaxReconnects(-1))
if err != nil {
    log.Fatal("Connection failed:", err)
}

js, err := nc.JetStream()
if err != nil {
    log.Fatal("JetStream context failed:", err)
}

// Create a stream
_, err = js.AddStream(&nats.StreamConfig{
    Name:     "ORDERS",
    Subjects: []string{"orders.>"},
})

OpenTelemetry helps you understand what’s happening inside your services. By integrating tracing, you can follow a request as it moves through your system. How do you add tracing to a Go service? It’s simpler than you might think:

func main() {
    tp := trace.NewTracerProvider()
    defer tp.Shutdown()

    otel.SetTracerProvider(tp)

    http.Handle("/order", otelhttp.NewHandler(
        http.HandlerFunc(handleOrder), "order_endpoint",
    ))

    log.Println("Starting server on :8080")
    http.ListenAndServe(":8080", nil)
}

Building the order service involves accepting orders and publishing events. Here’s a basic example:

func handleOrder(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    span := trace.SpanFromContext(ctx)
    defer span.End()

    var order Order
    if err := json.NewDecoder(r.Body).Decode(&order); err != nil {
        http.Error(w, "Invalid input", http.StatusBadRequest)
        return
    }

    // Publish an event
    js.Publish("orders.created", order.ToBytes())
    w.WriteHeader(http.StatusAccepted)
}

What happens when a service fails or becomes unresponsive? Implementing patterns like circuit breakers can prevent cascading failures. Using the gobreaker library, you can wrap external calls:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name: "PaymentService",
    Timeout: 5 * time.Second,
})

_, err := cb.Execute(func() (interface{}, error) {
    return processPayment(order)
})

Testing is key to ensuring your services work as expected. Write unit tests for event handlers and integration tests to verify service interactions. For example:

func TestOrderHandler(t *testing.T) {
    req := httptest.NewRequest("POST", "/order", strings.NewReader(`{"id":"123"}`))
    w := httptest.NewRecorder()

    handleOrder(w, req)
    if w.Code != http.StatusAccepted {
        t.Errorf("Expected status 202, got %d", w.Code)
    }
}

Deploying your services requires proper health checks and monitoring. Use tools like Docker and Kubernetes to manage your microservices. A basic health check endpoint in Go:

func healthCheck(w http.ResponseWriter, r *http.Request) {
    if isHealthy() {
        w.WriteHeader(http.StatusOK)
    } else {
        w.WriteHeader(http.StatusServiceUnavailable)
    }
}

Observability isn’t just about tracing—it also includes metrics and logs. OpenTelemetry can help you collect all three. Here’s how to set up a metric:

meter := otel.Meter("order_service")
orderCounter, _ := meter.Int64Counter("orders.processed")

orderCounter.Add(ctx, 1)

Building event-driven microservices with Go, NATS JetStream, and OpenTelemetry gives you a system that’s scalable, resilient, and easy to monitor. I hope this guide helps you in your next project. If you found it useful, feel free to share your thoughts in the comments or pass it along to others who might benefit. Happy coding!