Lately, I’ve been thinking a lot about how modern applications need to handle massive scale while remaining reliable and observable. This isn’t just about writing code—it’s about building systems that can withstand real-world chaos. That’s why I decided to explore event-driven microservices using Go, NATS JetStream, and OpenTelemetry. The combination offers a powerful way to create resilient, scalable applications. I want to share what I’ve learned because I believe these tools can transform how we approach distributed systems.

Go’s simplicity and performance make it ideal for microservices. Its built-in concurrency features, like goroutines and channels, align perfectly with event-driven patterns. When you add NATS JetStream’s persistent messaging and OpenTelemetry’s observability, you get a robust foundation. Have you ever wondered how to ensure messages aren’t lost when a service goes down? That’s where JetStream’s durability comes in.

Let me show you how to set up the core event structure. This is the foundation of our system.

// pkg/events/event.go
package events

import (
    "encoding/json"
    "time"
    "github.com/google/uuid"
)

type Event struct {
    ID       string                 `json:"id"`
    Type     string                 `json:"type"`
    Source   string                 `json:"source"`
    Subject  string                 `json:"subject"`
    Time     time.Time              `json:"time"`
    Data     json.RawMessage        `json:"data"`
    Metadata map[string]interface{} `json:"metadata,omitempty"`
    Version  string                 `json:"version"`
    TraceID  string                 `json:"trace_id,omitempty"`
    SpanID   string                 `json:"span_id,omitempty"`
}

func NewEvent(eventType, source, subject string, data interface{}) (*Event, error) {
    dataBytes, err := json.Marshal(data)
    if err != nil {
        return nil, err
    }
    return &Event{
        ID:      uuid.New().String(),
        Type:    eventType,
        Source:  source,
        Subject: subject,
        Time:    time.Now().UTC(),
        Data:    dataBytes,
        Version: "1.0",
    }, nil
}

This structure ensures every event is self-contained and traceable. Notice how we include fields for OpenTelemetry tracing—this becomes crucial when debugging distributed workflows. What happens if an event schema changes over time? We handle that with versioning and careful design.

Connecting to NATS JetStream is straightforward. Here’s a basic setup.

// pkg/messaging/jetstream.go
package messaging

import (
    "context"
    "github.com/nats-io/nats.go"
    "go.uber.org/zap"
)

type JetStreamClient struct {
    conn   *nats.Conn
    js     nats.JetStreamContext
    logger *zap.Logger
}

func NewJetStreamClient(url string, logger *zap.Logger) (*JetStreamClient, error) {
    nc, err := nats.Connect(url)
    if err != nil {
        return nil, err
    }
    js, err := nc.JetStream()
    if err != nil {
        return nil, err
    }
    return &JetStreamClient{conn: nc, js: js, logger: logger}, nil
}

This client manages our connection and provides a context for JetStream operations. In production, you’d add retry logic and proper configuration. How do you think we should handle network failures? Implementing a circuit breaker can prevent cascading issues.

Observability is non-negotiable in distributed systems. OpenTelemetry gives us distributed tracing, which I’ve found invaluable. Here’s a snippet for instrumenting a service.

// services/order/handler.go
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
)

func processOrder(ctx context.Context, orderID string) {
    tracer := otel.Tracer("order-service")
    ctx, span := tracer.Start(ctx, "processOrder")
    defer span.End()
    
    span.SetAttributes(attribute.String("order.id", orderID))
    // Business logic here
}

This traces each operation, making it easy to follow requests across services. When something goes wrong, you can pinpoint the exact failure point. Have you considered how tracing impacts performance? With proper sampling, the overhead is minimal.

Error handling and resilience are critical. I use the gobreaker library for circuit breakers.

// pkg/resilience/circuit_breaker.go
package resilience

import (
    "github.com/sony/gobreaker"
    "time"
)

func NewCircuitBreaker(name string) *gobreaker.CircuitBreaker {
    settings := gobreaker.Settings{
        Name:        name,
        MaxRequests: 5,
        Interval:    60 * time.Second,
        Timeout:     30 * time.Second,
    }
    return gobreaker.NewCircuitBreaker(settings)
}

This prevents a failing service from overwhelming the system. Combined with retry mechanisms, it ensures our microservices can recover gracefully. What strategies do you use for handling partial failures?

Testing event-driven systems requires a different approach. I leverage testcontainers to spin up NATS in tests.

// services/order/order_test.go
package order_test

import (
    "testing"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/modules/nats"
)

func TestOrderCreation(t *testing.T) {
    ctx := context.Background()
    natsContainer, err := nats.Run(ctx, "nats:latest")
    if err != nil {
        t.Fatal(err)
    }
    defer natsContainer.Terminate(ctx)
    // Test logic here
}

This ensures our tests run against a real message broker, catching integration issues early. How do you balance unit tests with integration tests in microservices?

Deployment involves Docker and Kubernetes. Here’s a simple Dockerfile for a Go service.

FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /order-service ./services/order

FROM alpine:latest
COPY --from=builder /order-service /order-service
EXPOSE 8080
CMD ["/order-service"]

In Kubernetes, you’d add liveness and readiness probes. Monitoring with Prometheus completes the picture, giving insights into system health.

Building production-ready event-driven microservices is a journey. By combining Go’s efficiency, NATS JetStream’s reliability, and OpenTelemetry’s visibility, we create systems that are both powerful and maintainable. I’ve seen firsthand how this stack handles real-world demands, from peak loads to complex failure scenarios.

If this resonates with you, I’d love to hear your thoughts. Please like, share, or comment with your experiences—let’s learn together and push the boundaries of what we can build.