I’ve been working with distributed systems for several years now, and one challenge keeps resurfacing: how to build resilient services that maintain data consistency across failures. Just last month, I struggled with an order processing system that lost transactions during network partitions. That frustration sparked this project - a complete event-driven solution using Go’s concurrency features with NATS and PostgreSQL. Let me share what I’ve built.

Our architecture handles orders through four specialized services communicating via events. When you create an order, it triggers a sequence of actions: inventory checks, payment processing, and user notifications. Each service operates independently, yet they form a cohesive system. How do we ensure one failure doesn’t cascade? Let’s explore.

First, we define our core building blocks - events. These immutable records capture every state change:

type Event struct {
    ID          uuid.UUID       `json:"id"`
    Type        EventType       `json:"type"` // e.g., OrderCreatedEvent
    AggregateID uuid.UUID       `json:"aggregate_id"`
    Data        json.RawMessage `json:"data"`
    Timestamp   time.Time       `json:"timestamp"`
}

Why use JSON for data? It gives flexibility to evolve schemas without breaking consumers. Each service only cares about relevant events - the inventory service listens for OrderCreated, while notifications watch for PaymentProcessed.

For the order service, we model our domain with clear state transitions:

func (o *Order) Confirm() {
    o.Status = OrderStatusConfirmed
    o.Version++ // Optimistic concurrency control
}

This version field prevents race conditions when multiple updates occur. Have you considered what happens when two services try to update the same order simultaneously? The version acts as a guard.

Now, the event store - our system’s memory. Using PostgreSQL’s JSONB type, we achieve both persistence and query flexibility:

func (s *Store) SaveEvent(ctx context.Context, event *events.Event) error {
    _, err := s.db.Exec(ctx, `INSERT INTO events VALUES ($1,$2,$3,$4,$5)`,
        event.ID, event.Type, event.AggregateID, event.Data, event.Timestamp)
    if err != nil {
        s.logger.Error("Event save failed", zap.Error(err))
        return fmt.Errorf("event save failed: %w", err)
    }
    return nil
}

Notice the structured logging with Zap? It’s crucial for debugging distributed transactions. We attach trace IDs to correlate events across services.

Connecting services requires careful concurrency handling. Here’s how we process NATS messages safely:

msgChan := make(chan *nats.Msg, 64)
sub, _ := nc.ChanSubscribe("orders.*", msgChan)

for i := 0; i < runtime.NumCPU(); i++ {
    go func() {
        for msg := range msgChan {
            var event events.Event
            if err := json.Unmarshal(msg.Data, &event); err != nil {
                logger.Warn("Invalid message format")
                continue
            }
            handleEvent(event) // Process in isolated goroutine
        }
    }()
}

We use worker pools matching CPU cores to prevent overloading. What happens during traffic spikes? The buffered channel absorbs temporary surges while circuit breakers protect downstream services.

For database resilience, pgx connection pooling is essential:

poolConfig, _ := pgxpool.ParseConfig(DATABASE_URL)
poolConfig.MaxConns = 25 // Match DB max_connections
pool, _ := pgxpool.NewWithConfig(ctx, poolConfig)

This prevents connection storms that crash databases. Combine this with exponential backoff retries:

retryStrategy := backoff.NewExponentialBackOff()
retryStrategy.MaxElapsedTime = 30 * time.Second

err := backoff.Retry(func() error {
    return paymentService.Process(orderID)
}, retryStrategy)

Graceful shutdown completes the resilience picture. On termination signals, we:

1. Stop accepting new requests
2. Finish in-flight operations
3. Close connections cleanly

go func() {
    <-ctx.Done()
    cancel() // Propagate cancellation
    server.Shutdown(timeoutCtx)
    natsConn.Drain()
    eventStore.Close()
}()

Finally, deployment considerations. Our Docker health checks verify service readiness:

HEALTHCHECK --interval=10s --timeout=3s \
  CMD curl -f http://localhost:8080/health || exit 1

Combining these patterns creates systems that withstand real-world chaos. The event sourcing approach provides an audit trail - you can reconstruct any entity’s state by replaying events. Did you know you can use this to fix data corruption by “rewinding” bad events?

What excites me most is how these services scale independently. During holiday sales, we can add payment processors without touching inventory systems. The loose coupling prevents deployment bottlenecks.

I’d love to hear about your distributed system challenges! If this approach resonates with you, please share it with others facing similar architecture decisions. Have questions about specific implementations? Let’s discuss in the comments.