Have you ever wondered how modern applications handle thousands of simultaneous operations while staying responsive? I recently faced this challenge while designing a cloud-native system that needed to process high-volume transactions without bottlenecks. That’s when I discovered the power of combining event-driven architecture with Go’s concurrency features. Let me share how we can build robust systems using NATS, Go, and MongoDB.

First, we establish our foundation. Here’s our project structure and dependency setup:

go mod init github.com/yourusername/event-driven-microservices
mkdir -p cmd/{order,inventory,notification}-service
mkdir pkg/{events,database,messaging}

Our go.mod includes critical dependencies:

module github.com/yourusername/event-driven-microservices

go 1.21

require (
    github.com/nats-io/nats.go v1.31.0
    go.mongodb.org/mongo-driver v1.13.1
    go.uber.org/zap v1.26.0
)

For NATS connectivity, we implement resilient connection handling:

// pkg/messaging/nats_client.go
func NewNATSClient(url string, logger *zap.Logger) (*nats.Conn, error) {
    nc, err := nats.Connect(url,
        nats.MaxReconnects(5),
        nats.ReconnectWait(2*time.Second),
        nats.DisconnectErrHandler(func(c *nats.Conn, err error) {
            logger.Warn("NATS connection lost", zap.Error(err))
        }),
        nats.ReconnectHandler(func(c *nats.Conn) {
            logger.Info("NATS connection restored")
        }),
    )
    if err != nil {
        return nil, fmt.Errorf("connection failed: %w", err)
    }
    return nc, nil
}

What happens if MongoDB becomes temporarily unavailable? We handle it with connection pooling:

// pkg/database/mongo_client.go
func NewMongoClient(uri string) (*mongo.Client, error) {
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    
    client, err := mongo.Connect(ctx, options.Client().
        ApplyURI(uri).
        SetMaxPoolSize(100).
        SetMinPoolSize(10).
        SetRetryWrites(true))
    
    if err != nil {
        return nil, fmt.Errorf("database connection error: %w", err)
    }
    return client, nil
}

Our order service processes requests using this pattern:

// internal/order/service.go
func (s *OrderService) CreateOrder(order Order) error {
    if err := s.repo.Save(order); err != nil {
        return err
    }
    
    event := events.OrderCreated{
        OrderID:   order.ID,
        UserEmail: order.Email,
        Items:     order.Items,
    }
    
    return s.nats.Publish("orders.created", event)
}

How do we ensure inventory updates stay consistent across services? The inventory service listens and reacts:

// internal/inventory/service.go
func (s *InventoryService) StartListener() {
    s.nats.Subscribe("orders.created", func(msg *nats.Msg) {
        var event events.OrderCreated
        if err := json.Unmarshal(msg.Data, &event); err != nil {
            s.logger.Error("message decode failed", zap.Error(err))
            return
        }
        
        if err := s.ProcessOrder(event); err != nil {
            s.logger.Error("inventory update failed", 
                zap.String("order_id", event.OrderID), 
                zap.Error(err))
        }
    })
}

For notifications, we use a similar subscription model but add deduplication:

// internal/notification/service.go
func (s *NotificationService) SendOrderConfirmation(orderID string) error {
    key := fmt.Sprintf("notif:%s", orderID)
    if exists := s.cache.Exists(key); exists {
        return nil // Avoid duplicate notifications
    }
    
    // Send email/SMS logic here
    s.cache.Set(key, "sent", 24*time.Hour)
    return nil
}

When designing distributed systems, how do we handle partial failures? We implement idempotency keys and retry queues. Each event includes a unique UUID that services use to detect duplicates. For critical operations like payment processing, we use NATS’ durable consumers that persist message positions.

For monitoring, we integrate Prometheus metrics:

// pkg/monitoring/metrics.go
var OrdersProcessed = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "orders_processed_total",
        Help: "Total processed orders",
    },
    []string{"service", "status"},
)

func init() {
    prometheus.MustRegister(OrdersProcessed)
}

During deployment, we use health checks for Kubernetes readiness probes:

// cmd/order-service/main.go
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
    if s.nats.Status() != nats.CONNECTED || s.db.Ping() != nil {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

What makes this architecture shine? The complete decoupling of services through events. When our payment service needed upgrading, we deployed it without touching other components. During peak sales, we scaled just the inventory service to handle stock updates.

I’ve found this combination of NATS for messaging, Go for performance, and MongoDB for flexible data storage creates exceptionally maintainable systems. The event-driven approach simplifies adding new capabilities too - when we introduced a recommendation service, it simply subscribed to existing order events.

What challenges have you faced with distributed systems? I’d love to hear about your experiences in the comments. If this approach resonates with you, consider sharing it with your network. Building robust systems requires continuous learning, and your insights might help others on their journey.