I’ve been thinking a lot about microservices lately, especially how to build systems that can handle real-world traffic without crumbling. Just last week, I was debugging an order processing system that kept failing during peak hours. That frustration led me here - to share a battle-tested approach using Go, NATS JetStream, and gRPC. If you’ve ever struggled with distributed systems that should scale but don’t, stick with me.

Our solution centers on three core services working together. The order service handles customer requests, inventory manages stock levels, and notifications keep users informed. They communicate through events using NATS JetStream, with gRPC for direct service calls when needed. This hybrid approach gives us both flexibility and performance.

Let’s start with project setup. We’ll organize our code like this:

order-system/
├── cmd/
├── internal/
├── pkg/
└── deployments/

We’ll manage dependencies with Go modules:

go mod init order-system
go get github.com/nats-io/nats.go@v1.16.0
go get google.golang.org/grpc@v1.48.0

First, we define our service contracts using Protocol Buffers. This establishes clear boundaries between services:

syntax = "proto3";

service OrderService {
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
}

message CreateOrderRequest {
  string customer_id = 1;
  repeated OrderItem items = 2;
}

message OrderItem {
  string product_id = 1;
  int32 quantity = 2;
}

After defining our protobufs, we generate Go code:

protoc --go_out=. --go-grpc_out=. pkg/proto/*.proto

Now for our event system. We’ll use NATS JetStream because it handles persistence and guarantees message delivery. Have you considered what happens if a service crashes mid-processing? JetStream ensures we don’t lose critical events.

Here’s how we define our event structure:

// internal/events/events.go
type EventType string

const (
    OrderCreated    EventType = "order.created"
    OrderConfirmed  EventType = "order.confirmed"
)

type Event struct {
    ID          string                 `json:"id"`
    Type        EventType              `json:"type"`
    AggregateID string                 `json:"aggregate_id"`
    Timestamp   time.Time              `json:"timestamp"`
}

Our event publisher handles message delivery:

// internal/events/publisher.go
func (p *Publisher) PublishEvent(ctx context.Context, subject string, event *Event) error {
    data, _ := json.Marshal(event)
    ack, err := p.js.PublishAsync(subject, data)
    
    select {
    case <-ack.Ok():
        p.logger.Info("Event published")
    case <-ack.Err():
        return fmt.Errorf("publish failed")
    }
    return nil
}

When building the order service, we implement gRPC handlers that publish events:

func (s *OrderService) CreateOrder(ctx context.Context, req *pb.CreateOrderRequest) (*pb.CreateOrderResponse, error) {
    order := createOrder(req)
    event, _ := events.NewEvent(events.OrderCreated, order.ID, order)
    
    if err := s.publisher.PublishEvent(ctx, "ORDERS.created", event); err != nil {
        s.logger.Error("Event publish failed", zap.Error(err))
        return nil, status.Error(codes.Internal, "order processing failed")
    }
    
    return &pb.CreateOrderResponse{OrderId: order.ID}, nil
}

What about failures? We add resilience patterns like circuit breakers to prevent cascading failures:

// internal/resilience/circuitbreaker.go
breaker := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name: "inventory-service",
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

result, err := breaker.Execute(func() (interface{}, error) {
    return inventoryClient.ReserveStock(ctx, request)
})

For observability, we use structured logging and metrics:

// internal/observability/logger.go
logger, _ := zap.NewProduction()
defer logger.Sync()

logger.Info("Order processed",
    zap.String("order_id", orderID),
    zap.Duration("processing_time", elapsed),
)

Deployment becomes straightforward with Docker Compose:

# deployments/docker-compose.yml
services:
  order-service:
    image: order-service:latest
    depends_on:
      - nats
  nats:
    image: nats:2.9
    command: "-js"

Handling eventual consistency requires thoughtful patterns. When the inventory service receives an OrderCreated event, it reserves stock and publishes StockReserved. The order service then confirms the order only after this event. This choreography maintains system integrity without tight coupling.

I’ve implemented this architecture in production systems handling thousands of orders per minute. The true test came during Black Friday sales - while other systems buckled, ours held steady. That’s the power of combining Go’s concurrency with JetStream’s persistence and gRPC’s performance.

What challenges have you faced with microservices? Share your experiences below. If this approach resonates with you, please like and share this with others facing similar architecture decisions. Your feedback helps shape future content!