I’ve been thinking about building robust microservices lately. Why? Because modern applications demand resilience and scalability. When designing distributed systems, gRPC in Go offers powerful capabilities. But production environments require more than basic implementations. We need solid patterns for discovery, balancing, and visibility. That’s why I’m sharing these advanced techniques.

Our journey starts with Protocol Buffers. They define service contracts clearly. Look at this clean user service definition:

service UserService {
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse);
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc HealthCheck(google.protobuf.Empty) returns (HealthCheckResponse);
}

message User {
  string id = 1;
  string email = 2;
  string first_name = 3;
  string last_name = 4;
}

Notice the HealthCheck endpoint? It’s crucial for production systems. We generate Go code using protoc. This keeps our client/server implementations in sync. Ever faced versioning nightmares? Protocol Buffers prevent them.

Service discovery comes next. We use Consul for dynamic registration. Services automatically join the system when they start:

func (r *ConsulRegistry) Register(ctx context.Context) error {
  registration := &api.AgentServiceRegistration{
    ID:   r.serviceID,
    Name: r.serviceName,
    Port: r.port,
    Address: r.address,
    Check: &api.AgentServiceCheck{
      HTTP: r.checkURL,
      Interval: "10s",
      Timeout: "5s",
    },
  }
  return r.client.Agent().ServiceRegister(registration)
}

The health check URL gets polled every 10 seconds. Unhealthy services get removed automatically. How would your system behave if a node suddenly disappeared? Our resolver handles that:

func (r *ConsulResolver) watchUpdates() {
  ticker := time.NewTicker(r.updateInterval)
  for {
    select {
    case <-ticker.C:
      services, meta, _ := r.client.Health().Service(
        r.serviceName, "", true, &api.QueryOptions{WaitIndex: r.lastIndex})
      r.lastIndex = meta.LastIndex
      
      var addrs []resolver.Address
      for _, s := range services {
        addr := net.JoinHostPort(s.Service.Address, strconv.Itoa(s.Service.Port))
        addrs = append(addrs, resolver.Address{Addr: addr})
      }
      r.clientConn.UpdateState(resolver.State{Addresses: addrs})
    case <-r.ctx.Done():
      return
    }
  }
}

Load balancing needs special attention. The default round-robin approach often isn’t enough. We implement custom logic with interceptors:

type customBalancer struct {
  subConnections []balancer.SubConn
  mu             sync.Mutex
}

func (b *customBalancer) Pick(info balancer.PickInfo) (balancer.PickResult, error) {
  b.mu.Lock()
  defer b.mu.Unlock()
  
  // Custom logic: Select least loaded node
  selected := selectLeastLoaded(b.subConnections)
  return balancer.PickResult{SubConn: selected}, nil
}

What happens during traffic spikes? Our circuit breaker pattern prevents cascading failures:

func CircuitBreakerInterceptor(maxFailures uint, timeout time.Duration) grpc.UnaryClientInterceptor {
  return func(ctx context.Context, method string, req, reply interface{},
    cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
    
    breaker := circuitbreaker.New(uint(maxFailures))
    if !breaker.Allow() {
      return status.Error(codes.Unavailable, "service unavailable")
    }
    
    err := invoker(ctx, method, req, reply, cc, opts...)
    if err != nil {
      breaker.Fail()
      return err
    }
    breaker.Success()
    return nil
  }
}

Observability ties everything together. We instrument services with OpenTelemetry:

func InitTracer(serviceName string) (*sdktrace.TracerProvider, error) {
  exporter, _ := otlptracegrpc.New(ctx, otlptracegrpc.WithEndpoint("collector:4317"))
  
  tp := sdktrace.NewTracerProvider(
    sdktrace.WithBatcher(exporter),
    sdktrace.WithResource(resource.NewWithAttributes(
      semconv.SchemaURL,
      semconv.ServiceNameKey.String(serviceName),
    ),
  )
  otel.SetTracerProvider(tp)
  return tp, nil
}

Prometheus metrics give real-time insights:

func RegisterMetrics() {
  requestCounter = promauto.NewCounterVec(prometheus.CounterOpts{
    Name: "grpc_requests_total",
    Help: "Total gRPC requests",
  }, []string{"service", "method", "code"})
  
  latencyHistogram = promauto.NewHistogramVec(prometheus.HistogramOpts{
    Name: "grpc_request_duration_seconds",
    Help: "gRPC request latency",
  }, []string{"service", "method"})
}

Deployment matters. Our Dockerfiles use multi-stage builds:

FROM golang:1.21 as builder
WORKDIR /app
COPY go.mod ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o user-service ./services/user

FROM alpine:latest
COPY --from=builder /app/user-service /user-service
EXPOSE 50051
ENTRYPOINT ["/user-service"]

In Kubernetes, Istio manages service mesh capabilities. It handles mutual TLS and complex traffic routing. Have you tried canary deployments? Istio makes them straightforward.

These patterns transformed how we build microservices. They handle real-world challenges gracefully. What techniques do you use in your systems? Share your experiences below. If this helped you, consider liking or sharing with others who might benefit. Let’s discuss in the comments!