I’ve been thinking about microservices a lot lately. As distributed systems become more complex, I needed a robust way to build services that communicate efficiently while maintaining security and observability. That’s why I turned to gRPC in Go - it handles the heavy lifting of service-to-service communication while providing type safety and performance benefits. Let me share how I built production-ready services with authentication, monitoring, and resilience.

When starting with gRPC, defining clear service contracts is crucial. I used Protocol Buffers to establish the communication rules between services. For our user service, the definition specifies operations like creating users and health checks. Notice how we include timestamps and pagination parameters - these small details prevent headaches later. Have you considered how your data structures might evolve over time?

service UserService {
  rpc GetUser(GetUserRequest) returns (GetUserResponse);
  rpc HealthCheck(google.protobuf.Empty) returns (HealthCheckResponse);
}

message GetUserRequest {
  string id = 1;
}

message HealthCheckResponse {
  string status = 1;
  google.protobuf.Timestamp timestamp = 2;
}

For authentication, we defined a separate Auth service. The JWT token handling includes refresh tokens and validation endpoints. Security isn’t an afterthought - it’s baked into our communication protocol. How often do you review your token expiration policies?

Configuration management keeps services flexible across environments. I created a unified configuration loader that handles different data types and environment variables:

type Config struct {
  Server   ServerConfig
  JWT      JWTConfig
}

type JWTConfig struct {
  SecretKey      string
  AccessExpiry   time.Duration
}

func LoadConfig() (*Config, error) {
  accessExpiry, _ := strconv.Atoi(os.Getenv("JWT_ACCESS_EXPIRY"))
  return &Config{
    JWT: JWTConfig{
      SecretKey:    os.Getenv("JWT_SECRET"),
      AccessExpiry: time.Minute * time.Duration(accessExpiry),
    },
  }, nil
}

Authentication interceptors protect our gRPC endpoints. This middleware validates tokens before requests reach business logic. Notice how we extract metadata from incoming requests. What authentication edge cases might you encounter in your services?

func AuthInterceptor(ctx context.Context) (context.Context, error) {
  md, ok := metadata.FromIncomingContext(ctx)
  if !ok {
    return nil, status.Error(codes.Unauthenticated, "missing credentials")
  }

  token := md.Get("authorization")
  if len(token) == 0 {
    return nil, status.Error(codes.Unauthenticated, "invalid token")
  }
  
  claims, err := validateToken(token[0])
  if err != nil {
    return nil, status.Errorf(codes.Unauthenticated, "invalid token: %v", err)
  }
  
  return context.WithValue(ctx, userKey, claims), nil
}

Observability is non-negotiable in production. I integrated OpenTelemetry for distributed tracing across services. This snippet initializes tracing with Jaeger. Can you trace a request across multiple services in your current setup?

func InitTracing(serviceName string) func(context.Context) error {
  exporter, _ := jaeger.New(jaeger.WithCollectorEndpoint())
  provider := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
    trace.WithResource(resource.NewWithAttributes(
      semconv.SchemaURL,
      semconv.ServiceNameKey.String(serviceName),
    ),
  )
  
  otel.SetTracerProvider(provider)
  return provider.Shutdown
}

Graceful shutdown prevents data loss during deployments. This pattern listens for termination signals while allowing active connections to complete:

func RunServer(server *grpc.Server, lis net.Listener) {
  go func() {
    if err := server.Serve(lis); err != nil {
      log.Fatalf("failed to serve: %v", err)
    }
  }()

  quit := make(chan os.Signal, 1)
  signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
  <-quit
  
  log.Println("shutting down server...")
  server.GracefulStop()
  log.Println("server exited")
}

Health checks keep our services reliable. We implement a simple endpoint that reports service status. How quickly could you detect a failing service in your system?

rpc HealthCheck(google.protobuf.Empty) returns (HealthCheckResponse) {
  message HealthCheckResponse {
    string status = 1;
    google.protobuf.Timestamp timestamp = 2;
  }
}

Metrics collection with Prometheus helps track performance. We expose key metrics like request latency and error rates:

func RegisterMetrics() {
  requestCounter = prometheus.NewCounterVec(
    prometheus.CounterOpts{
      Name: "grpc_requests_total",
      Help: "Count of gRPC requests",
    },
    []string{"service", "method", "code"},
  )
  prometheus.MustRegister(requestCounter)
}

For deployment, Docker containers package our services consistently. The Dockerfile builds minimal images for secure production use:

FROM golang:1.21-alpine AS builder
RUN apk add --no-cache git
WORKDIR /app
COPY . .
RUN go build -ldflags="-s -w" -o /bin/service ./cmd/service

FROM alpine:latest
COPY --from=builder /bin/service /bin/service
ENTRYPOINT ["/bin/service"]

Building these services taught me valuable lessons. Clear contracts prevent integration issues. Security must be implemented at multiple layers. Observability requires intentional instrumentation. Resilient systems handle failures gracefully.

If you’re building microservices, consider these patterns. They’ve saved me countless debugging hours. What challenges are you facing with your distributed systems? Share your experiences below - I’d love to hear what solutions you’ve discovered. If this helped you, pass it along to others who might benefit!