Skip to content

Kafka standalone

1. Review Kafka Client Configuration

Kafka clients in Spring, including the ones using the spring.kafka.bootstrap-servers property, typically attempt reconnections when they lose connectivity. However, certain configurations can improve resilience:

  • Enable Automatic Reconnection: Ensure the client configuration includes a reconnection policy by setting spring.kafka.consumer.auto-offset-reset to "latest" or "earliest", depending on your requirements. This can help handle abrupt restarts.

  • Configure Connection Backoff Settings: 

    spring.kafka.consumer.properties.reconnect.backoff.ms: 500
spring.kafka.consumer.properties.reconnect.backoff.max.ms: 10000
spring.kafka.producer.properties.reconnect.backoff.ms: 500
spring.kafka.producer.properties.reconnect.backoff.max.ms: 10000
    These settings control the delay between reconnection attempts, which can help the client reconnect more smoothly after a restart.

2. Set Up Load Balancing for Kafka on Kubernetes

If you’re using a Kubernetes Service for Kafka (for example, a ClusterIP or NodePort service), ensure the service configuration points to all Kafka brokers in the cluster. Here’s what you can consider:

  • Headless Service for Kafka: A headless service (serviceName without a cluster IP) for Kafka allows your Kafka clients to access individual Kafka broker pods, which can improve connection resilience. Define the service in your Kubernetes manifest like this:
apiVersion: v1
kind: Service
metadata:
  name: kafka-headless
  labels:
    app: kafka
spec:
  ports:
    - port: 9092
  clusterIP: None
  selector:
    app: kafka

Adjust your spring.kafka.bootstrap-servers configuration to include the individual broker endpoints (e.g., kafka-headless-0.kafka-headless:9092,kafka-headless-1.kafka-headless:9092,...).

3. Ensure Kubernetes Readiness and Liveness Probes

Configure Kafka pods with liveness and readiness probes to ensure that Kubernetes only routes traffic to Kafka pods when they are ready. If you already have probes in place, consider fine-tuning them to handle Kafka's startup time better.

Here’s an example of Kafka liveness and readiness probes:

livenessProbe:
  exec:
    command:
      - sh
      - -c
      - "echo ruok | nc localhost 9092 | grep imok"
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  exec:
    command:
      - sh
      - -c
      - "echo ruok | nc localhost 9092 | grep imok"
  initialDelaySeconds: 10
  periodSeconds: 5

4. Update Application Retry and Exception Handling Logic

If the Kafka pod restarts trigger errors in your application, consider enhancing exception handling and retry logic in your application’s Kafka listener or producer code:

  • Retry Logic: Use Spring’s RetryTemplate for consumers or producers, which retries upon connection failure and adds resilience during transient connectivity issues.
  • Exception Handling: Configure a dedicated error handler for your Kafka listeners to catch connection errors and prevent application crashes.

Here's a simple way to add retry in Spring Kafka configuration:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory());
    factory.setRetryTemplate(new RetryTemplate());
    return factory;
}

5. Increase Timeout Settings in Kafka Client Configuration

A short Kafka connection timeout may lead to premature failures. Try increasing the session.timeout.ms and request.timeout.ms configurations in application.properties:

spring.kafka.consumer.session.timeout.ms=30000
spring.kafka.consumer.request.timeout.ms=40000
spring.kafka.producer.request.timeout.ms=40000

6. Check Kubernetes Resource Limits

If the Kafka pods or your application pods hit resource limits (CPU/memory), Kubernetes may restart the pods. To avoid this, ensure your Kafka and application pods have sufficient resource limits and requests defined.

7. Validate Network Policies and DNS Resolution in K8s

In some cases, Kubernetes network policies or DNS issues can interfere with Kafka connectivity. Confirm that your Kafka and application namespaces allow open communication on the required ports and that the Kubernetes DNS resolves the Kafka service endpoints accurately.

Summary

Implementing the above changes should improve the resilience of your application’s connection to Kafka:

  • Adjust Kafka client configurations for retry and timeout.
  • Use a headless service for better broker management.
  • Set liveness and readiness probes on Kafka.
  • Enhance application-level exception handling and retry logic.
  • Verify that Kubernetes resource limits and network policies aren’t blocking connectivity.

These steps should help minimize the frequency of errors due to Kafka restarts. Let me know if you need further clarification on any of these steps!

livenessProbe: 

    livenessProbe:
      enabled: true
      initialDelaySeconds: 5
      periodSeconds: 20
      timeoutSeconds: 3
      successThreshold: 1
      failureThreshold: 3
resources: 
  resources:
      requests:
        cpu: 500m
        memory: 1024Mi
      limits:
        cpu: 750m
        memory: 1536Mi