Skip to content

Keda

Keda is a Kubernetes-based event-driven autoscaler that can scale applications based on various triggers, such as message queue length, CPU utilization, and more. It allows you to define scaling rules and automatically adjusts the number of replicas of your application based on the defined metrics.

Key Features

  • Event-driven scaling: Keda can scale your applications based on events from various sources, such as Kafka, RabbitMQ, and more.
  • Custom metrics: You can define custom metrics for scaling, allowing for more granular control over your application's scaling behavior.
  • Seamless integration: Keda integrates easily with existing Kubernetes deployments and can be used alongside other Kubernetes tools.

🚀 KEDA Scale in Unibeam Applications

KEDA (Kubernetes Event-Driven Autoscaler) is a powerful tool used in Unibeam microservices to enable dynamic, event-driven scaling based on real-time metrics such as Kafka lag, CPU, and memory utilization. This approach ensures that each service can automatically adjust its replica count to meet demand, optimizing resource usage and maintaining high availability.

⚡ How KEDA Scale Works

KEDA monitors defined triggers (e.g., Kafka consumer lag, CPU, memory) and automatically scales the number of pods for each deployment. This is especially useful for event-driven workloads, such as those processing messages from Kafka topics.

🧩 Example: KEDA ScaledObject for sim-service

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sim-service-scaler
  namespace: sim-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sim-il-prod-sim-service
  minReplicaCount: 6
  maxReplicaCount: 15
  pollingInterval: 15
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
      consumerGroup: sia.api-sim.service.consumer.group
      topic: sia.api-sim.service
      lagThreshold: '200'
      offsetResetPolicy: latest
  - type: cpu
    metricType: Utilization
    metadata:
      value: "80"
  - type: memory
    metricType: Utilization
    metadata:
      value: "85"

Key Points: - Kafka Trigger: Scales pods based on message lag in the sia.api-sim.service topic. If lag exceeds 200, more pods are added. - CPU & Memory Triggers: Scales up if average CPU usage exceeds 80% or memory usage exceeds 85%. - Polling Interval: KEDA checks metrics every 15 seconds. - minReplicaCount / maxReplicaCount: Ensures the service runs between 6 and 15 pods.

🧩 Example: KEDA ScaledObject for sms-service

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sms-service-scaler
  namespace: sms-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sms-il-prod-sms-service
  minReplicaCount: 3
  maxReplicaCount: 6
  pollingInterval: 15
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
      consumerGroup: sim.service-sms.service.consumer.group
      topic: sim.service-sms.service
      lagThreshold: '200'
      offsetResetPolicy: latest
  - type: cpu
    metricType: Utilization
    metadata:
      value: "80"
  - type: memory
    metricType: Utilization
    metadata:
      value: "85"

🧩 Example: KEDA ScaledObject for timer-service

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: timer-service-scaler
  namespace: timer-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: timer-service
  minReplicaCount: 3
  maxReplicaCount: 10
  pollingInterval: 15
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
      consumerGroup: timer.service.requests
      topic: timer.service.requests
      lagThreshold: '200'
      offsetResetPolicy: latest
  - type: cpu
    metricType: Utilization
    metadata:
      value: "80"
  - type: memory
    metricType: Utilization
    metadata:
      value: "85"

🧩 Example: KEDA ScaledObject for sia-service

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sia-service-scaler
  namespace: sia-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sia-il-prod-sia-service
  minReplicaCount: 3
  maxReplicaCount: 10
  pollingInterval: 15
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
      consumerGroup: timer.service.requests
      topic: timer.service.requests
      lagThreshold: '200'
      offsetResetPolicy: latest
  - type: kafka
    metadata:
      bootstrapServers: cluster-kafka-bootstrap.kafka.svc.cluster.local:9092
      consumerGroup: sia.api_sim
      topic: sim.service-sia.api
      lagThreshold: '200'
      offsetResetPolicy: latest
  - type: cpu
    metricType: Utilization
    metadata:
      value: "80"
  - type: memory
    metricType: Utilization
    metadata:
      value: "85"

🛡️ Benefits of KEDA Scale in Unibeam

  • Event-Driven: Instantly reacts to spikes in Kafka lag or resource usage.
  • Efficient: Prevents over-provisioning by scaling down when demand drops.
  • Flexible: Supports multiple triggers for robust scaling logic.
  • Seamless Integration: Works with existing Kubernetes deployments and namespaces.

Best Practice

  • Align minReplicaCount and maxReplicaCount with your application's requirements.
  • Use multiple triggers for more resilient and responsive scaling.
  • Monitor scaling events to fine-tune thresholds and resource requests.

📖 References