🛠️ Kubernetes Tips & Tricks¶
📌 Namespace Overview¶
This section provides a comprehensive overview of the core Kubernetes components deployed in the Unibeam platform on AWS EKS, including networking, ingress, monitoring, and application namespaces.
It is optimized for environments using AWS Load Balancer Controller for ALB/NLB ingress.
🗂️ Core Components & Namespaces¶
| Namespace | Component/Stack | Description |
|---|---|---|
| monitoring | Kube-Prometheus-Stack | Cluster monitoring, metrics, and alerting |
| loki | Loki Stack | Centralized log aggregation and querying |
| kafka | Kafka (Strimzi) | Distributed event streaming platform |
| promtail | Promtail | Log shipping to Loki |
| sim-service | SIM Service | SIM TCP service provisioning |
| sms-service | SMS Service | SMS gateway and processing |
| sia-service | SIA Service | Secure Identity & Authentication API Provider |
| mno-service | MNO Service | Mobile Network Operator integrations |
| audit-service | Audit Service | Audit logging and compliance |
| dashboard-service | Dashboard Service | Web dashboards and analytics |
| timer-service | Timer Service | Scheduled and timed operations |
| schedule-service | Schedule Service | Job scheduling and orchestration |
🌐 Networking & Ingress¶
- AWS Load Balancer Controller:
Manages the provisioning of AWS Application Load Balancers (ALB) and Network Load Balancers (NLB) for Kubernetes Ingress resources. - Ingress Resources:
- ALB Ingress: Used for HTTP/HTTPS traffic routing to services (e.g., dashboard, APIs).
- NLB Ingress: Used for TCP/UDP traffic, ideal for Kafka brokers, SMS gateways, and other non-HTTP workloads.
- Service Discovery:
- All services are discoverable via internal DNS (
<service>.<namespace>.svc.cluster.local). - External endpoints are exposed via ALB/NLB DNS names.
Ingress Controller
Deploy the AWS Load Balancer Controller in your cluster to automate ALB/NLB provisioning and management for ingress resources.
📊 Monitoring & Logging¶
- Prometheus (monitoring):
Collects metrics from all cluster components and custom applications. - Grafana (monitoring):
Visualizes metrics and dashboards for cluster health and application performance. - Loki (loki):
Aggregates logs from all namespaces, searchable via Grafana. - Promtail (promtail):
Ships logs from pods to Loki.
Namespace Separation
Monitoring and logging components are isolated in their own namespaces for security and resource management.
🦾 Kafka & Messaging¶
- Kafka (kafka):
Deployed via Strimzi operator for scalable event streaming. - KRaft (Kafka Raft Metadata mode):
Manages Kafka cluster coordination and metadata internally, eliminating the need for Zookeeper.
🔐 Application Namespaces¶
- Unibeam Application Stack:
sim-service,sms-service,sia-service,mno-service,audit-service,dashboard-service,timer-service,schedule-service- Each namespace hosts a dedicated microservice or application component.
- Services communicate via internal DNS and are exposed externally via ALB/NLB as needed.
Namespace Best Practice
Use dedicated namespaces for each major service to improve isolation, scalability, and security.
🛡️ Security & IAM Integration¶
-
IAM Roles for Service Accounts (IRSA):
Pods assume IAM roles for secure access to AWS resources (e.g., S3, Secrets Manager). -
AWS Pod Identity Service:
Enables fine-grained IAM role assignment to pods without modifying service accounts.
Use Pod Identity Associations to map pods directly to IAM roles, improving security and simplifying access management.
Recommended for clusters where multiple workloads require distinct AWS permissions.
AWS Pod Identity Service
- Use Pod Identity Associations to assign IAM roles to pods based on labels and selectors.
- No need to annotate service accounts; identity is managed at the pod level.
- See [AWS Pod Identity Documentation](https://docs.aws.amazon.com/eks/latesuserguide/pod-identity.html) for setup and best practices.
- Network Policies:
Restrict traffic between namespaces and control access to sensitive services.
Security
Always use IRSA and network policies to enforce least privilege and secure communication between components.
📋 Common Exit Codes¶
This section lists the most common exit codes encountered in Kubernetes pods and containers, helping you diagnose application and infrastructure issues efficiently.
🧩 Common Exit Codes¶
| Exit Code | Meaning | Typical Cause |
|---|---|---|
| 0 | Success | Container ran and exited normally |
| 1 | General error | Application error, misconfiguration |
| 2 | Misuse of shell builtins | Incorrect shell command usage |
| 126 | Command invoked cannot execute | Permission denied, not executable |
| 127 | Command not found | Binary/script missing in container |
| 128 | Invalid exit argument | Fatal error in shell or application |
| 137 | Killed (SIGKILL, OOMKilled) | Out of memory, container killed by system |
| 139 | Segmentation fault (SIGSEGV) | Application crash due to invalid memory access |
| 143 | Terminated (SIGTERM) | Graceful shutdown, pod deletion, rolling update |
| 255 | Exit status out of range | Application-specific error |
🛠️ Special Kubernetes Exit Codes¶
| Exit Code | Kubernetes Context | Description |
|---|---|---|
| 0 | Completed | Pod/Container finished successfully |
| 1 | Error | Pod/Container failed, check logs |
| 137 | OOMKilled | Container killed due to memory limits |
| 143 | SIGTERM | Container terminated by Kubernetes |
Troubleshooting Exit Codes
- Use
kubectl describe pod <pod_name> -n <namespace>to see exit codes and reason for container termination. - For OOMKilled (137), consider increasing memory limits or optimizing your application.
- For permission errors (126), ensure your entrypoint script is executable.
Further Reading
See Kubernetes Pod Lifecycle Documentation for more details on pod and container
🔧 Essential kubectl Commands¶
This section provides a quick reference for the most commonly used kubectl commands, tailored for the Unibeam platform and its associated namespaces.
🛠️ Essential kubectl Commands¶
List All Namespaces¶
Get All Pods in a Namespace¶
Describe a Pod¶
View Pod Logs¶
Get All Services in a Namespace¶
Get All Deployments in a Namespace¶
Get All Events in a Namespace (sorted by creation time)¶
Delete a Pod¶
Exec Into a Running Pod¶
Apply a Manifest File¶
Port Forward a Service¶
🚀 Example: Unibeam Application Namespace¶
kubectl get pods -n unibeam
kubectl describe pod <pod_name> -n unibeam
kubectl logs <pod_name> -n unibeam
kubectl get services -n unibeam
kubectl get deployments -n unibeam
kubectl get events -n unibeam --sort-by='.metadata.creationTimestamp'
kubectl delete pod <pod_name> -n unibeam
kubectl exec -it <pod_name> -n unibeam -- /bin/bash
kubectl apply -f <manifest_file.yaml> -n unibeam
kubectl port-forward svc/<service_name> -n unibeam <local_port>:<service_port>
Namespace Usage
Always specify the correct namespace with -n
Troubleshooting
Use kubectl get events -n
🚦 Kubernetes Pod & Container Exit Codes¶
This section lists the most common exit codes encountered in Kubernetes pods and containers, helping you diagnose application and infrastructure issues efficiently.
🧩 Common Exit Codes¶
| Exit Code | Meaning | Typical Cause |
|---|---|---|
| 0 | Success | Container ran and exited normally |
| 1 | General error | Application error, misconfiguration |
| 2 | Misuse of shell builtins | Incorrect shell command usage |
| 126 | Command invoked cannot execute | Permission denied, not executable |
| 127 | Command not found | Binary/script missing in container |
| 128 | Invalid exit argument | Fatal error in shell or application |
| 137 | Killed (SIGKILL, OOMKilled) | Out of memory, container killed by system |
| 139 | Segmentation fault (SIGSEGV) | Application crash due to invalid memory access |
| 143 | Terminated (SIGTERM) | Graceful shutdown, pod deletion, rolling update |
| 255 | Exit status out of range | Application-specific error |
🛠️ Special Kubernetes Exit Codes¶
| Exit Code | Kubernetes Context | Description |
|---|---|---|
| 0 | Completed | Pod/Container finished successfully |
| 1 | Error | Pod/Container failed, check logs |
| 137 | OOMKilled | Container killed due to memory limits |
| 143 | SIGTERM | Container terminated by Kubernetes |
Troubleshooting Exit Codes
- Use
kubectl describe pod <pod_name> -n <namespace>to see exit codes and reason for container termination. - For OOMKilled (137), consider increasing memory limits or optimizing your application.
- For permission errors (126), ensure your entrypoint script is executable.
Further Reading
See Kubernetes Pod Lifecycle Documentation for more details on pod and container