Skip to content

📊 Monitoring/Logs Flow

How Kube-Prometheus-Stack Monitors Applications

Kube-Prometheus-Stack automates metrics collection, alerting, and visualization for all application workloads in your Kubernetes cluster.
It discovers application endpoints using ServiceMonitors, scrapes metrics with Prometheus, sends alerts via Alertmanager, and visualizes data in Grafana.

Monitoring Flow Overview

  • ServiceMonitors: Discover application /metrics endpoints.
  • Prometheus: Scrapes and stores metrics.
  • Alertmanager: Handles alerts and notifications.
  • Grafana: Visualizes metrics and logs.
  • Promtail & Loki: Aggregate and store logs for troubleshooting.
%% Monitoring flow for Kube-Prometheus-Stack
graph TD
    A[Application Pods<br>/metrics endpoint] --> B(ServiceMonitors)
    B --> C[Prometheus]
    C --> D[Alertmanager]
    C --> E[Grafana]
    A --> F[Promtail]
    F --> G[Loki]
    G --> E
    E --> H[User/Dashboard]

    style A fill:#f9f,stroke:#333,stroke-width:1px
    style C fill:#bbf,stroke:#333,stroke-width:1px
    style G fill:#bfb,stroke:#333,stroke-width:1px
    style E fill:#ffd,stroke:#333,stroke-width:1px

Diagram Legend

  • Purple: Application pods
  • Blue: Metrics pipeline
  • Green: Log pipeline
  • Yellow: Visualization

📝 How Promtail Sends Logs to Loki

Promtail runs as a DaemonSet in the promtail namespace, collecting logs from all pods across the cluster.
It applies filtering and relabeling, then pushes logs to the Loki service in the loki namespace.
Grafana queries Loki for log visualization and troubleshooting.

Log Flow Overview

  • Promtail: Collects and processes logs from pods.
  • Loki: Stores and indexes logs for fast querying.
  • Grafana: Visualizes logs from Loki.
%% Promtail to Loki log flow
graph LR
    A[Application Pods<br>stdout/stderr logs] --> B[Promtail<br>DaemonSet]
    B --> C[Log Filtering & Relabeling]
    C --> D[Loki<br>Log Aggregation]
    D --> E[Grafana<br>Log Query & Visualization]
    E --> F[User/Dashboard]

    style A fill:#f9f,stroke:#333,stroke-width:1px
    style B fill:#bff,stroke:#333,stroke-width:1px
    style D fill:#bfb,stroke:#333,stroke-width:1px
    style E fill:#ffd,stroke:#333,stroke-width:1px

Diagram Legend

  • Purple: Application pods
  • Blue: Log collection
  • Green: Log aggregation
  • Yellow: Visualization

How Logs are Collected and Processed

Logs are collected from application pods by Promtail, which is deployed as a DaemonSet on each node.
Promtail tails the logs from the specified log files, applies any configured pipelines for parsing and filtering, and then sends the logs to Loki.
Loki indexes the logs and makes them available for querying. Grafana can be used to query Loki and visualize the logs in real-time.

For detailed information about log collection and processing, refer to the Loki documentation.