Kubernetes Logging with Grafana Alloy and Loki: A Complete Observability Guide

In the world of Kubernetes, effective log management isn’t a luxury — it’s a necessity. Application logs are vital for debugging issues, measuring performance, ensuring security, and gaining visibility into how your systems behave. Without a structured logging solution, keeping track of what’s happening inside your cluster becomes frustrating, error-prone, and time-consuming. This guide walks through how Grafana Alloy, paired with Loki, can help you collect, enrich, and centralize logs from across your Kubernetes workloads..

What is Grafana Alloy?

Grafana Alloy is an open-source telemetry collection agent, based on the OpenTelemetry (OTel) Collector. It’s designed to be flexible, lightweight, and powerful — capable of handling logs, metrics, and traces from multiple sources.
(See: Grafana Alloy Docs)

It allows you to:

Collect telemetry data from pods, nodes, and services
Apply processing pipelines (e.g., parsing, filtering, labeling)
Forward enriched data to backends like Loki, Prometheus, or Grafana Cloud

Its modular configuration structure makes it ideal for use cases that require high customization.

Project Goal: Full-Cluster Pod Log Collection with Contextual Enrichment

In this setup, we aim to:

Deploy Grafana Alloy as a DaemonSet so that it runs on every node in the cluster
Automatically discover logs from all running application pods, using Kubernetes labels and annotations
Apply custom processing rules for NGINX-style front-end logs — extracting fields like HTTP methods and response codes as structured labels
Send all processed logs to a centralized Loki instance, where they can be queried and visualized via Grafana

This creates a scalable and structured logging pipeline that fits naturally into cloud-native operations.

Implementing Grafana Alloy Using Helm

We leverage the official Grafana Alloy Helm chart. Instead of embedding Alloy configuration directly in Helm values, we use a pre-established Kubernetes ConfigMap for better manageability.

Key custom Helm values (values.yaml):

alloy:
  configMap:
    create: false          # We supply and manage the ConfigMap independently
    name: alloy-logs-config # Identifier of our existing ConfigMap
    key: config.alloy      # Key within the ConfigMap holding Alloy config
  clustering:
    enabled: false          # Clustering is disabled for simplicity
  mounts:
    varlog: true            # Grants Alloy access to pod logs on the node
  controller:
    type: 'daemonset'       # Alloy runs on every node to collect local pod logs

Significant aspects of this configuration:

configMap.create: false coupled with configMap.name: alloy-logs-config, These directives instruct Helm to utilize an existing ConfigMap identified as alloy-logs-config.
mounts.varlog: true: This setting ensures that the host machine's /var/log directory is mounted into the Alloy pod's filesystem. This directory is the conventional location for container log files (e.g., /var/log/pods/...).
controller.type: 'daemonset': This controller type guarantees that an instance of Alloy is scheduled on each node within the cluster, enabling efficient log collection from pods resident on those respective nodes.

Constructing the Alloy Configuration: The Core Logic

The efficacy of our logging setup is fundamentally determined by the Alloy configuration itself. This configuration will reside within a ConfigMap named alloy-logs-config, situated in the monitoring namespace, consistent with your provided example.

apiVersion: v1
kind: ConfigMap
metadata:
  name: alloy-logs-config
  namespace: monitoring
data:
  config.alloy: |
    discovery.kubernetes "pods" {
      role = "pod"
    }

    discovery.relabel "pod_logs" {
      targets = discovery.kubernetes.pods.targets
      rule {
        source_labels = ["__meta_kubernetes_namespace"]
        target_label = "namespace"
      }
      rule {
        source_labels = ["__meta_kubernetes_pod_name"]
        target_label = "pod"
      }
      rule {
        source_labels = ["__meta_kubernetes_pod_container_name"]
        target_label = "container"
      }
      rule {
        source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_name"]
        separator = "/"
        target_label = "job"
      }
      rule {
        source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
        separator = "/"
        action = "replace"
        replacement = "/var/log/pods/*$1/*.log" # Tailor this pattern to accurately reflect K8s log path structure
        target_label = "__path__"
      }
      rule {
        action = "replace"
        source_labels = ["__meta_kubernetes_pod_container_id"]
        regex = "^(\\w+):\\/\\/.+$"
        replacement = "$1"
        target_label = "tmp_container_runtime"
      }
    }

    local.file_match "pod_logs" {
      path_targets = discovery.relabel.pod_logs.output
    }

    loki.source.file "pod_logs" {
      targets = local.file_match.pod_logs.targets
      forward_to = [loki.process.pod_logs.receiver]
    }

    loki.process "pod_logs" {
      // Stage 1: Tailored processing for frontend container logs (NGINX-style)
      stage.match {
        selector = "{container=~\"(survey|jobs|contact)-frontend\"}" // Applies exclusively to these specified containers
        stage.regex {
          expression = "(?P<kong_gateway>192\\.168\\.\\d+\\.\\d+)" // Example IP; adjust regex as per your network configuration
        }
        stage.regex {
          expression = "(?P<method>GET|PUT|DELETE|POST)"
        }
        stage.regex {
          expression = "(?P<status_code_with_http_version>HTTP.{6}\\d{3})"
        }
        stage.regex {
          expression = "(?P<status_code>\\d{3})"
          source = "status_code_with_http_version" // Extracts from a previously captured group
        }
        stage.labels {
          values = {
            kong_gateway = "", // Elevates the extracted 'kong_gateway' to a distinct label
            method = "", // Elevates the extracted 'method' to a distinct label
            status_code = "", // Elevates the extracted 'status_code' to a distinct label
          }
        }
      }

      // Stage 2: Parsing for Containerd log formats
      stage.match {
        selector = "{tmp_container_runtime=\"containerd\"}"
        stage.cri {} // The CRI stage is designed to parse logs originating from containerd or CRI-O runtimes
        stage.labels {
          values = {
            flags = "", // Incorporates 'flags' derived from CRI parsing as a label
            stream = "", // Incorporates 'stream' (stdout/stderr) from CRI parsing as a label
          }
        }
      }

      // Stage 3: Parsing for Docker log formats
      stage.match {
        selector = "{tmp_container_runtime=\"docker\"}"
        stage.docker {} // The Docker stage is adept at parsing logs formatted in Docker's JSON structure
        stage.labels {
          values = {
            stream = "", // Incorporates 'stream' (stdout/stderr) from Docker parsing as a label
          }
        }
      }

      // Stage 4: Disposal of temporary labels
      stage.label_drop {
        values = ["tmp_container_runtime"]
      }

      forward_to = [loki.write.loki.receiver]
    }

    loki.write "loki" {
      endpoint {
        url = "http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push"
      }
    }

Let us dissect this Alloy configuration:

discovery.kubernetes "pods": This component is responsible for discovering all pod resources operating within your Kubernetes cluster.
discovery.relabel "pod_logs": This section is pivotal for defining log targets.
- It receives the discovered pods as its input.
- It systematically adds valuable labels such as namespace, pod, container, and job, deriving them from Kubernetes metadata.
- Critically, the rule governing __path__ dynamically constructs the file path to the actual log files resident on the node (e.g., /var/log/pods/<namespace>_<pod_name>_<pod_uid>/<container_name>/*.log). The *$1/* segment in your original setup might have specific connotations; the example uses a more generalized pattern. The fundamental aspect is that $1 captures elements like __meta_kubernetes_pod_uid and __meta_kubernetes_pod_container_name. It's imperative that your replacement pattern accurately mirrors your Kubernetes log file organization.
- It also ingeniously extracts the container runtime environment (containerd or docker) and stores it in a provisional label named tmp_container_runtime for subsequent conditional processing.
local.file_match "pod_logs": This component ingests the targets furnished by discovery.relabel (which are now augmented with the __path__ information) and filters them to ensure Alloy exclusively attempts to read from extant files.
loki.source.file "pod_logs": This component actively tails the log files pinpointed by local.file_match and relays the collected log lines to the loki.process component for further refinement.
loki.process "pod_logs": This constitutes our sophisticated multi-stage processing pipeline.
- Stage 1 (NGINX-style log parsing):
  - selector = "{container=~"(survey|jobs|contact)-frontend"}": This processing stage is selectively applied only to logs generated by containers whose names conform to the provided regular expression (e.g., survey-frontend, jobs-frontend).
  - stage.regex: A sequence of regular expressions is employed to parse individual log lines, extracting named capture groups such as kong_gateway, method, and status_code.
  - stage.labels: The data extracted from these capture groups is then elevated to become distinct Loki labels. This capability is exceptionally potent for formulating queries in Loki (e.g., sum by (status_code) (rate({job="mynamespace/mypod"}[5m]))).
- Stage 2 & 3 (Runtime-specific log parsing):
  - Leveraging the tmp_container_runtime label established earlier, this logic conditionally applies either stage.cri (for containerd environments) or stage.docker (for Docker JSON-formatted logs). These stages are designed to parse the standard log output of these respective runtimes, frequently extracting elements like timestamps and stream identifiers (stdout/stderr).
  - stage.labels is again utilized to promote useful fields, such as stream, into Loki labels.
- Stage 4 (stage.label_drop): The ephemeral tmp_container_runtime label is discarded at this point, as its utility has been fulfilled.
Ultimately, the fully processed logs are directed to the loki.write.loki.receiver.
loki.write "loki": This final component is tasked with forwarding the refined log entries to your designated Loki instance, the address of which is specified by the url parameter.

With this complete Alloy pipeline, your refined log entries are reliably forwarded to Loki, establishing a crucial layer of observability for your Kubernetes cluster. Such detailed operational insight is invaluable for maintaining system health and diagnosing issues, and it ideally works in concert with robust data protection measures. For stateful applications, ensuring you also have reliable Kubernetes backup and restore mechanisms, such as those provided by Longhorn, is essential for complete resilience.

Key Advantages of This Configuration

Selective Log Processing: Intensive regex-based parsing is judiciously applied only to logs from specific front-end containers, thereby conserving system resources.
Rich, Interrogable Labels: The extraction of information like HTTP status codes and methods as labels dramatically augments your capacity to query and scrutinize logs within Loki.
Uniform Kubernetes Metadata: Essential Kubernetes metadata attributes (pod, namespace, container, job) are consistently applied across all collected logs, ensuring uniformity.
Runtime-Cognizant Parsing: Alloy adeptly manages and parses the native log formats characteristic of different container runtimes.
Centralized Configuration Stewardship: Employing a ConfigMap for Alloy's configuration, managed independently of the Helm release cycle, facilitates more straightforward updates and robust version control for your logging pipeline logic.

Concluding Thoughts

Grafana Alloy presents a resilient and adaptable framework for constructing sophisticated log collection and processing pipelines tailored for Kubernetes environments. Through the astute utilization of its discovery components and the formidable loki.process stage, you can transmute raw application logs into richly annotated, readily queryable data within Loki. The example delineated herein illustrates how to implement targeted parsing for specific applications while concurrently managing generic log formats for others, granting you granular command over your observability data.

It is advisable to meticulously adjust the regular expressions, selectors, and the Loki endpoint to harmonize with the unique characteristics of your environment and application log structures. May your logging endeavors be fruitful!

Advance Your Kubernetes Observability

Setting up a robust logging pipeline with Grafana Alloy and Loki, as detailed in this guide, is a significant step towards achieving comprehensive observability in your Kubernetes environment. However, every cluster has its unique complexities, and tailoring a solution to perfectly fit your specific workloads, scale, and operational requirements can be a challenging endeavor.

If you're looking to:

Implement this Grafana Alloy and Loki logging solution efficiently and effectively.
Develop a more advanced, customized logging and observability strategy that goes beyond this guide, incorporating metrics, traces, and sophisticated alerting.
Analyze and optimize your existing Kubernetes logging setup for better performance, cost-efficiency, and deeper insights.

Our team of experienced DevOps and Observability engineers can help. We specialize in designing and implementing tailored observability solutions for Kubernetes, ensuring you have the visibility needed to operate your systems confidently and resolve issues swiftly.

Reach out to us to discuss your Kubernetes logging and observability needs, and let's build a solution that empowers your team.