Kubernetes Logging with Grafana Alloy and Loki: A Complete Observability Guide

In Kubernetes, effective log management is a necessity. Application logs are vital for debugging issues, measuring performance, ensuring security, and understanding system behavior.

This guide walks through how Grafana Alloy, paired with Loki, can collect, enrich, and centralize logs across Kubernetes workloads.

What is Grafana Alloy?

Grafana Alloy is an open-source telemetry collection agent based on the OpenTelemetry Collector. It can handle logs, metrics, and traces from multiple sources.

It allows you to:

• Collect telemetry data from pods, nodes, and services
• Apply processing pipelines for parsing, filtering, and labeling
• Forward enriched data to backends such as Loki, Prometheus, or Grafana Cloud

Project Goal

In this setup, we aim to:

• Deploy Grafana Alloy as a DaemonSet on every node
• Discover logs from all running application pods
• Apply custom processing rules for NGINX-style frontend logs
• Send processed logs to a centralized Loki instance

This creates a scalable and structured logging pipeline for cloud-native operations.

Implementing Grafana Alloy Using Helm

We use the official Grafana Alloy Helm chart. Instead of embedding Alloy configuration directly in Helm values, we use a pre-established Kubernetes ConfigMap.

alloy:
  configMap:
    create: false
    name: alloy-logs-config
    key: config.alloy
  clustering:
    enabled: false
  mounts:
    varlog: true
  controller:
    type: daemonset

Important settings:

• configMap.create: false tells Helm to use an existing ConfigMap.
• mounts.varlog: true gives Alloy access to pod logs on the node.
• controller.type: daemonset ensures one Alloy instance runs per node.

The Core Alloy Configuration

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
}

The discovery configuration finds pod resources and enriches log targets with Kubernetes metadata.

Processing Logs

The processing pipeline can selectively parse frontend container logs, extract fields such as HTTP method and status code, and promote them to Loki labels.

Runtime-specific parsing can handle both containerd and Docker formats, while temporary labels are dropped before forwarding logs to Loki.

Key Advantages

• Selective processing: Regex parsing is applied only where needed.
• Rich labels: HTTP status codes, methods, namespace, pod, and container labels make querying easier.
• Consistent metadata: Kubernetes context is attached to every collected log.
• Runtime-aware parsing: Different container runtimes are handled cleanly.
• Centralized configuration: A ConfigMap keeps logging logic versionable and easier to update.

Concluding Thoughts

Grafana Alloy provides a resilient and adaptable foundation for Kubernetes logging. With discovery, relabeling, and Loki processing stages, raw application logs become queryable operational data.

Every cluster has unique workloads and requirements, so the regular expressions, selectors, and Loki endpoint should be adapted to the environment.

Advance Your Kubernetes Observability

If you want to implement Grafana Alloy and Loki efficiently, develop a more advanced observability strategy, or optimize your existing Kubernetes logging setup, Karakoo can help design and operate a solution that fits your workloads.