Get started with observability with Grafana, Loki, and Promtail

In today’s dynamic and ever-evolving IT landscape, the need for comprehensive observability has become more crucial than ever before.

Kubernetes, the container orchestration platform, has revolutionized application deployment and management, but its complexity also demands robust observability capabilities.

This article delves into the significance of Kubernetes observability and its role in securing the infrastructure.

I will demonstrate the synergistic combination of Grafana, Loki, and Promtail — three essential tools that empower you to gain deeper insights into logs generated by various Kubernetes components and applications deployed on it.

Visualize K8s logs on Grafana using Promtail and Loki

Defining Kubernetes Observability

Kubernetes observability encompasses the practice of collecting, aggregating, and analyzing data from various sources within the Kubernetes ecosystem, including metrics, logs, and traces.

This data provides valuable insights into the health, performance, and behavior of the cluster, enabling proactive identification and resolution of issues.

“This article focuses on the logging part only; metrics and traces will be covered in future publications.”

The Importance of Observability for Securing Kubernetes

Observability plays a critical role in securing Kubernetes environments by providing the visibility necessary to detect and mitigate security threats.

By monitoring logs for suspicious patterns, resource utilization, and network traffic, administrators can identify potential attacks, anomalous behavior, and unauthorized access attempts.

Photo by Scott Webb on Unsplash

Introducing the Trio of Observability Tools

Grafana, Loki, and Promtail form a powerful trio for comprehensive Kubernetes observability.

Grafana

Grafana is a visualization tool that transforms raw data into insightful dashboards and data visualizations, enabling users to analyze trends, identify patterns, and troubleshoot issues effectively.

Loki

Loki is a log aggregator that collects, stores, and analyzes logs from various sources, including Kubernetes components and applications.

It provides powerful search and filtering capabilities, allowing you to pinpoint specific log entries and identify potential issues.

Promtail

Promtail is a log collector that ship logs to Loki, ensuring that all relevant log data is centralized and easily accessible for analysis.

Promtail also supports structured logging, enabling rich metadata extraction for enhanced insights.

Synergy in Action

When combined, Grafana, Loki, and Promtail form an unstoppable force for getting better insights in Kubernetes environments.

Grafana provides the visualization layer, Loki handles log aggregation and analysis, and Promtail ensures seamless log collection.

To bootstrap our little demonstration lab, we will use a local Kubernetes cluster with Minikube and provision all our infrastructure with Terraform. If you’re new to either of the two, please refer to my previous articles to learn how to install and start in a local development setup.

This is all the terraform script to start the collection, storage and visualization of logs within a kubernetes cluster:

Impressive, huh !? … I know 😜 🚀.

# Initialize terraform providers
provider "kubernetes" {
  config_path = "~/.kube/config"
}
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}
# Create a namespace for observability
resource "kubernetes_namespace" "observability-namespace" {
  metadata {
    name = "observability"
  }
}

# Helm chart for Grafana
resource "helm_release" "grafana" {
  name             = "grafana"
  repository       = "https://grafana.github.io/helm-charts"
  chart            = "grafana"
  version          = "7.1.0"
  namespace        = "observability"

  values = [file("${path.module}/values/grafana.yaml")]
  depends_on = [ kubernetes_namespace.observability-namespace ]
}

# Helm chart for Loki
resource "helm_release" "loki" {
  name       = "loki"
  repository = "https://grafana.github.io/helm-charts"
  chart      = "loki"
  version    = "5.41.5"
  namespace  = "observability"

  values = [file("${path.module}/values/loki.yaml")]
  depends_on = [ kubernetes_namespace.observability-namespace ]
}

# Helm chart for promtail
resource "helm_release" "promtail" {
  name       = "promtail"
  repository = "https://grafana.github.io/helm-charts"
  chart      = "promtail"
  version    = "6.15.3"
  namespace  = "observability"

  values = [file("${path.module}/values/promtail.yaml")]
  depends_on = [ kubernetes_namespace.observability-namespace ]
}

In the code snipet above, I define the only two provides needed (helm and kubernetes), the three charts for the three tools we are using, along with values files that define some minor customizations.

As example, here is the only customization of our Promtail chart:

extraVolumes:
  - name: node-logs
    hostPath:
      path: /var/log

extraVolumeMounts:
  - name: node-logs
    mountPath: /var/log/host
    readOnly: true

# Add Loki as a client to Promtail
clients:
  - url: http://loki-gateway.observability.svc.cluster.local/loki/api/v1/push

This customization tell Promtail to ship logs from the local machine and we define where to ship those logs (Loki in this case).

After running the terraform script, we end-up with a working monitoring setup fetching logs and visualizing them on Grafana, ready to do some powerful visualizations thanks to the powerful LogQL language provided by Loki/Grafana.

The full code for this article is available in my Github repository here:

devops-labs/30 at main · aminrj/devops-labs

To follow along with me while I create this setup, a video tutorial is available here:

Conclusion

Kubernetes observability, particularly with Grafana, Loki, and Promtail, is not just a luxury; it’s a necessity for securing and maintaining the stability and performance of Kubernetes environments.

By leveraging these powerful tools, organizations can gain deep insights into their Kubernetes infrastructure, proactively identify, and resolve issues, and safeguard their applications from security threats.