Kubernetes Logging with Grafana's Loki

Publish date: Jan 2, 2021
Tags: ops

Happy 2021! And to get the year started, let’s look at some modern logging stacks!

I love modern application and stack observability. Anyone that remembers the old days of stringing together Nagios and Syslog into a cobbled-together stack can certainly sympathize. Now solutions exist that provide a more integrated experience across metrics, logs, and tracing. Today I’ll be concentrating on the logs component with Grafana’s Loki, and demonstrate that stack on a Civo k3s cluster.

Kubernetes itself provides some basic log utilities. A simple “kubectl logs” can be used to get logs for a specific pod, or the Kubernetes Dashboard can provide the same information in a web interface. The built-in solution does suffer from some moderate deficiencies:

A broader logging solution can help overcome these deficiencies. Even in development clusters, there tends to be a need to review logs beyond the lifespan of current pods. I present one such solution, Grafana’s Loki.

Loki - Not Quite the Norse God

Loki is a lightweight logging solution, developed by Grafana. It has a similar design methodology to the popular monitoring system Prometheus, and even has the tagline “Prometheus for logs”. It certainly lives up to that name by providing a lightweight log management solution.

Let’s test out Loki leveraging the Civo Kube100 Platform. This guide assumes you are familiar with the Civo Platform and have the kubectl command available on your local device. If you want to test on your own Kubernetes platform, see the Source Code section at the end of this post.

  1. First, spin up a new cluster, and make sure to select the “Loki Stack” from the marketplace: Civo Cluster Creation

  2. Once the cluster is up, download the kubeconfig file, and make sure a kubectl command works:

    $ kubectl get nodes
    kube-master-6b7c Ready master 4m v1.18.6+k3s1
    kube-node-d7e6 Ready <none> 2m12s v1.18.6+k3s1
    kube-node-c0b1 Ready <none> 2m7s v1.18.6+k3s1
  3. Grafana and Loki are both running as part of the Marketplace installations. However, Grafana needs to be configured to attach to our local Loki instance in order to view logs.

Now Loki is set up! Test it out by going to the Explore (Compass Looking) icon on the left-hand side, and choosing Loki as a data source.

Loki Explore

The examples listed should work, but feel free to explore further. Say, if you wanted to get logs from all pods in the kube-system namespace:

Loki Kube System

Check out the “Log labels” drop down to see all the detected fields you can use to narrow down log results

Available Fields

Production Considerations

Loki makes a great log collection system for small to medium sized clusters. If you want to keep logs across multiple clusters, there are a few options, current as of January 2021:

Source Code

If you wish to apply this configuration to your own Kubernetes cluster, or are just interested in how it works, check out the manifests for Loki as well as Prometheus. The Prometheus operator, as configured, will require Rancher’s Helm-Controller.

comments powered by Disqus
Steve Miller BY-NC 4.0 | Rendered by Hugo | Subscribe