Kubernetes Monitoring with Heapster, InfluxDB and Grafana

containers, couchbase, ,

Kubernetes provides detailed insights about resource usage in the cluster. This is enabled by using Heapster, cAdvisor, InfluxDB and Grafana.

Heapster is installed as a cluster-wide pod. It gathers monitoring and events data for all pods on each node by talking to the Kubelet. Kubelet itself fetches this data from cAdvisor. This data is persisted in InfluxDB and then visualized using Grafana.

kubernetes-logging

Resource Usage Monitoring provide more details about monitoring resources in Kubernetes.

Heapster, InfluxDB and Grafana are Kubernetes addons. They are enabled by default if you are running the cluster on Amazon Web Services or Google Cloud. But need to be explicitly enabled if the cluster is started using minikube or kops addons.

Start a Kubernetes cluster on Amazon Web Services as:

KUBERNETES_PROVIDER=aws; kube-up.sh

More details about starting a Kubernetes cluster are available at Getting Started with Kubernetes 1.4.

By default, it creates a 4-node Kubernetes cluster in us-west-2a region. More details about the cluster can be seen using the command kubectl cluster-info and it shows the results as:

Note the URL for the Grafana service. Open this URL in a browser window. You’ll be prompted for an invalid certificate warning but this can be safely ignored at this time. In production system, appropriate certificates should be installed.

Then you’ll be prompted for credentials. These can be obtained using kubectl config view command. It will show the output as:

Use the value from username and password fields.

This shows the default dashboard:

kubernetes-grafana-empty-dashboard

It consists of two dashboards – one for cluster and another for pods.

kubernetes-grafana-dashboards

For this blog, a 4-node Couchbase cluster was created following the steps outlined in Create a Couchbase Cluster using Kubernetes.

A cluster-wide dashboard shows CPU, Memory, Filesystem and Network usage across all the hosts and looks like:

kubernetes-grafana-cluster

CPU, memory, filesystem and network usage for all nodes may be seen:

kubernetes-grafana-cluster-per-node

Details for each node may be seen by selecting the node:

kubernetes-grafana-cluster-nodelist

CPU, memory, filesystem and network usage for each node is displayed:

kubernetes-grafana-cluster-one-node

Pods dashboard shows CPU, memory, filesystem and network usage for each pod:

kubernetes-grafana-pods

A different pod may be chosen:

kubernetes-grafana-pods-list

A complete list of all services running in the Kubernetes can be seen using kubectl get services --all-namespaces command. It shows the output as:

A complete list of all the pods running in the Kubernetes cluster can be seen using kubectl get pods --all-namespaces. It shows the output as:

kubectl.sh get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default couchbase-master-rc-q9awd 1/1 Running 17 56m
default couchbase-worker-rc-b1qkc 1/1 Running 15 54m
default couchbase-worker-rc-j1c5z 1/1 Running 17 52m
default couchbase-worker-rc-ju7z3 1/1 Running 15 52m
kube-system elasticsearch-logging-v1-18ylh 1/1 Running 0 1h
kube-system elasticsearch-logging-v1-fupap 1/1 Running 0 1h
kube-system fluentd-elasticsearch-ip-172-20-0-94.us-west-2.compute.internal 1/1 Running 0 1h
kube-system fluentd-elasticsearch-ip-172-20-0-95.us-west-2.compute.internal 1/1 Running 0 1h
kube-system fluentd-elasticsearch-ip-172-20-0-96.us-west-2.compute.internal 1/1 Running 15 1h
kube-system fluentd-elasticsearch-ip-172-20-0-97.us-west-2.compute.internal 1/1 Running 17 1h
kube-system heapster-v1.2.0-1374379659-jms8e 4/4 Running 0 1h
kube-system kibana-logging-v1-fcg4b 1/1 Running 3 1h
kube-system kube-dns-v20-wpip4 3/3 Running 0 1h
kube-system kube-proxy-ip-172-20-0-94.us-west-2.compute.internal 1/1 Running 0 1h
kube-system kube-proxy-ip-172-20-0-95.us-west-2.compute.internal 1/1 Running 0 1h
kube-system kube-proxy-ip-172-20-0-96.us-west-2.compute.internal 1/1 Running 15 1h
kube-system kube-proxy-ip-172-20-0-97.us-west-2.compute.internal 1/1 Running 17 1h
kube-system kubernetes-dashboard-v1.4.0-yxxgx 1/1 Running 0 1h
kube-system monitoring-influxdb-grafana-v4-7asy4 2/2 Running 0 1h

Some references:

  • Kubernetes Resource Monitoring
  • Couchbase Cluster using Kubernetes, Docker Swarm, DC/OS and Amazon ECS
  • Follow us @couchbasedev

Source: blog.couchbase.com/2016/december/kubernetes-monitoring-heapster-influxdb-grafana

Be Sociable, Share!
  • Tweet
Related posts:
  1. Kubernetes Cluster on Amazon and Expose Couchbase Service
  2. Couchbase on Kubernetes
  3. Kubernetes Namespaces, Resource Quota, and Limits for QoS in Cluster
  4. Stateful Containers on Kubernetes using Persistent Volume and Amazon EBS
  5. Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase

3 thoughts on “Kubernetes Monitoring with Heapster, InfluxDB and Grafana”

  1. HI Arun,
    I have created a 3 node (ubuntu16.04 VM on Azure) kubernetes cluster using rancher and trying to access the grafana dashboard using the url exposed by clusterinfo command. However i am getting Unauthorized error. Below is my url.
    monitoring-grafana is running at https://mypublicip:8080/r/projects/1a7/kubernetes:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
    Thanks.

  2. Thank you so much to this extra ordinary Mind twister video game.I was looking for this and you guys did a great work here.

  3. You have posted a incredibly detail document. I examine your entire article and I really like it, I actually understand your point of view.
    https://street-view.org/

Leave a Reply

Your email address will not be published. Required fields are marked *