Monthly Archives: November 2016

Couchbase Cluster on Mesos with DC/OS

apache-mesos-logoapache-mesos-marathon-logoDocker LogoCouchbase Logo

Apache Mesos is an open source cluster manager developed at UC Berkeley. It provides resource isolation and sharing across distributed applications. Mesos consists of a master daemon that manages slave daemons running on each cluster node.Mesos frameworks are applications that runs on Mesos and run tasks on these slaves. Marathon is a container orchestration platform running on Mesos.Multiple container formats are supported and Docker is certainly the most common one!

Docker Container using Apache Mesos and Marathon explains how to setup Mesos and Marathon. The setup is quite involving and a bit flaky. It required to download and Install Mesos Master and Slave, ZooKeeper, Docker Engine, and Marathon.

DC/OS is a distributed operating system using Mesos as its kernel. Couchbase on Mesos using DC/OS and Amazon explained how to run a single Couchbase container on DC/OS using CloudFormation templates.

Running a single node Couchbase may work during initial development phase. The need to start a multi-node Couchbase cluster becomes eminent as you move along further in development, and certainly needed during the production phase.

So, you’d like to run a Couchbase cluster on DC/OS?

Couchbase Cluster on DC/OS is complete walk through of how to setup a Couchbase cluster on DC/OS. It walks through the following steps:

  • What is Couchbase?
  • Couchbase Cluster
  • Setup DC/OS on Amazon Web Services
  • Configure CLI and Install Marathon Load Balancer
  • Create Couchbase “startup” service
  • Create Couchbase “node” service
  • Scale Couchbase Cluster
  • Rebalance Couchbase Cluster
  • Conclusion

DC/OS dashboard with a Couchbase cluster looks like:

For further information check out:

  • Couchbase on Containers
  • Couchbase Developer Portal
  • Ask questions on Couchbase Forums or Stack Overflow
  • Download Couchbase

You can also follow us at @couchbasedev and @couchbase.


Health Check of Docker Containers

One of the new features in Docker 1.12 is how health check for a container can be baked into the image definition. And this can be overridden at the command line.

Just like the CMD instruction, there can be multiple HEALTHCHECK instructions in Dockerfile but only the last one is effective.

This is a great addition because a container reporting status as Up 1 hour may return errors. The container may be up but there is no way for the application inside the container to provide a status. This instruction fixes that.

The Dockerfile that builds arungupta/couchbase image is:

It uses script to configure the server using Couchbase REST API. The new instruction to notice here is HEALTHCHECK.

This instruction can be specified as:

The <options> can be:

  • --interval=DURATION (default 30s)
  • --timeout=DURATION (default 30s)
  • --retries=N (default 3)

The <command> is the command that runs inside the container to check the health.

If health check is enabled, then the container can have three states:

  • starting – Initial status when the container is still starting
  • healthy – If the command succeeds then the container is healthy
  • unhealthy – If a single run of the <command> takes longer than the specified timeout then it is considered unhealthy. If a health check fails then the <command> will run retries number of times and will be declared unhealthy if the <command> still fails.

The commands exit status indicates the health status of the container. The following values are allowed:

  • 0 – container is healthy
  • 1 – container is not healthy

In our instruction, /pools REST API is invoked using curl. If the command fails then an exit status of 1 is returned, and this marks the container unhealthy for that attempt. This command is invoked every 5 seconds. The container is marked unhealthy if the command does not return successfully within 3 seconds.

Run the container as:

Check the status:

Notice how health: starting status is reported in the STATUS column. Checking after a few seconds shows the status:

And now its reported healthy.

More details about this HEALTHCHECK instruction can be found on

Now, if you are running an image that does not have HEALTHCHECK instruction then the docker run command can be used to specify similar values. An equivalent runtime command would be:

Last 5 health checks for a container can be obtained using the docker inspect command:

The output is shown as:




Docker for AWS – Getting Started Video

Want to create a highly-available Docker cluster on Amazon Web Services? Run multi-container applications on it using Docker Services?

Docker Logo
Couchbase Logo

Docker for AWS allows you to exactly do that! This video shows:

  • Create a highly-available Docker cluster on Amazon Web Services (0:00)
  • Check configuration (5:43)
  • Use Docker services to create a Couchbase cluster (8:23)

Enjoy! provide more details about how to run Couchbase in different container frameworks. More information about Couchbase:

  • Couchbase Developer Portal
  • Couchbase Forums
  • @couchbasedev or @couchbase


Multimaster Kubernetes Cluster on Amazon Using Kops

Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase explains how to get started with Kubernetes  1.4 on Amazon Web Services. A Couchbase service is created in the cluster and a Spring Boot application stores a JSON document in the database. It uses script from the Kubernetes binary download at  to start the cluster. This script is capable of creating a Kubernetes cluster with single master only. This is a fundamental flaw of distributed applications where the master becomes a Single Point of Failure.

Meet kops – short for Kubernetes Operations.

This is the easiest way to get a highly-available Kubernetes cluster up and running. The kubectl script is the CLI for running commands against running clusters. Think of kops as kubectl for cluster.

This blog will show how to create a highly-available Kubernetes cluster on Amazon using kops. And once the cluster is created, then it’ll create a Couchbase service on it and run a Spring Boot application to store JSON document in the database.

Many thanks to justinsb, sarahz, razic, jaygorrell, shrugs, bkpandey and others at Kubernetes slack channel for helping me through the details!

Download kops and kubectl

  • Download Kops latest release. This blog was tested with 1.4.1 on OSX.Complete set of commands for kops can be seen:
  • Download kubectl:
  •  Include kubectl in your PATH.

Create Bucket and NS Records on Amazon

There is a bit of setup involved at this time, and hopefully this will get cleaned up over next releases. Bringing up a cluster on AWS provide detailed steps and more background. Here is what the blog followed:

  • Pick a domain where Kubernetes cluster will be hosted. This blog uses domain. You can pick a top level domain or a sub-domain.
  • Amazon Route 53 is a highly available and scalable DNS service. Login to Amazon Console and created a hosted zone for this domain using Route 53 service.kops-hosted-zoneCreated zone looks like:kops-hosted-zone-createdThe values shown in the Value column are important as they’ll be used later for creating NS records.
  • Create a S3 bucket using Amazon Console to store cluster configuration – this is called state store.kops-s3-bucket
  • The domain is hosted on GoDaddy. For each value shown in the Value column of Route53 hosted zone, create a NS record using GoDaddy Domain Control Center for this domain.Select the type of record:
    kops-godaddy-add-zone-recordFor each value, add the record as shown:kops-godaddy-add-ns-recordCompleted set of records look like:kops-godaddy-ns-records

Start Kubernetes Multimaster Cluster

Let’s understand a bit about Amazon regions and zones:

Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of regions and Availability Zones. Each region is a separate geographic area. Each region has multiple, isolated locations known as Availability Zones.

Amazon Docs

A highly-available Kubernetes cluster can be created across zones but not across regions.

  • Find out availability zones within a region:
  • Create a multi-master cluster:
    Most of the switches are self-explanatory. Some switches need a bit of explanation:

    • Specifying multiple zones using --master-zones (must be odd number) create multiple masters across AZ
    • --cloud=aws is optional if cloud can be inferred from zones
    • --yes is used to specify the immediate creation of cluster. Otherwise only the state is stored in the bucket, and the cluster needs to be created separately.

    Complete set of CLI switches can be seen:

  • Once the cluster is created, get more details about the cluster:

  • Check cluster client and server version:

  • Check all nodes in the cluster:

    Or find out only the master nodes:

  • Check all the clusters:

Kubernetes Dashboard Addon

By default, a cluster created using kops does not have the UI dashboard. But this can be added as an add on:

Now complete details about the cluster can be seen:

And the Kubernetes UI dashboard is at the shown URL. In our case, this is and looks like:


Credentials for accessing this dashboard can be obtained using the kubectl config view command. The values are shown like:

Deploy Couchbase Service

As explained in Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase, let’s run a Couchbase service:

This configuration file is at

Get the list of services:

Describe the service:

Get the pods:

Run Spring Boot Application

The Spring Boot application runs against the Couchbase service and stores a JSON document in it.

Start the Spring Boot application:

This configuration file is at

See list of all the pods:

Check logs of the complete pod:

The updated dashboard now looks like:


Delete the Kubernetes Cluster

Kubernetes cluster can be deleted as: provide more details about how to run Couchbase in different container frameworks.

More information about Couchbase:

  • Couchbase Developer Portal
  • Couchbase Forums
  • @couchbasedev or @couchbase