Apache Mesos is an open source cluster manager developed at UC Berkeley. It provides resource isolation and sharing across distributed applications. Mesos consists of a master daemon that manages slave daemons running on each cluster node.Mesos frameworks are applications that runs on Mesos and run tasks on these slaves. Marathon is a container orchestration platform running on Mesos.Multiple container formats are supported and Docker is certainly the most common one!

Docker Container using Apache Mesos and Marathon explains how to setup Mesos and Marathon. The setup is quite involving and a bit flaky. It required to download and Install Mesos Master and Slave, ZooKeeper, Docker Engine, and Marathon.

DC/OS is a distributed operating system using Mesos as its kernel. Couchbase on Mesos using DC/OS and Amazon explained how to run a single Couchbase container on DC/OS using CloudFormation templates.

Running a single node Couchbase may work during initial development phase. The need to start a multi-node Couchbase cluster becomes eminent as you move along further in development, and certainly needed during the production phase.

So, you’d like to run a Couchbase cluster on DC/OS?

Couchbase Cluster on DC/OS is complete walk through of how to setup a Couchbase cluster on DC/OS. It walks through the following steps:

• What is Couchbase?
• Couchbase Cluster
• Setup DC/OS on Amazon Web Services
• Configure CLI and Install Marathon Load Balancer
• Create Couchbase “startup” service
• Create Couchbase “node” service
• Scale Couchbase Cluster
• Rebalance Couchbase Cluster
• Conclusion

DC/OS dashboard with a Couchbase cluster looks like:

For further information check out:

• Couchbase on Containers
• Couchbase Developer Portal
• Ask questions on Couchbase Forums or Stack Overflow
• Download Couchbase

You can also follow us at @couchbasedev and @couchbase.

Source: http://blog.couchbase.com/2016/november/couchbase-cluster-mesos-dcos

Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase explains how to get started with Kubernetes  1.4 on Amazon Web Services. A Couchbase service is created in the cluster and a Spring Boot application stores a JSON document in the database. It uses kube-up.sh script from the Kubernetes binary download at github.com/kubernetes/kubernetes/releases/download/v1.4.0/kubernetes.tar.gz  to start the cluster. This script is capable of creating a Kubernetes cluster with single master only. This is a fundamental flaw of distributed applications where the master becomes a Single Point of Failure.

Meet kops – short for Kubernetes Operations.

This is the easiest way to get a highly-available Kubernetes cluster up and running. The kubectl script is the CLI for running commands against running clusters. Think of kops as kubectl for cluster.

This blog will show how to create a highly-available Kubernetes cluster on Amazon using kops. And once the cluster is created, then it’ll create a Couchbase service on it and run a Spring Boot application to store JSON document in the database.

Many thanks to justinsb, sarahz, razic, jaygorrell, shrugs, bkpandey and others at Kubernetes slack channel for helping me through the details!

Download kops and kubectl

• Download Kops latest release. This blog was tested with 1.4.1 on OSX.Complete set of commands for kops can be seen:
  
  kops-darwin-amd64 --help kops is kubernetes ops. It allows you to create, destroy, upgrade and maintain clusters. Usage: kops [command] Available Commands: create create resources delete delete clusters describe describe objects edit edit items export export clusters/kubecfg get list or get objects import import clusters rolling-update rolling update clusters secrets Manage secrets & keys toolbox Misc infrequently used commands update update clusters upgrade upgrade clusters version Print the client version information Flags: --alsologtostderr log to standard error as well as files --config string config file (default is $HOME/.kops.yaml) --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0) --log_dir string If non-empty, write log files in this directory --logtostderr log to standard error instead of files (default false) --name string Name of cluster --state string Location of state storage --stderrthreshold severity logs at or above this threshold go to stderr (default 2) -v, --v Level log level for V logs --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging Use "kops [command] --help" for more information about a command.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

  kops-darwin-amd64 --help kops is kubernetes ops. It allows you to create, destroy, upgrade and maintain clusters.   Usage:   kops [command]   Available Commands:   create         create resources   delete         delete clusters   describe       describe objects   edit           edit items   export         export clusters/kubecfg   get            list or get objects   import         import clusters   rolling-update rolling update clusters   secrets        Manage secrets & keys   toolbox        Misc infrequently used commands   update         update clusters   upgrade        upgrade clusters   version        Print the client version information   Flags:       --alsologtostderr                  log to standard error as well as files       --config string                    config file (default is $HOME/.kops.yaml)       --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)       --log_dir string                   If non-empty, write log files in this directory       --logtostderr                      log to standard error instead of files (default false)       --name string                      Name of cluster       --state string                     Location of state storage       --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)   -v, --v Level                          log level for V logs       --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging   Use "kops [command] --help" for more information about a command.

• Download kubectl:
  
  curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/v1.4.1/bin/darwin/amd64/kubectl && chmod +x kubectl

  1 2 3

  curl -Lo kubectl http://storage.googleapis.com/kubernetes-release/release/v1.4.1/bin/darwin/amd64/kubectl && chmod +x kubectl

•  Include kubectl in your PATH.

Create Bucket and NS Records on Amazon

There is a bit of setup involved at this time, and hopefully this will get cleaned up over next releases. Bringing up a cluster on AWS provide detailed steps and more background. Here is what the blog followed:

• Pick a domain where Kubernetes cluster will be hosted. This blog uses kubernetes.arungupta.me domain. You can pick a top level domain or a sub-domain.
• Amazon Route 53 is a highly available and scalable DNS service. Login to Amazon Console and created a hosted zone for this domain using Route 53 service.Created zone looks like:The values shown in the Value column are important as they’ll be used later for creating NS records.
• Create a S3 bucket using Amazon Console to store cluster configuration – this is called state store.
• The domain kubernetes.arungupta.me is hosted on GoDaddy. For each value shown in the Value column of Route53 hosted zone, create a NS record using GoDaddy Domain Control Center for this domain.Select the type of record:
  For each value, add the record as shown:Completed set of records look like:

Start Kubernetes Multimaster Cluster

Let’s understand a bit about Amazon regions and zones:

Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of regions and Availability Zones. Each region is a separate geographic area. Each region has multiple, isolated locations known as Availability Zones.

A highly-available Kubernetes cluster can be created across zones but not across regions.

• Find out availability zones within a region:
  
  aws ec2 describe-availability-zones --region us-west-2 { "AvailabilityZones": [ { "State": "available", "RegionName": "us-west-2", "Messages": [], "ZoneName": "us-west-2a" }, { "State": "available", "RegionName": "us-west-2", "Messages": [], "ZoneName": "us-west-2b" }, { "State": "available", "RegionName": "us-west-2", "Messages": [], "ZoneName": "us-west-2c" } ] }

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

  aws ec2 describe-availability-zones --region us-west-2 {     "AvailabilityZones": [         {             "State": "available",             "RegionName": "us-west-2",             "Messages": [],             "ZoneName": "us-west-2a"         },         {             "State": "available",             "RegionName": "us-west-2",             "Messages": [],             "ZoneName": "us-west-2b"         },         {             "State": "available",             "RegionName": "us-west-2",             "Messages": [],             "ZoneName": "us-west-2c"         }     ] }

• Create a multi-master cluster:
  
  kops-darwin-amd64 create cluster --name=kubernetes.arungupta.me --cloud=aws --zones=us-west-2a,us-west-2b,us-west-2c --master-size=m4.large --node-count=3 --node-size=m4.2xlarge --master-zones=us-west-2a,us-west-2b,us-west-2c --state=s3://kops-couchbase --yes

  1 2 3

  kops-darwin-amd64 create cluster --name=kubernetes.arungupta.me --cloud=aws --zones=us-west-2a,us-west-2b,us-west-2c --master-size=m4.large --node-count=3 --node-size=m4.2xlarge --master-zones=us-west-2a,us-west-2b,us-west-2c --state=s3://kops-couchbase --yes

  Most of the switches are self-explanatory. Some switches need a bit of explanation:
  • Specifying multiple zones using --master-zones (must be odd number) create multiple masters across AZ
  • --cloud=aws is optional if cloud can be inferred from zones
  • --yes is used to specify the immediate creation of cluster. Otherwise only the state is stored in the bucket, and the cluster needs to be created separately.

  Complete set of CLI switches can be seen:

  ./kops-darwin-amd64 create cluster --help Creates a k8s cluster. Usage: kops create cluster [flags] Flags: --admin-access string Restrict access to admin endpoints (SSH, HTTPS) to this CIDR. If not set, access will not be restricted by IP. --associate-public-ip Specify --associate-public-ip=[true|false] to enable/disable association of public IP for master ASG and nodes. Default is 'true'. (default true) --channel string Channel for default versions and configuration to use (default "stable") --cloud string Cloud provider to use - gce, aws --dns-zone string DNS hosted zone to use (defaults to last two components of cluster name) --image string Image to use --kubernetes-version string Version of kubernetes to run (defaults to version in channel) --master-size string Set instance size for masters --master-zones string Zones in which to run masters (must be an odd number) --model string Models to apply (separate multiple models with commas) (default "config,proto,cloudup") --network-cidr string Set to override the default network CIDR --networking string Networking mode to use. kubenet (default), classic, external. (default "kubenet") --node-count int Set the number of nodes --node-size string Set instance size for nodes --out string Path to write any local output --project string Project to use (must be set on GCE) --ssh-public-key string SSH public key to use (default "~/.ssh/id_rsa.pub") --target string Target - direct, terraform (default "direct") --vpc string Set to use a shared VPC --yes Specify --yes to immediately create the cluster --zones string Zones in which to run the cluster Global Flags: --alsologtostderr log to standard error as well as files --config string config file (default is $HOME/.kops.yaml) --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0) --log_dir string If non-empty, write log files in this directory --logtostderr log to standard error instead of files (default false) --name string Name of cluster --state string Location of state storage --stderrthreshold severity logs at or above this threshold go to stderr (default 2) -v, --v Level log level for V logs --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42

  ./kops-darwin-amd64 create cluster --help Creates a k8s cluster.   Usage:   kops create cluster [flags]   Flags:       --admin-access string         Restrict access to admin endpoints (SSH, HTTPS) to this CIDR.  If not set, access will not be restricted by IP.       --associate-public-ip         Specify --associate-public-ip=[true|false] to enable/disable association of public IP for master ASG and nodes. Default is 'true'. (default true)       --channel string              Channel for default versions and configuration to use (default "stable")       --cloud string                Cloud provider to use - gce, aws       --dns-zone string             DNS hosted zone to use (defaults to last two components of cluster name)       --image string                Image to use       --kubernetes-version string   Version of kubernetes to run (defaults to version in channel)       --master-size string          Set instance size for masters       --master-zones string         Zones in which to run masters (must be an odd number)       --model string                Models to apply (separate multiple models with commas) (default "config,proto,cloudup")       --network-cidr string         Set to override the default network CIDR       --networking string           Networking mode to use.  kubenet (default), classic, external. (default "kubenet")       --node-count int              Set the number of nodes       --node-size string            Set instance size for nodes       --out string                  Path to write any local output       --project string              Project to use (must be set on GCE)       --ssh-public-key string       SSH public key to use (default "~/.ssh/id_rsa.pub")       --target string               Target - direct, terraform (default "direct")       --vpc string                  Set to use a shared VPC       --yes                         Specify --yes to immediately create the cluster       --zones string                Zones in which to run the cluster   Global Flags:       --alsologtostderr                  log to standard error as well as files       --config string                    config file (default is $HOME/.kops.yaml)       --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)       --log_dir string                   If non-empty, write log files in this directory       --logtostderr                      log to standard error instead of files (default false)       --name string                      Name of cluster       --state string                     Location of state storage       --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)   -v, --v Level                          log level for V logs       --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

• Once the cluster is created, get more details about the cluster:
  kubectl cluster-info Kubernetes master is running at https://api.kubernetes.arungupta.me KubeDNS is running at https://api.kubernetes.arungupta.me/api/v1/proxy/namespaces/kube-system/services/kube-dns To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

  1 2 3 4 5 6 7

  kubectl cluster-info Kubernetes master is running at https://api.kubernetes.arungupta.me KubeDNS is running at https://api.kubernetes.arungupta.me/api/v1/proxy/namespaces/kube-system/services/kube-dns   To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

• Check cluster client and server version:
  kubectl version Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", GitCommit:"4957b090e9a4f6a68b4a40375408fdc74a212260", GitTreeState:"clean", BuildDate:"2016-10-16T06:20:04Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

  1 2 3 4 5

  kubectl version Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", GitCommit:"4957b090e9a4f6a68b4a40375408fdc74a212260", GitTreeState:"clean", BuildDate:"2016-10-16T06:20:04Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

• Check all nodes in the cluster:
  kubectl get nodes NAME STATUS AGE ip-172-20-111-151.us-west-2.compute.internal Ready 1h ip-172-20-116-40.us-west-2.compute.internal Ready 1h ip-172-20-48-41.us-west-2.compute.internal Ready 1h ip-172-20-49-105.us-west-2.compute.internal Ready 1h ip-172-20-80-233.us-west-2.compute.internal Ready 1h ip-172-20-82-93.us-west-2.compute.internal Ready 1h

  1 2 3 4 5 6 7 8 9 10

  kubectl get nodes NAME                                           STATUS    AGE ip-172-20-111-151.us-west-2.compute.internal   Ready     1h ip-172-20-116-40.us-west-2.compute.internal    Ready     1h ip-172-20-48-41.us-west-2.compute.internal     Ready     1h ip-172-20-49-105.us-west-2.compute.internal    Ready     1h ip-172-20-80-233.us-west-2.compute.internal    Ready     1h ip-172-20-82-93.us-west-2.compute.internal     Ready     1h

  Or find out only the master nodes:

  kubectl get nodes -l kubernetes.io/role=master NAME STATUS AGE ip-172-20-111-151.us-west-2.compute.internal Ready 1h ip-172-20-48-41.us-west-2.compute.internal Ready 1h ip-172-20-82-93.us-west-2.compute.internal Ready 1h

  1 2 3 4 5 6 7

  kubectl get nodes -l kubernetes.io/role=master NAME                                           STATUS    AGE ip-172-20-111-151.us-west-2.compute.internal   Ready     1h ip-172-20-48-41.us-west-2.compute.internal     Ready     1h ip-172-20-82-93.us-west-2.compute.internal     Ready     1h

• Check all the clusters:
  kops-darwin-amd64 get clusters --state=s3://kops-couchbase NAME CLOUD ZONES kubernetes.arungupta.me aws us-west-2a,us-west-2b,us-west-2c

  1 2 3 4 5

  kops-darwin-amd64 get clusters --state=s3://kops-couchbase NAME            CLOUD    ZONES kubernetes.arungupta.me    aws    us-west-2a,us-west-2b,us-west-2c

Kubernetes Dashboard Addon

By default, a cluster created using kops does not have the UI dashboard. But this can be added as an add on:

Now complete details about the cluster can be seen:

And the Kubernetes UI dashboard is at the shown URL. In our case, this is https://api.kubernetes.arungupta.me/ui and looks like:

Credentials for accessing this dashboard can be obtained using the kubectl config view command. The values are shown like:

Deploy Couchbase Service

As explained in Getting Started with Kubernetes 1.4 using Spring Boot and Couchbase, let’s run a Couchbase service:

This configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/couchbase-service.yml.

Get the list of services:

Describe the service:

Get the pods:

Run Spring Boot Application

The Spring Boot application runs against the Couchbase service and stores a JSON document in it.

Start the Spring Boot application:

This configuration file is at github.com/arun-gupta/kubernetes-java-sample/blob/master/maven/bootiful-couchbase.yml.

See list of all the pods:

Check logs of the complete pod:

The updated dashboard now looks like:

Delete the Kubernetes Cluster

Kubernetes cluster can be deleted as:

couchbase.com/containers provide more details about how to run Couchbase in different container frameworks.

More information about Couchbase:

• Couchbase Developer Portal
• Couchbase Forums
• @couchbasedev or @couchbase

Source: blog.couchbase.com/2016/november/multimaster-kubernetes-cluster-amazon-kops