Monthly Archives: May 2016

Docker for Java Developers with IntelliJ IDEA Webinar Replay

May 31, 2016conferences, containerscouchbase, docker, intellijarungupta

Video recording from Docker for Java Developers webinar is now available:

Here are answers to the questions asked:

What do you think of Kubernetes compared to Docker and Docker Swarm? Kubernetes or Docker Swarm? Which way to go?
It depends 😉

Docker Swarm has a much simpler learning curve as opposed to Kubernetes. Kubernetes is very opinionated but can be very powerful once you understand how it works.

Some of the points worth considering are:
- Easy to setup/start
- Need to learn new concepts or aligned with Docker
- How can it be aligned with your existing workflow
- Single or multiple master
- Support for multiple clouds
- Cluster across multiple clouds
- Dynamic scaling up/down
- Rescheduling containers on node/container failure
- Pluggable scheduler strategies (if you want to go fancy)
- Service discovery
- Persistent volumes
If this question was asked a few months ago, then Kubernetes would have been the obvious choice. But Docker Swarm is really cranking out releases and features and has caught up well. Check out Docker Swarm Exceeds Kubernetes Performance at Scale. Hackernews has some interesting discussion on this topic as well. Also read Evaluating Container Platforms at Scale.

In addition, you should also look at Mesos+Marathon or DC/OS.

Here are some blogs to get started:
- Couchbase cluster on Docker Swarm using Compose and Machine
- Kubernetes Cluster on Google Cloud and Expose Couchbase Service
- Kubernetes Cluster on Amazon and Expose Couchbase Service
- Couchbase on DC/OS and Amazon
What is the best practice to develop against a docker WildFly container if you do not want to bundle the WAR file after each change but use an exploded archive instead in order to just repackage static resources for example?
Docker allows mapping directories on your filesystem as volumes in the container. This can be used to develop your application against a WildFly container without the need to be bundle the WAR with each change. This is explained in more detail at Deploy to WildFly Docker from Eclipse.
Does IntelliJ Idea support native docker on Mac OS/Windows?
Andrey: Not yet but it’s on the roadmap.
What are pros and cons of providing environment variable vs link like consul:consul?
Docker Links are legacy and so environment variables and Docker Networking are the recommended way to build any new applications. Read more details in another answer below.
How do we attach a debugger to a Java process running in a container?
Read this great blog entry by Daniel Bryant on Debugging Java Applications running in Docker.
What is the relation between Docker and Vagrant? Can both work together? Is docker a substitute of Vagrant?
Docker and Vagrant solve different problems.

Vagrant allows you to provision Virtual Machines independent of the host machine. This is done by using a hypervisor, such as VirtualBox. The definition for the VM is stored in a Vagrantfile, and defines how to setup the VM and what scripts to provision the environment. These VMs can be created on Linux, Windows or Mac.

Docker allows you to create containers. It is a native Linux technology based upon lxc and coming to Windows Server 2016 this Summer. Docker for Mac provides development environment for Mac.

If you need to provision a Docker Engine, then you can create a Vagrantfile that provisions Ubuntu or CentOS using Virtual Box. Then run apt get or yum install scripts to install Docker Engine. If a new version of Docker Engine is available, then you are responsible for updating the Vagrant image as well, or run the script again.

Another option is to use Docker Machine and use VirtualBox driver to create a machine that will have Docker Engine installed. The docker-machine script has commands like upgrade that simply upgrades the Docker Engine.

Some more interesting discussions are at StackOverflow and Quora.

To make this whole equation further interesting, Vagrant can be backed by Docker containers instead of VMs!
Docker tool window is missing in my IntelliJ IDEA 15. Plugin is installed.
Andrey: The separate tool window was added in 2016.1. Before that Docker was a part of the Application Servers tool window.
Is it possible to change index.jsp and see the change in the browser without building the image again? In the spirit of update resources action in IntelliJ IDEA?
Andrey: Sounds like a great idea. Please submit it as a feature request to tracker: https://youtrack.jetbrains.com/issues/IDEA.
Is there a performance increase for using xhyve/Hyper-V over VirtualBox?

Docker for Mac runs Docker Engine in Alpine Linux distribution on top of xhyve Virtual Machine for Mac OS X or on Hyper-V on Windows. xhyve is built on Hypervisor.framework in OS X 10.10+. Some of the primary advantages of using a native technology as opposed to a Type 2 hypervisor is native look-and-feel, better performance, and better volume management.

Read more details in (3-months old blog) xhyve vs VirtualBox driver benchmarks. This area is fast evolving and keep an eye on blog.docker.com for more details.
In Docker Compose, how do I update only one container, does it lose the “link” do I need to redeploy everything?
Docker Links are now legacy. It’s highly recommended to migrate your application to use Docker Networking. The main difference between Links and Networking are:
1. Connect containers to each other across different physical or virtual hosts
2. Containers using Networking can be easily stopped, started and restarted without disrupting the connections to other containers
3. You don’t need to create a container before you can link to it. With Networking containers be created in any order and discover each other using their container names
How can one expose a host port to a container running on the host?
Ports in a container are exposed on the host using -p with docker run. Ports on the host can be accessed using <host-ip>:<port> directly.
Official Java Docker image (OpenJDK-based) is not TCK certified. What do you suggest in this case?
OpenJDK is the Reference Implementation of Java SE and so is TCK compliant. Official Java Docker image is based on OpenJDK.Azul Systems provide commercial support for OpenJDK. Another option is to use OpenJDK images available from Azul systems. These are fully tested and compatibility verified builds of the latest versions of the OpenJDK 8, 7, and 6 platforms.

Oracle publishes Dockerfile for JDK but don’t publish the images. Note that this definition uses oraclelinux:latest as the base image and you may want to use ubuntu, centos or alpine. In this case, you can create your own Dockerfile.

Running Java on Docker? You’re breaking the law provides a good summary of why Oracle JDK-based image should not be pushed to Docker Hub.

How do you have a container with WildFly use a container with the db on another server?

Docker Links are legacy and only allow containers to talk to each on the same host. The recommended way for two containers to communicate is using Docker Networks.By default, Docker Swarm cluster creates an overlay network that allows multiple containers to talk across hosts. Multi-container and multi-host applications using WildFly and Couchbase shows how to connect WildFly with Couchbase where each container is running on two separate hosts on a Docker Swarm cluster.Here is the Compose file for reference:

mycouchbase:
  container_name: "db"
  image: couchbase/server
  ports:
    - 8091:8091
    - 8092:8092
    - 8093:8093
    - 11210:11210
mywildfly:
  image: arungupta/wildfly-admin
  environment:
    - COUCHBASE_URI=db
  ports:
    - 8080:8080
    - 9990:9990

mycouchbase:

container_name: "db"

image: couchbase/server

ports:

- 8091:8091

- 8092:8092

- 8093:8093

- 11210:11210

mywildfly:

image: arungupta/wildfly-admin

environment:

- COUCHBASE_URI=db

ports:

- 8080:8080

- 9990:9990

Here are some more references:

Learn all about IntelliJ and Docker at jetbrains.com/idea/docker/
Self-paced Docker for Java Developers workshop
Docker for Java developers Introductory slides
Script for Docker and IntelliJ demo shown in the video
Docker Docs
Couchbase on Containers

Source: http://blog.couchbase.com/2016/may/docker-java-intellij-idea-webinar-replay

Couchbase XDCR using Docker Swarm, Machine and Compose

May 28, 2016containers, couchbasecouchbase, dockerarungupta

Cross Datacenter Replication (XDCR) in Couchbase provides an easy way to replicate data from one cluster to another. The clusters are typically set in geographically diverse data centers. This enables for disaster recovery or to bring data closer to users for faster data access.

This blog will show:

Setup two data centers using Docker Swarm
Run Couchbase containers on each node of Docker Swarm
Setup a Couchbase cluster on each Docker Swarm cluster
Configure one-way XDCR between two Couchbase clusters

For the purpose of this blog, the two data centers will be setup on a local machine using Docker Machine.

Complete code used in this blog is available at: github.com/arun-gupta/couchbase-xdcr-docker.

Create Consul Discovery Service

Each node in Docker Swarm needs to be registered with a discovery service. This blog will use Consul for that purpose. And even Consul will be running on a Docker Machine.

Typically, you’ll run a cluster of Consul but for simplicity a single instance is started in our case.

Create a Docker Machine and start Consul using this script:

# Docker Machine for Consul
docker-machine \
   create \
   -d virtualbox \
   consul-machine

# Start Consul
docker $(docker-machine config consul-machine) run -d --restart=always \
         -p "8500:8500" \
         -h "consul" \
         progrium/consul -server -bootstrap

# Docker Machine for Consul

docker-machine \

create \

-d virtualbox \

consul-machine

# Start Consul

docker $(docker-machine config consul-machine) run -d --restart=always \

-p "8500:8500" \

-h "consul" \

progrium/consul -server -bootstrap

Create Docker Swarm cluster

Docker Swarm allows multiple Docker hosts to be viewed as a single unit. This allows your multi-container applications to easily run on multiple hosts. Docker Swarm serves the same Remote API as served by a single host. This allows your existing tools to target a single host or a cluster of hosts.

Both the Docker Swarm clusters will be registered with a single discovery service. This is achieved by using the following value for --swarm-discovery:

consul://$(docker-machine ip consul-machine):8500/v1/kv/<key>

consul://$(docker-machine ip consul-machine):8500/v1/kv/<key>

Create a Docker Swarm cluster using Docker Machine using this script:

# Docker Swarm master
docker-machine \
  create \
  -d virtualbox \
  --swarm \
  --swarm-master \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \
  --engine-opt="cluster-advertise=eth1:2376" \
  swarm-master-$1

# Docker Swarm node-01
docker-machine \
  create \
  -d virtualbox \
  --swarm \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \
  --engine-opt="cluster-advertise=eth1:2376" \
  swarm-node-$1-01

# Docker Swarm node-02
docker-machine \
  create \
  -d virtualbox \
  --swarm \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \
  --engine-opt="cluster-advertise=eth1:2376" \
  swarm-node-$1-02

# Configure to use Docker Swarm cluster
eval "$(docker-machine env --swarm swarm-master-$1)"

# Docker Swarm master

docker-machine \

create \

-d virtualbox \

--swarm \

--swarm-master \

--swarm-discovery="consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \

--engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \

--engine-opt="cluster-advertise=eth1:2376" \

swarm-master-$1

# Docker Swarm node-01

docker-machine \

create \

-d virtualbox \

--swarm \

--swarm-discovery="consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \

--engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \

--engine-opt="cluster-advertise=eth1:2376" \

swarm-node-$1-01

# Docker Swarm node-02

docker-machine \

create \

-d virtualbox \

--swarm \

--swarm-discovery="consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \

--engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500/v1/kv/cluster$1" \

--engine-opt="cluster-advertise=eth1:2376" \

swarm-node-$1-02

# Configure to use Docker Swarm cluster

eval "$(docker-machine env --swarm swarm-master-$1)"

The script needs to be invoked as:

./create-docker-swarm-cluster.sh A
./create-docker-swarm-cluster.sh B

./create-docker-swarm-cluster.sh A

./create-docker-swarm-cluster.sh B

This will create two Docker Swarm clusters with one “master” and two “worker” as shown below:

NAME              ACTIVE   DRIVER       STATE     URL                         SWARM                     DOCKER    ERRORS
consul-machine    -        virtualbox   Running   tcp://192.168.99.101:2376                             v1.11.1   
default           *        virtualbox   Running   tcp://192.168.99.100:2376                             v1.11.1   
swarm-master-A    -        virtualbox   Running   tcp://192.168.99.102:2376   swarm-master-A (master)   v1.11.1   
swarm-master-B    -        virtualbox   Running   tcp://192.168.99.105:2376   swarm-master-B (master)   v1.11.1   
swarm-node-A-01   -        virtualbox   Running   tcp://192.168.99.103:2376   swarm-master-A            v1.11.1   
swarm-node-A-02   -        virtualbox   Running   tcp://192.168.99.104:2376   swarm-master-A            v1.11.1   
swarm-node-B-01   -        virtualbox   Running   tcp://192.168.99.106:2376   swarm-master-B            v1.11.1   
swarm-node-B-02   -        virtualbox   Running   tcp://192.168.99.107:2376   swarm-master-B            v1.11.1

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS

consul-machine - virtualbox Running tcp://192.168.99.101:2376 v1.11.1

default * virtualbox Running tcp://192.168.99.100:2376 v1.11.1

swarm-master-A - virtualbox Running tcp://192.168.99.102:2376 swarm-master-A (master) v1.11.1

swarm-master-B - virtualbox Running tcp://192.168.99.105:2376 swarm-master-B (master) v1.11.1

swarm-node-A-01 - virtualbox Running tcp://192.168.99.103:2376 swarm-master-A v1.11.1

swarm-node-A-02 - virtualbox Running tcp://192.168.99.104:2376 swarm-master-A v1.11.1

swarm-node-B-01 - virtualbox Running tcp://192.168.99.106:2376 swarm-master-B v1.11.1

swarm-node-B-02 - virtualbox Running tcp://192.168.99.107:2376 swarm-master-B v1.11.1

Consul is running on Docker Machine with IP address 192.168.99.101. And so Consul UI is accessible at http://192.168.99.101:8500:

It shows two Docker Swarm clusters that have been registered.

Exact list of nodes for each cluster can also be seen. Nodes in clusterA are shown:

swarm-xdcr-consul-clusterA

Nodes in clusterB are shown:

swarm-xdcr-consul-clusterB

Run Couchbase containers

Run Couchbase container on each node of Docker Swarm cluster using this Compose file.

version: "2"
services:
  db:
    image: arungupta/couchbase
    network_mode: "host"
    ports:
      - 8091:8091
      - 8092:8092
      - 8093:8093
      - 11210:11210

version: "2"

services:

db:

image: arungupta/couchbase

network_mode: "host"

ports:

- 8091:8091

- 8092:8092

- 8093:8093

- 11210:11210

Configure Docker CLI for the first cluster and run 3 containers:

eval "$(docker-machine env --swarm swarm-master-A)"
docker-compose scale db=3

eval "$(docker-machine env --swarm swarm-master-A)"

docker-compose scale db=3

Check the running containers:

> docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
3ec0f15aaee0        arungupta/couchbase   "/entrypoint.sh /opt/"   3 hours ago         Up 3 hours                              swarm-master-A/couchbasexdcrdocker_db_3
07af2ac53539        arungupta/couchbase   "/entrypoint.sh /opt/"   3 hours ago         Up 3 hours                              swarm-node-A-02/couchbasexdcrdocker_db_2
c94878f543fd        arungupta/couchbase   "/entrypoint.sh /opt/"   3 hours ago         Up 3 hours                              swarm-node-A-01/couchbasexdcrdocker_db_1

> docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

3ec0f15aaee0 arungupta/couchbase "/entrypoint.sh /opt/" 3 hours ago Up 3 hours swarm-master-A/couchbasexdcrdocker_db_3

07af2ac53539 arungupta/couchbase "/entrypoint.sh /opt/" 3 hours ago Up 3 hours swarm-node-A-02/couchbasexdcrdocker_db_2

c94878f543fd arungupta/couchbase "/entrypoint.sh /opt/" 3 hours ago Up 3 hours swarm-node-A-01/couchbasexdcrdocker_db_1

Configure Docker CLI for the second cluster and run 3 containers:

eval "$(docker-machine env --swarm swarm-master-B)"
docker-compose scale db=3

eval "$(docker-machine env --swarm swarm-master-B)"

docker-compose scale db=3

Check the running containers:

> eval "$(docker-machine env --swarm swarm-master-B)"
> docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
3e3a45480939        arungupta/couchbase   "/entrypoint.sh /opt/"   3 hours ago         Up 3 hours                              swarm-master-B/couchbasexdcrdocker_db_3
1f31f23e337d        arungupta/couchbase   "/entrypoint.sh /opt/"   3 hours ago         Up 3 hours                              swarm-node-B-01/couchbasexdcrdocker_db_1
1feab04c494c        arungupta/couchbase   "/entrypoint.sh /opt/"   3 hours ago         Up 3 hours                              swarm-node-B-02/couchbasexdcrdocker_db_2

> eval "$(docker-machine env --swarm swarm-master-B)"

> docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

3e3a45480939 arungupta/couchbase "/entrypoint.sh /opt/" 3 hours ago Up 3 hours swarm-master-B/couchbasexdcrdocker_db_3

1f31f23e337d arungupta/couchbase "/entrypoint.sh /opt/" 3 hours ago Up 3 hours swarm-node-B-01/couchbasexdcrdocker_db_1

1feab04c494c arungupta/couchbase "/entrypoint.sh /opt/" 3 hours ago Up 3 hours swarm-node-B-02/couchbasexdcrdocker_db_2

Create/Rebalance Couchbase cluster

Scaling and Rebalancing Couchbase Cluster using CLI explains how to create a cluster of Couchbase nodes and rebalance an existing cluster using Couchbase CLI.

Create Couchbase cluster on each Swarm cluster using this script.

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli
for node in 01 02
do
    $COUCHBASE_CLI \
        server-add \
        --cluster=$(docker-machine ip swarm-master-$1):8091 \
        --user Administrator \
        --password password \
        --server-add=$(docker-machine ip swarm-node-$1-$node) \
        --server-add-username=Administrator \
        --server-add-password=password
done

$COUCHBASE_CLI \
        setting-cluster \
        --cluster=$(docker-machine ip swarm-master-$1):8091 \
        --user Administrator \
        --password password \
        --cluster-name=cluster$1

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli

for node in 01 02

$COUCHBASE_CLI \

server-add \

--cluster=$(docker-machine ip swarm-master-$1):8091 \

--user Administrator \

--password password \

--server-add=$(docker-machine ip swarm-node-$1-$node) \

--server-add-username=Administrator \

--server-add-password=password

done

$COUCHBASE_CLI \

setting-cluster \

--cluster=$(docker-machine ip swarm-master-$1):8091 \

--user Administrator \

--password password \

--cluster-name=cluster$1

The script needs to be invoked as:

./create-couchbase-cluster.sh A

./create-couchbase-cluster.sh A

And now rebalance this cluster using this script:

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli
$COUCHBASE_CLI \
rebalance \
--cluster=$(docker-machine ip swarm-master-$1):8091 \
--user Administrator \
--password password \
--server-add-username=Administrator \
--server-add-password=password

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli

$COUCHBASE_CLI \

rebalance \

--cluster=$(docker-machine ip swarm-master-$1):8091 \

--user Administrator \

--password password \

--server-add-username=Administrator \

--server-add-password=password

This script is invoked as:

./rebalance-couchbase-cluster.sh A

./rebalance-couchbase-cluster.sh A

Couchbase Web Console for any node in the cluster will show the output:

swarm-xdcr-couchbase-clusterA

Invoke this script to create the second Couchbase cluster as:

./create-couchbase-cluster.sh B

./create-couchbase-cluster.sh B

Rebalance this cluster as:

./rebalance-couchbase-cluster.sh B

./rebalance-couchbase-cluster.sh B

Couchbase Web Console for any node in the second cluster will show the output:

swarm-xdcr-couchbase-clusterB

Setup XDCR

Cross datacenter replication can be setup to be uni-directional, bi-directional or multi-directional. Uni-directional allows data to replicated from source cluster to destination cluster, bi-directional allows replication both ways, multi-directional allows to configure in any direction.

We’ll create a simple uni-directional replication using this script:

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli
$COUCHBASE_CLI \
        xdcr-setup \
        --cluster=$(docker-machine ip swarm-master-$1):8091 \
        --user Administrator \
        --password password \
        --create \
        --xdcr-cluster-name=cluster$1 \
        --xdcr-hostname=$(docker-machine ip swarm-master-$2):8091 \
        --xdcr-username=Administrator \
        --xdcr-password=password \
        --xdcr-demand-encryption=0

$COUCHBASE_CLI \
        xdcr-replicate \
        --cluster $(docker-machine ip swarm-master-$1):8091 \
        --xdcr-cluster-name=cluster$1 \
        --user Administrator \
        --password password \
        --create \
        --xdcr-from-bucket=travel-sample \
        --xdcr-to-bucket=travel-sample

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli

$COUCHBASE_CLI \

xdcr-setup \

--cluster=$(docker-machine ip swarm-master-$1):8091 \

--user Administrator \

--password password \

--create \

--xdcr-cluster-name=cluster$1 \

--xdcr-hostname=$(docker-machine ip swarm-master-$2):8091 \

--xdcr-username=Administrator \

--xdcr-password=password \

--xdcr-demand-encryption=0

$COUCHBASE_CLI \

xdcr-replicate \

--cluster $(docker-machine ip swarm-master-$1):8091 \

--xdcr-cluster-name=cluster$1 \

--user Administrator \

--password password \

--create \

--xdcr-from-bucket=travel-sample \

--xdcr-to-bucket=travel-sample

This script is invoked as:

./setup-xdcr.sh A B

./setup-xdcr.sh A B

A bi-directional replication can be easily created by executing the commands again but reversing the source and destination cluster.

Couchbase Web Console for the source cluster will show:

Couchbase Web Console for the destination cluster will show:

Enjoy!

This blog shows how you can simplify your complex deployments using Docker Machine, Docker Swarm, and Docker Compose.

Source: http://blog.couchbase.com/2016/may/couchbase-xdcr-docker-swarm-machine-compose

Scaling and Rebalancing a Couchbase Cluster using CLI

May 20, 2016couchbasecouchbasearungupta

Couchbase provides high availability and disaster recovery in several ways:

Homogenous cluster
Replication
- Intra-cluster replication
- Cross-data center replication (XDCR)
Backup and restore
Rack zone awareness

This blog will show how to create a Couchbase cluster using Couchbase Command Line Interface (CLI). In addition, these steps can also be performed using the Couchbase REST API and the Couchbase Web Console.

Couchbase nodes in a cluster are homogeneous. Applications can write to any node and read from any node. There is no write-to-master and read-from-slave architecture that inherently gives you scalability problems. This allows the Couchbase cluster to truly scale horizontally to meet your growing application demands.

Creating a Couchbase cluster involves the following steps:

Provision Couchbase: Provision 1 or more Couchbase nodes, say on Amazon Web Services
Initialize master: Initialize any node to be the “master” of the cluster by calling cluster-init CLI command
Create cluster: For all other nodes, create a cluster by invoking the server-add CLI command
Rebalance cluster: Finally, rebalance the cluster by calling the rebalance CLI command

Provision Couchbase

Provision a Couchbase node on Amazon:

aws ec2 run-instances \
--image-id ami-db95ffbb \
--count 1 \
--instance-type m3.large \
--key-name my-couchbase-key \
--security-groups "Couchbase Server Community Edition-4-0-0-AutogenByAWSMP-"

aws ec2 run-instances \

--image-id ami-db95ffbb \

--count 1 \

--instance-type m3.large \

--key-name my-couchbase-key \

--security-groups "Couchbase Server Community Edition-4-0-0-AutogenByAWSMP-"

Security group name is explained in Couchbase Cluster on Amazon using CLI.

Let’s call this node as “master”.

Initialize Couchbase “master”

Configure the “master” instance and initialize the cluster:

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli
$COUCHBASE_CLI \
        cluster-init \
        -c <master-ip>:8091 \
        -u Administrator \
        -p password \
        --cluster-username Administrator \
        --cluster-password password \
        --cluster-index-ramsize=300 \
        --cluster-ramsize=300 \
        --services=data,index,query

export COUCHBASE_CLI=/Users/arungupta/tools/Couchbase-Server-4.0.app/Contents/Resources/couchbase-core/bin/couchbase-cli

$COUCHBASE_CLI \

cluster-init \

-c <master-ip>:8091 \

-u Administrator \

-p password \

--cluster-username Administrator \

--cluster-password password \

--cluster-index-ramsize=300 \

--cluster-ramsize=300 \

--services=data,index,query

Create another instance, lets call it “worker”. Note, this is not a master/slave architecture. Couchbase cluster is homogenous where any node in the cluster can be “master”.

Create Couchbase Cluster

Add this newly created “worker” instance to the cluster:

$COUCHBASE_CLI \
    server-add \
    --cluster=<master-ip>:8091 \
    --user Administrator \
    --password password \
    --server-add=<worker-ip> \
    --server-add-username=Administrator \
    --server-add-password=password

$COUCHBASE_CLI \

server-add \

--cluster=<master-ip>:8091 \

--user Administrator \

--password password \

--server-add=<worker-ip> \

--server-add-username=Administrator \

--server-add-password=password

Typically, you’ll create and add multiple nodes to the cluster before rebalancing.

Rebalance Couchbase Cluster

Rebalance the cluster:

$COUCHBASE_CLI \
rebalance \
--cluster=<master-ip>:8091 \
--user Administrator \
--password password

$COUCHBASE_CLI \

rebalance \

--cluster=<master-ip>:8091 \

--user Administrator \

--password password

Now, you can create as many instances and easy include them in the cluster.

Adding a single node and rebalancing the cluster can be easily done as a single step:

$COUCHBASE_CLI \
rebalance \
--cluster=<master-ip>:8091 \
--user Administrator \
--password password
--server-add=<worker-ip>
--server-add-username=Administrator
--server-add-password=password

$COUCHBASE_CLI \

rebalance \

--cluster=<master-ip>:8091 \

--user Administrator \

--password password

--server-add=<worker-ip>

--server-add-username=Administrator

--server-add-password=password

Now, your cluster is accessible at http://<master-ip>:8091 or http://<worker-ip>:8091.

Further references …

Couchbase Server Developer Portal
Hello Couchbase Example
Couchbase CLI
Questions on StackOverflow, Forums or Slack Channel
Follow us @couchbasedev
Couchbase 4.5 Beta

Enjoy!

Source: http://blog.couchbase.com/2016/scaling-rebalancing-couchbase-cluster-cli

Couchbase Cluster on Amazon using CLI

May 16, 2016couchbaseamazon, cluster, couchbasearungupta

Couchbase on Amazon Marketplace showed how to setup a single Couchbase node using EC2 Console. But typically you provision these nodes en masse, and more commonly create a cluster of them. Couchbase clusters are homogenous, scales horizontally and thus ensure that database does not become a bottleneck for your high performing application.

This blog will show how to create, scale, and rebalance a Couchbase cluster using AWS Command Line Interface (CLI).

Install AWS CLI

Install the AWS CLI provide complete details, but here is what worked on my machine:

tools > pip install awscli
Collecting awscli
  Downloading awscli-1.10.27-py2.py3-none-any.whl (937kB)
    100% |████████████████████████████████| 937kB 561kB/s 
Collecting rsa<=3.3.0,>=3.1.2 (from awscli)
  Downloading rsa-3.3-py2.py3-none-any.whl (44kB)
    100% |████████████████████████████████| 45kB 7.7MB/s 
Collecting s3transfer==0.0.1 (from awscli)
  Downloading s3transfer-0.0.1-py2.py3-none-any.whl
Collecting colorama<=0.3.3,>=0.2.5 (from awscli)
  Downloading colorama-0.3.3.tar.gz
Collecting botocore==1.4.18 (from awscli)
  Downloading botocore-1.4.18-py2.py3-none-any.whl (2.3MB)
    100% |████████████████████████████████| 2.3MB 221kB/s 
Collecting docutils>=0.10 (from awscli)
  Downloading docutils-0.12.tar.gz (1.6MB)
    100% |████████████████████████████████| 1.6MB 328kB/s 
Collecting pyasn1>=0.1.3 (from rsa<=3.3.0,>=3.1.2->awscli)
  Using cached pyasn1-0.1.9-py2.py3-none-any.whl
Collecting futures<4.0.0,>=2.2.0 (from s3transfer==0.0.1->awscli)
  Using cached futures-3.0.5-py2-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.4.18->awscli)
  Downloading jmespath-0.9.0-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.4.18->awscli)
  Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)
    100% |████████████████████████████████| 204kB 2.4MB/s 
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.4.18->awscli)
  Using cached six-1.10.0-py2.py3-none-any.whl
Building wheels for collected packages: colorama, docutils
  Running setup.py bdist_wheel for colorama ... done
  Stored in directory: /Users/arungupta/Library/Caches/pip/wheels/21/c5/cf/63fb92293f3ad402644ccaf882903cacdb8fe87c80b62c84df
  Running setup.py bdist_wheel for docutils ... done
  Stored in directory: /Users/arungupta/Library/Caches/pip/wheels/db/de/bd/b99b1e12d321fbc950766c58894c6576b1a73ae3131b29a151
Successfully built colorama docutils
Installing collected packages: pyasn1, rsa, futures, jmespath, six, python-dateutil, docutils, botocore, s3transfer, colorama, awscli
Successfully installed awscli-1.10.27 botocore-1.4.18 colorama-0.3.3 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyasn1-0.1.9 python-dateutil-2.5.3 rsa-3.3 s3transfer-0.0.1 six-1.10.0

tools > pip install awscli

Collecting awscli

Downloading awscli-1.10.27-py2.py3-none-any.whl (937kB)

100% |████████████████████████████████| 937kB 561kB/s

Collecting rsa<=3.3.0,>=3.1.2 (from awscli)

Downloading rsa-3.3-py2.py3-none-any.whl (44kB)

100% |████████████████████████████████| 45kB 7.7MB/s

Collecting s3transfer==0.0.1 (from awscli)

Downloading s3transfer-0.0.1-py2.py3-none-any.whl

Collecting colorama<=0.3.3,>=0.2.5 (from awscli)

Downloading colorama-0.3.3.tar.gz

Collecting botocore==1.4.18 (from awscli)

Downloading botocore-1.4.18-py2.py3-none-any.whl (2.3MB)

100% |████████████████████████████████| 2.3MB 221kB/s

Collecting docutils>=0.10 (from awscli)

Downloading docutils-0.12.tar.gz (1.6MB)

100% |████████████████████████████████| 1.6MB 328kB/s

Collecting pyasn1>=0.1.3 (from rsa<=3.3.0,>=3.1.2->awscli)

Using cached pyasn1-0.1.9-py2.py3-none-any.whl

Collecting futures<4.0.0,>=2.2.0 (from s3transfer==0.0.1->awscli)

Using cached futures-3.0.5-py2-none-any.whl

Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.4.18->awscli)

Downloading jmespath-0.9.0-py2.py3-none-any.whl

Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.4.18->awscli)

Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)

100% |████████████████████████████████| 204kB 2.4MB/s

Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.4.18->awscli)

Using cached six-1.10.0-py2.py3-none-any.whl

Building wheels for collected packages: colorama, docutils

Running setup.py bdist_wheel for colorama ... done

Stored in directory: /Users/arungupta/Library/Caches/pip/wheels/21/c5/cf/63fb92293f3ad402644ccaf882903cacdb8fe87c80b62c84df

Running setup.py bdist_wheel for docutils ... done

Stored in directory: /Users/arungupta/Library/Caches/pip/wheels/db/de/bd/b99b1e12d321fbc950766c58894c6576b1a73ae3131b29a151

Successfully built colorama docutils

Installing collected packages: pyasn1, rsa, futures, jmespath, six, python-dateutil, docutils, botocore, s3transfer, colorama, awscli

Successfully installed awscli-1.10.27 botocore-1.4.18 colorama-0.3.3 docutils-0.12 futures-3.0.5 jmespath-0.9.0 pyasn1-0.1.9 python-dateutil-2.5.3 rsa-3.3 s3transfer-0.0.1 six-1.10.0

Configure the CLI:

aws configure
AWS Access Key ID [****************Q5FQ]: YOUR ACCESS KEY ID
AWS Secret Access Key [****************thBR]: YOUR SECRET ACCESS KEY
Default region name [us-west-1]: 
Default output format [None]:

aws configure

AWS Access Key ID [****************Q5FQ]: YOUR ACCESS KEY ID

AWS Secret Access Key [****************thBR]: YOUR SECRET ACCESS KEY

Default region name [us-west-1]:

Default output format [None]:

Enter your access key id and secret access key. These can be obtained as explained in Getting Your Access Key Id and Secret Access Key.

Create AWS Security Group

If you provisioned a server earlier using Amazon 1-click then a security group by the name Couchbase Server Community Edition-4-0-0-AutogenByAWSMP- is created for you. This security group has all the ports authorized required for creating a Couchbase cluster and can be used for creating the instance.

Alternatively, you can create a new security group and explicitly authorize ports.

Create a security group:

aws ec2 create-security-group --group-name my-couchbase-sg --description "My Couchbase Security Group"
{
    "GroupId": "sg-c9bb19ad"
}

aws ec2 create-security-group --group-name my-couchbase-sg --description "My Couchbase Security Group"

{

"GroupId": "sg-c9bb19ad"

}

Authorize ports in the security group:

aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 8091-8093 --protocol tcp --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 11209-11211 --protocol tcp --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 21100-21299 --protocol tcp --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 4369 --protocol tcp --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 8091-8093 --protocol tcp --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 11209-11211 --protocol tcp --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 21100-21299 --protocol tcp --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress --group-name my-couchbase-sg --port 4369 --protocol tcp --cidr 0.0.0.0/0

Create an AWS Key Pair

Read more about creating key pair. Create a key pair:

aws ec2 create-key-pair --key-name my-couchbase-key > my-couchbase-key.pem

aws ec2 create-key-pair --key-name my-couchbase-key > my-couchbase-key.pem

Note the key name as that will be used later.

Create Couchbase Nodes on Amazon

Create two instances using the newly created security group as:

aws ec2 run-instances --image-id ami-db95ffbb --count 2 --instance-type m3.large --key-name my-couchbase-key --security-groups my-couchbase-sg

aws ec2 run-instances --image-id ami-db95ffbb --count 2 --instance-type m3.large --key-name my-couchbase-key --security-groups my-couchbase-sg

Note, the number of instances are specified using --count 2.

AMI ID can be obtained using EC2 console: https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#Images:visibility=public-images;search=couchbase;sort=desc:name.

Or create two instances using the pre-created security group as:

aws ec2 run-instances --image-id ami-db95ffbb --count 2 --instance-type m3.large --key-name my-couchbase-key --security-groups "Couchbase Server Community Edition-4-0-0-AutogenByAWSMP-"

aws ec2 run-instances --image-id ami-db95ffbb --count 2 --instance-type m3.large --key-name my-couchbase-key --security-groups "Couchbase Server Community Edition-4-0-0-AutogenByAWSMP-"

This will show the output as:

{
    "OwnerId": "598307997273", 
    "ReservationId": "r-cb952179", 
    "Groups": [], 
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            }, 
            "PublicDnsName": "", 
            "RootDeviceType": "ebs", 
            "State": {
                "Code": 0, 
                "Name": "pending"
            }, 
            "EbsOptimized": false, 
            "LaunchTime": "2016-05-16T16:11:59.000Z", 
            "PrivateIpAddress": "172.31.15.200", 
            "ProductCodes": [], 
            "VpcId": "vpc-c561f9a0", 
            "StateTransitionReason": "", 
            "InstanceId": "i-90199025", 
            "ImageId": "ami-db95ffbb", 
            "PrivateDnsName": "ip-172-31-15-200.us-west-1.compute.internal", 
            "KeyName": "my-couchbase-key", 
            "SecurityGroups": [
                {
                    "GroupName": "my-couchbase-sg", 
                    "GroupId": "sg-6172dd05"
                }
            ], 
            "ClientToken": "", 
            "SubnetId": "subnet-625bd23b", 
            "InstanceType": "m3.large", 
            "NetworkInterfaces": [
                {
                    "Status": "in-use", 
                    "MacAddress": "06:b1:52:a6:87:f7", 
                    "SourceDestCheck": true, 
                    "VpcId": "vpc-c561f9a0", 
                    "Description": "", 
                    "NetworkInterfaceId": "eni-86ac6fdb", 
                    "PrivateIpAddresses": [
                        {
                            "PrivateDnsName": "ip-172-31-15-200.us-west-1.compute.internal", 
                            "Primary": true, 
                            "PrivateIpAddress": "172.31.15.200"
                        }
                    ], 
                    "PrivateDnsName": "ip-172-31-15-200.us-west-1.compute.internal", 
                    "Attachment": {
                        "Status": "attaching", 
                        "DeviceIndex": 0, 
                        "DeleteOnTermination": true, 
                        "AttachmentId": "eni-attach-530b7293", 
                        "AttachTime": "2016-05-16T16:11:59.000Z"
                    }, 
                    "Groups": [
                        {
                            "GroupName": "my-couchbase-sg", 
                            "GroupId": "sg-6172dd05"
                        }
                    ], 
                    "SubnetId": "subnet-625bd23b", 
                    "OwnerId": "598307997273", 
                    "PrivateIpAddress": "172.31.15.200"
                }
            ], 
            "SourceDestCheck": true, 
            "Placement": {
                "Tenancy": "default", 
                "GroupName": "", 
                "AvailabilityZone": "us-west-1c"
            }, 
            "Hypervisor": "xen", 
            "BlockDeviceMappings": [], 
            "Architecture": "x86_64", 
            "StateReason": {
                "Message": "pending", 
                "Code": "pending"
            }, 
            "RootDeviceName": "/dev/xvda", 
            "VirtualizationType": "hvm", 
            "AmiLaunchIndex": 0
        }, 
        {
            "Monitoring": {
                "State": "disabled"
            }, 
            "PublicDnsName": "", 
            "RootDeviceType": "ebs", 
            "State": {
                "Code": 0, 
                "Name": "pending"
            }, 
            "EbsOptimized": false, 
            "LaunchTime": "2016-05-16T16:11:59.000Z", 
            "PrivateIpAddress": "172.31.15.201", 
            "ProductCodes": [], 
            "VpcId": "vpc-c561f9a0", 
            "StateTransitionReason": "", 
            "InstanceId": "i-93199026", 
            "ImageId": "ami-db95ffbb", 
            "PrivateDnsName": "ip-172-31-15-201.us-west-1.compute.internal", 
            "KeyName": "my-couchbase-key", 
            "SecurityGroups": [
                {
                    "GroupName": "my-couchbase-sg", 
                    "GroupId": "sg-6172dd05"
                }
            ], 
            "ClientToken": "", 
            "SubnetId": "subnet-625bd23b", 
            "InstanceType": "m3.large", 
            "NetworkInterfaces": [
                {
                    "Status": "in-use", 
                    "MacAddress": "06:83:e8:06:2a:f1", 
                    "SourceDestCheck": true, 
                    "VpcId": "vpc-c561f9a0", 
                    "Description": "", 
                    "NetworkInterfaceId": "eni-81ac6fdc", 
                    "PrivateIpAddresses": [
                        {
                            "PrivateDnsName": "ip-172-31-15-201.us-west-1.compute.internal", 
                            "Primary": true, 
                            "PrivateIpAddress": "172.31.15.201"
                        }
                    ], 
                    "PrivateDnsName": "ip-172-31-15-201.us-west-1.compute.internal", 
                    "Attachment": {
                        "Status": "attaching", 
                        "DeviceIndex": 0, 
                        "DeleteOnTermination": true, 
                        "AttachmentId": "eni-attach-680b72a8", 
                        "AttachTime": "2016-05-16T16:11:59.000Z"
                    }, 
                    "Groups": [
                        {
                            "GroupName": "my-couchbase-sg", 
                            "GroupId": "sg-6172dd05"
                        }
                    ], 
                    "SubnetId": "subnet-625bd23b", 
                    "OwnerId": "598307997273", 
                    "PrivateIpAddress": "172.31.15.201"
                }
            ], 
            "SourceDestCheck": true, 
            "Placement": {
                "Tenancy": "default", 
                "GroupName": "", 
                "AvailabilityZone": "us-west-1c"
            }, 
            "Hypervisor": "xen", 
            "BlockDeviceMappings": [], 
            "Architecture": "x86_64", 
            "StateReason": {
                "Message": "pending", 
                "Code": "pending"
            }, 
            "RootDeviceName": "/dev/xvda", 
            "VirtualizationType": "hvm", 
            "AmiLaunchIndex": 1
        }
    ]
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

{

"OwnerId": "598307997273",

"ReservationId": "r-cb952179",

"Groups": [],

"Instances": [

{

"Monitoring": {

"State": "disabled"

"PublicDnsName": "",

"RootDeviceType": "ebs",

"State": {

"Code": 0,

"Name": "pending"

"EbsOptimized": false,

"LaunchTime": "2016-05-16T16:11:59.000Z",

"PrivateIpAddress": "172.31.15.200",

"ProductCodes": [],

"VpcId": "vpc-c561f9a0",

"StateTransitionReason": "",

"InstanceId": "i-90199025",

"ImageId": "ami-db95ffbb",

"PrivateDnsName": "ip-172-31-15-200.us-west-1.compute.internal",

"KeyName": "my-couchbase-key",

"SecurityGroups": [

{

"GroupName": "my-couchbase-sg",

"GroupId": "sg-6172dd05"

}

"ClientToken": "",

"SubnetId": "subnet-625bd23b",

"InstanceType": "m3.large",

"NetworkInterfaces": [

{

"Status": "in-use",

"MacAddress": "06:b1:52:a6:87:f7",

"SourceDestCheck": true,

"VpcId": "vpc-c561f9a0",

"Description": "",

"NetworkInterfaceId": "eni-86ac6fdb",

"PrivateIpAddresses": [

{

"PrivateDnsName": "ip-172-31-15-200.us-west-1.compute.internal",

"Primary": true,

"PrivateIpAddress": "172.31.15.200"

}

"PrivateDnsName": "ip-172-31-15-200.us-west-1.compute.internal",

"Attachment": {

"Status": "attaching",

"DeviceIndex": 0,

"DeleteOnTermination": true,

"AttachmentId": "eni-attach-530b7293",

"AttachTime": "2016-05-16T16:11:59.000Z"

"Groups": [

{

"GroupName": "my-couchbase-sg",

"GroupId": "sg-6172dd05"

}

"SubnetId": "subnet-625bd23b",

"OwnerId": "598307997273",

"PrivateIpAddress": "172.31.15.200"

}

"SourceDestCheck": true,

"Placement": {

"Tenancy": "default",

"GroupName": "",

"AvailabilityZone": "us-west-1c"

"Hypervisor": "xen",

"BlockDeviceMappings": [],

"Architecture": "x86_64",

"StateReason": {

"Message": "pending",

"Code": "pending"

"RootDeviceName": "/dev/xvda",

"VirtualizationType": "hvm",

"AmiLaunchIndex": 0

{

"Monitoring": {

"State": "disabled"

"PublicDnsName": "",

"RootDeviceType": "ebs",

"State": {

"Code": 0,

"Name": "pending"

"EbsOptimized": false,

"LaunchTime": "2016-05-16T16:11:59.000Z",

"PrivateIpAddress": "172.31.15.201",

"ProductCodes": [],

"VpcId": "vpc-c561f9a0",

"StateTransitionReason": "",

"InstanceId": "i-93199026",

"ImageId": "ami-db95ffbb",

"PrivateDnsName": "ip-172-31-15-201.us-west-1.compute.internal",

"KeyName": "my-couchbase-key",

"SecurityGroups": [

{

"GroupName": "my-couchbase-sg",

"GroupId": "sg-6172dd05"

}

"ClientToken": "",

"SubnetId": "subnet-625bd23b",

"InstanceType": "m3.large",

"NetworkInterfaces": [

{

"Status": "in-use",

"MacAddress": "06:83:e8:06:2a:f1",

"SourceDestCheck": true,

"VpcId": "vpc-c561f9a0",

"Description": "",

"NetworkInterfaceId": "eni-81ac6fdc",

"PrivateIpAddresses": [

{

"PrivateDnsName": "ip-172-31-15-201.us-west-1.compute.internal",

"Primary": true,

"PrivateIpAddress": "172.31.15.201"

}

"PrivateDnsName": "ip-172-31-15-201.us-west-1.compute.internal",

"Attachment": {

"Status": "attaching",

"DeviceIndex": 0,

"DeleteOnTermination": true,

"AttachmentId": "eni-attach-680b72a8",

"AttachTime": "2016-05-16T16:11:59.000Z"

"Groups": [

{

"GroupName": "my-couchbase-sg",

"GroupId": "sg-6172dd05"

}

"SubnetId": "subnet-625bd23b",

"OwnerId": "598307997273",

"PrivateIpAddress": "172.31.15.201"

}

"SourceDestCheck": true,

"Placement": {

"Tenancy": "default",

"GroupName": "",

"AvailabilityZone": "us-west-1c"

"Hypervisor": "xen",

"BlockDeviceMappings": [],

"Architecture": "x86_64",

"StateReason": {

"Message": "pending",

"Code": "pending"

"RootDeviceName": "/dev/xvda",

"VirtualizationType": "hvm",

"AmiLaunchIndex": 1

}

]

}

Status of the instances can be checked as:

aws ec2 describe-instance-status --instance-ids `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters Name=instance-state-name,Values=running --output text`

aws ec2 describe-instance-status --instance-ids `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters Name=instance-state-name,Values=running --output text`

And shows the output as:

{
    "InstanceStatuses": [
        {
            "InstanceId": "i-90199025", 
            "InstanceState": {
                "Code": 16, 
                "Name": "running"
            }, 
            "AvailabilityZone": "us-west-1c", 
            "SystemStatus": {
                "Status": "initializing", 
                "Details": [
                    {
                        "Status": "initializing", 
                        "Name": "reachability"
                    }
                ]
            }, 
            "InstanceStatus": {
                "Status": "initializing", 
                "Details": [
                    {
                        "Status": "initializing", 
                        "Name": "reachability"
                    }
                ]
            }
        }, 
        {
            "InstanceId": "i-93199026", 
            "InstanceState": {
                "Code": 16, 
                "Name": "running"
            }, 
            "AvailabilityZone": "us-west-1c", 
            "SystemStatus": {
                "Status": "initializing", 
                "Details": [
                    {
                        "Status": "initializing", 
                        "Name": "reachability"
                    }
                ]
            }, 
            "InstanceStatus": {
                "Status": "initializing", 
                "Details": [
                    {
                        "Status": "initializing", 
                        "Name": "reachability"
                    }
                ]
            }
        }
    ]
}

{

"InstanceStatuses": [

{

"InstanceId": "i-90199025",

"InstanceState": {

"Code": 16,

"Name": "running"

"AvailabilityZone": "us-west-1c",

"SystemStatus": {

"Status": "initializing",

"Details": [

{

"Status": "initializing",

"Name": "reachability"

}

]

"InstanceStatus": {

"Status": "initializing",

"Details": [

{

"Status": "initializing",

"Name": "reachability"

}

]

}

{

"InstanceId": "i-93199026",

"InstanceState": {

"Code": 16,

"Name": "running"

"AvailabilityZone": "us-west-1c",

"SystemStatus": {

"Status": "initializing",

"Details": [

{

"Status": "initializing",

"Name": "reachability"

}

]

"InstanceStatus": {

"Status": "initializing",

"Details": [

{

"Status": "initializing",

"Name": "reachability"

}

]

}

]

}

Here the status is shown as initializing. It takes a few minutes for the instances to be provisioned. Instances that have passed all the checks can be verified as:

aws ec2 describe-instance-status --filters Name=instance-status.reachability,Values=passed --instance-ids `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters Name=instance-state-name,Values=running --output text`

aws ec2 describe-instance-status --filters Name=instance-status.reachability,Values=passed --instance-ids `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters Name=instance-state-name,Values=running --output text`

At first, it shows the result as:

{
    "InstanceStatuses": []
}

{

"InstanceStatuses": []

}

But once all the instances pass the check, then the results look like:

{
    "InstanceStatuses": [
        {
            "InstanceId": "i-90199025", 
            "InstanceState": {
                "Code": 16, 
                "Name": "running"
            }, 
            "AvailabilityZone": "us-west-1c", 
            "SystemStatus": {
                "Status": "ok", 
                "Details": [
                    {
                        "Status": "passed", 
                        "Name": "reachability"
                    }
                ]
            }, 
            "InstanceStatus": {
                "Status": "ok", 
                "Details": [
                    {
                        "Status": "passed", 
                        "Name": "reachability"
                    }
                ]
            }
        }, 
        {
            "InstanceId": "i-93199026", 
            "InstanceState": {
                "Code": 16, 
                "Name": "running"
            }, 
            "AvailabilityZone": "us-west-1c", 
            "SystemStatus": {
                "Status": "ok", 
                "Details": [
                    {
                        "Status": "passed", 
                        "Name": "reachability"
                    }
                ]
            }, 
            "InstanceStatus": {
                "Status": "ok", 
                "Details": [
                    {
                        "Status": "passed", 
                        "Name": "reachability"
                    }
                ]
            }
        }
    ]
}

{

"InstanceStatuses": [

{

"InstanceId": "i-90199025",

"InstanceState": {

"Code": 16,

"Name": "running"

"AvailabilityZone": "us-west-1c",

"SystemStatus": {

"Status": "ok",

"Details": [

{

"Status": "passed",

"Name": "reachability"

}

]

"InstanceStatus": {

"Status": "ok",

"Details": [

{

"Status": "passed",

"Name": "reachability"

}

]

}

{

"InstanceId": "i-93199026",

"InstanceState": {

"Code": 16,

"Name": "running"

"AvailabilityZone": "us-west-1c",

"SystemStatus": {

"Status": "ok",

"Details": [

{

"Status": "passed",

"Name": "reachability"

}

]

"InstanceStatus": {

"Status": "ok",

"Details": [

{

"Status": "passed",

"Name": "reachability"

}

]

}

]

}

Here the status is shown as passed.

Configure Couchbase Nodes

Each Couchbase node needs to be provisioned with the following details:

Memory
Services (index, data and query)
Auth credentials (username: Administrator, password: password)
Loads travel-sample bucket

This can be done using the script:

for ip in "$@"
do
        curl -v -X POST http://$ip:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300
        curl -v http://$ip:8091/node/controller/setupServices -d services=kv%2Cn1ql%2Cindex
        curl -v http://$ip:8091/settings/web -d port=8091 -d username=Administrator -d password=password
        curl -v -u Administrator:password -X POST http://$ip:8091/sampleBuckets/install -d '["travel-sample"]'
done

for ip in "$@"

curl -v -X POST http://$ip:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300

curl -v http://$ip:8091/node/controller/setupServices -d services=kv%2Cn1ql%2Cindex

curl -v http://$ip:8091/settings/web -d port=8091 -d username=Administrator -d password=password

curl -v -u Administrator:password -X POST http://$ip:8091/sampleBuckets/install -d '["travel-sample"]'

done

This is available at: https://github.com/arun-gupta/couchbase-amazon/blob/master/configure-instance.sh.

It can be invoked as:

./configure-instance.sh `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicDnsName]' --filters Name=instance-state-name,Values=running --output text`

./configure-instance.sh `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicDnsName]' --filters Name=instance-state-name,Values=running --output text`

And shows the output as:

*   Trying 52.53.171.49...
* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)
> POST /pools/default HTTP/1.1
> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 36
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 36 out of 36 bytes
< HTTP/1.1 200 OK
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:47 GMT
< Content-Length: 0
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact
*   Trying 52.53.171.49...
* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)
> POST /node/controller/setupServices HTTP/1.1
> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 26
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 26 out of 26 bytes
< HTTP/1.1 200 OK
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:47 GMT
< Content-Length: 0
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact
*   Trying 52.53.171.49...
* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)
> POST /settings/web HTTP/1.1
> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 50
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 50 out of 50 bytes
< HTTP/1.1 200 OK
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:47 GMT
< Content-Type: application/json
< Content-Length: 78
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact
{"newBaseUri":"http://ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091/"}*   Trying 52.53.171.49...
* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)
* Server auth using Basic with user 'Administrator'
> POST /sampleBuckets/install HTTP/1.1
> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091
> Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 202 Accepted
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:47 GMT
< Content-Type: application/json
< Content-Length: 2
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact
[]*   Trying 52.53.239.100...
* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)
> POST /pools/default HTTP/1.1
> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 36
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 36 out of 36 bytes
< HTTP/1.1 200 OK
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:48 GMT
< Content-Length: 0
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact
*   Trying 52.53.239.100...
* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)
> POST /node/controller/setupServices HTTP/1.1
> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 26
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 26 out of 26 bytes
< HTTP/1.1 200 OK
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:48 GMT
< Content-Length: 0
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact
*   Trying 52.53.239.100...
* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)
> POST /settings/web HTTP/1.1
> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 50
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 50 out of 50 bytes
< HTTP/1.1 200 OK
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:48 GMT
< Content-Type: application/json
< Content-Length: 79
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact
{"newBaseUri":"http://ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091/"}*   Trying 52.53.239.100...
* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)
* Server auth using Basic with user 'Administrator'
> POST /sampleBuckets/install HTTP/1.1
> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091
> Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 17
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 202 Accepted
< Server: Couchbase Server
< Pragma: no-cache
< Date: Mon, 16 May 2016 16:18:48 GMT
< Content-Type: application/json
< Content-Length: 2
< Cache-Control: no-cache
< 
* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact
[]

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

* Trying 52.53.171.49...

* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)

> POST /pools/default HTTP/1.1

> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 36

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 36 out of 36 bytes

< HTTP/1.1 200 OK

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:47 GMT

< Content-Length: 0

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact

* Trying 52.53.171.49...

* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)

> POST /node/controller/setupServices HTTP/1.1

> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 26

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 26 out of 26 bytes

< HTTP/1.1 200 OK

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:47 GMT

< Content-Length: 0

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact

* Trying 52.53.171.49...

* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)

> POST /settings/web HTTP/1.1

> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 50

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 50 out of 50 bytes

< HTTP/1.1 200 OK

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:47 GMT

< Content-Type: application/json

< Content-Length: 78

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact

{"newBaseUri":"http://ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091/"}* Trying 52.53.171.49...

* Connected to ec2-52-53-171-49.us-west-1.compute.amazonaws.com (52.53.171.49) port 8091 (#0)

* Server auth using Basic with user 'Administrator'

> POST /sampleBuckets/install HTTP/1.1

> Host: ec2-52-53-171-49.us-west-1.compute.amazonaws.com:8091

> Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 17

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 17 out of 17 bytes

< HTTP/1.1 202 Accepted

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:47 GMT

< Content-Type: application/json

< Content-Length: 2

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-171-49.us-west-1.compute.amazonaws.com left intact

[]* Trying 52.53.239.100...

* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)

> POST /pools/default HTTP/1.1

> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 36

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 36 out of 36 bytes

< HTTP/1.1 200 OK

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:48 GMT

< Content-Length: 0

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact

* Trying 52.53.239.100...

* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)

> POST /node/controller/setupServices HTTP/1.1

> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 26

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 26 out of 26 bytes

< HTTP/1.1 200 OK

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:48 GMT

< Content-Length: 0

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact

* Trying 52.53.239.100...

* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)

> POST /settings/web HTTP/1.1

> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 50

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 50 out of 50 bytes

< HTTP/1.1 200 OK

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:48 GMT

< Content-Type: application/json

< Content-Length: 79

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact

{"newBaseUri":"http://ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091/"}* Trying 52.53.239.100...

* Connected to ec2-52-53-239-100.us-west-1.compute.amazonaws.com (52.53.239.100) port 8091 (#0)

* Server auth using Basic with user 'Administrator'

> POST /sampleBuckets/install HTTP/1.1

> Host: ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091

> Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==

> User-Agent: curl/7.43.0

> Accept: */*

> Content-Length: 17

> Content-Type: application/x-www-form-urlencoded

* upload completely sent off: 17 out of 17 bytes

< HTTP/1.1 202 Accepted

< Server: Couchbase Server

< Pragma: no-cache

< Date: Mon, 16 May 2016 16:18:48 GMT

< Content-Type: application/json

< Content-Length: 2

< Cache-Control: no-cache

* Connection #0 to host ec2-52-53-239-100.us-west-1.compute.amazonaws.com left intact

[]

This is invoking Couchbase REST API to configure each Couchbase node.

Now that each Couchbase node is configured, lets access them. Find public IP address of the instances:

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicDnsName]' --filters Name=instance-state-name,Values=running --output text

aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicDnsName]' --filters Name=instance-state-name,Values=running --output text

It shows the output as:

ec2-52-53-171-49.us-west-1.compute.amazonaws.com
ec2-52-53-239-100.us-west-1.compute.amazonaws.com

ec2-52-53-171-49.us-west-1.compute.amazonaws.com

ec2-52-53-239-100.us-west-1.compute.amazonaws.com

Pick one of the IP addresses and access it at <public-ip-1>:8091 to see the output:

Each Couchbase node is configured with username as Administrator and password as password. Entering the credentials shows Couchbase Web Console:

Click on Server Nodes to see that only a single node is in the cluster:

Create and Rebalance Couchbase Cluster

This is typically a two step process. The first step is to add one or more nodes. The second step is to rebalance the cluster where data on the existing nodes is rebalanced across the updated cluster.

In our case, a Couchbase node is running on each AMI. Lets pick IP address of any one Couchbase node and add IP address of the other node. This can be done using the script:

export primary=$1
shift
for ip in "$@"
do
    $COUCHBASE_CLI \
    server-add \
    --cluster=$primary:8091 \
    --user Administrator \
    --password password \
    --server-add=$ip \
    --server-add-username=Administrator \
    --server-add-password=password
done

export primary=$1

shift

for ip in "$@"

$COUCHBASE_CLI \

server-add \

--cluster=$primary:8091 \

--user Administrator \

--password password \

--server-add=$ip \

--server-add-username=Administrator \

--server-add-password=password

done

This script is available at https://github.com/arun-gupta/couchbase-amazon/blob/master/create-cluster.sh and can be invoked as:

./create-cluster.sh `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicDnsName]' --filters Name=instance-state-name,Values=running --output text`

./create-cluster.sh `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[PublicDnsName]' --filters Name=instance-state-name,Values=running --output text`

And shows the output as:

SUCCESS: server-add ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091

SUCCESS: server-add ec2-52-53-239-100.us-west-1.compute.amazonaws.com:8091

Couchbase Web Console is updated to show:

Finally, rebalance the cluster using the script:

export primary=$1
shift
for ip in "$@" 
do
        $COUCHBASE_CLI \
        rebalance \
        --cluster=$primary:8091 \
        --user Administrator \
        --password password \
        --server-add-username=Administrator \
        --server-add-password=password
done

export primary=$1

shift

for ip in "$@"

$COUCHBASE_CLI \

rebalance \

--cluster=$primary:8091 \

--user Administrator \

--password password \

--server-add-username=Administrator \

--server-add-password=password

done

This shows the output as:

INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
SUCCESS: rebalanced cluster

SUCCESS: rebalanced cluster

Couchbase Web Console is now updated:

Once the cluster is up and running, try out Hello Couchbase Example.

Terminate Couchbase Nodes

Finally, killing the cluster is fairly straight forward:

aws ec2 terminate-instances --instance-ids `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters Name=instance-state-name,Values=running --output text`

aws ec2 terminate-instances --instance-ids `aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId]' --filters Name=instance-state-name,Values=running --output text`

This blog showed how to spin up, scale and rebalance a Couchbase cluster using AWS CLI. All scripts are available at https://github.com/arun-gupta/couchbase-amazon.

Further references …

Couchbase Server Developer Portal
Hello Couchbase Example
Questions on StackOverflow, Forums or Slack Channel
Follow us @couchbasedev
Couchbase 4.5 Beta

Source: http://blog.couchbase.com/2016/may/couchbase-cluster-amazon-using-cli

Couchbase on Amazon Marketplace

May 14, 2016couchbaseamazon, couchbasearungupta

Couchbase Server can be easily downloaded and installed on your local machine. However a common way to run it is on Amazon. This blog will explain how you can run Couchbase on Amazon.

Couchbase Server can be easily launched on Amazon Web Services using Couchbase on AWS Marketplace. Click on Continue and review the settings in 1-Click Launch as shown below:

This is using Couchbase Server 4.0 Community Edition and an m3.large instance.

A new security group with all the relevant ports exposed will be automatically created. This group’s name is Couchbase Server Community Edition-4-0-0-AutogenByAWSMP-. This group is created once and can be repurposed if you launch another instance.

You may have to create a new key pair as explained here.

Click on “Accept Software Terms and Launch with 1-click”. The following confirmation window will be shown:

Wait for a few minutes for the image to be provisioned. AWS EC2 Console should show the status as “2/2 checks passed” as shown:

Get the public DNS address and access Couchbase at http://<public-dns-address>:8091. In our case, this will be http://ec2-54-153-6-241.us-west-1.compute.amazonaws.com:8091/index.html. Note, that this instance is now shutdown and so is not available 😉

Couchbase on Amazon Usage Instructions provide more details about how to use Couchbase on Amazon.

<public-ip-address>:8091 shows a login screen. A default username is Administrator and the password is your instance id:

And clicking on Sign In shows the Couchbase Web Console:

Further references …

Couchbase Server Developer Portal
Couchbase Downloads
Couchbase Server Developer Portal
Questions on StackOverflow, Forums or Slack Channel
Follow us @couchbasedev
Couchbase 4.5 Beta

Enjoy!

Run #Couchbase on #Amazon Market Place using 1-click Click To Tweet

Source: https://blog.couchbase.com/2016/may/couchbase-amazon-marketplace

Couchbase Cluster on Docker Swarm using Docker Compose and Docker Machine

May 13, 2016containers, couchbasecontainers, couchbase, dockerarungupta

This blog post will explain how to create and scale a Couchbase Cluster using full armor of Docker – Docker Machine, Docker Swarm and Docker Compose.

Here is what we’ll do:

Create a 3-node Docker Swarm Cluster using Docker Machine
Run a Couchbase instance on two nodes
Create a cluster
Rebalance the cluster
Scale and rebalance the cluster again

Docker Swarm Cluster using Consul

Create a three-node Docker Swarm cluster using Docker Machine:

# Docker Machine for Consul
docker-machine \
   create \
   -d virtualbox \
   consul-machine

# Start Consul
docker $(docker-machine config consul-machine) run -d --restart=always \
         -p "8500:8500" \
         -h "consul" \
         progrium/consul -server -bootstrap

# Docker Swarm master
docker-machine \
  create \
  -d virtualbox \
  --swarm \
  --swarm-master \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-advertise=eth1:2376" \
  swarm-master

# Docker Swarm node-01
docker-machine \
  create \
  -d virtualbox \
  --swarm \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-advertise=eth1:2376" \
  swarm-node-01

# Docker Swarm node-02
docker-machine \
  create \
  -d virtualbox \
  --virtualbox-disk-size "5000" \
  --swarm \
  --swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \
  --engine-opt="cluster-advertise=eth1:2376" \
  swarm-node-02

# Configure to use Docker Swarm cluster
eval "$(docker-machine env --swarm swarm-master)"

# Docker Machine for Consul

docker-machine \

create \

-d virtualbox \

consul-machine

# Start Consul

docker $(docker-machine config consul-machine) run -d --restart=always \

-p "8500:8500" \

-h "consul" \

progrium/consul -server -bootstrap

# Docker Swarm master

docker-machine \

create \

-d virtualbox \

--swarm \

--swarm-master \

--swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \

--engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \

--engine-opt="cluster-advertise=eth1:2376" \

swarm-master

# Docker Swarm node-01

docker-machine \

create \

-d virtualbox \

--swarm \

--swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \

--engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \

--engine-opt="cluster-advertise=eth1:2376" \

swarm-node-01

# Docker Swarm node-02

docker-machine \

create \

-d virtualbox \

--virtualbox-disk-size "5000" \

--swarm \

--swarm-discovery="consul://$(docker-machine ip consul-machine):8500" \

--engine-opt="cluster-store=consul://$(docker-machine ip consul-machine):8500" \

--engine-opt="cluster-advertise=eth1:2376" \

swarm-node-02

# Configure to use Docker Swarm cluster

eval "$(docker-machine env --swarm swarm-master)"

Provision a Swarm cluster with Docker Machine provide more details about why and what’s done in this script. Here is a summary:

Create a Docker Machine and run Consul for service discovery
Create three Docker Machines – one for Master and two for Worker nodes.Each machine is configured to be part of a Swarm cluster using --swarm. It also uses the Consul service discovery specified using --swarm-discovery.

Couchbase Nodes on Docker Swarm

Create two instances of Couchbase using Docker Compose:

version: "2"
services:
  db:
    image: arungupta/couchbase
    network_mode: "host"
    ports:
      - 8091:8091
      - 8092:8092
      - 8093:8093
      - 11210:11210

version: "2"

services:

db:

image: arungupta/couchbase

network_mode: "host"

ports:

- 8091:8091

- 8092:8092

- 8093:8093

- 11210:11210

arungupta/couchbase image is used here. This image is defined at Couchbase Docker Image. It uses the Official Couchbase Docker Image add configures it as explained:

Setups memory for Index and Data
Configures the Couchbase server with Index, Data, and Query service
Sets up username and password credentials
Loads the travel-sample bucket

Compose file uses host network. This is equivalent to using --net=host on docker run CLI. It allows the container to use the host networking stack. It also limits only a single Couchbase container to run on a single Docker Machine. So this means that our Couchbase cluster can scale based upon the number of Docker Machines – 3 in our case.

The exact command to use this Compose file is:

docker-compose scale db=2
WARNING: The "db" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating and starting couchbasedockerswarm_db_1 ... 
Creating and starting couchbasedockerswarm_db_2 ... 
Pulling db (arungupta/couchbase:latest)...
swarm-node-02: Pulling arungupta/couchbase:latest...
swarm-master: Pulling arungupta/couchbase:latest...
swarm-node-01: Pulling arungupta/couchbase:latest...
Pulling db (arungupta/couchbase:latest)...
swarm-node-02: Pulling arungupta/couchbase:latest... : downloaded
Creating and starting couchbasedockerswarm_db_1 ... done
Creating and starting couchbasedockerswarm_db_2 ... done

docker-compose scale db=2

WARNING: The "db" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.

Creating and starting couchbasedockerswarm_db_1 ...

Creating and starting couchbasedockerswarm_db_2 ...

Pulling db (arungupta/couchbase:latest)...

swarm-node-02: Pulling arungupta/couchbase:latest...

swarm-master: Pulling arungupta/couchbase:latest...

swarm-node-01: Pulling arungupta/couchbase:latest...

Pulling db (arungupta/couchbase:latest)...

swarm-node-02: Pulling arungupta/couchbase:latest... : downloaded

Creating and starting couchbasedockerswarm_db_1 ... done

Creating and starting couchbasedockerswarm_db_2 ... done

There are three nodes in the Docker Swarm cluster. The default scheduler strategy is spread and so the containers will be spread on different hosts.

This is evident by docker ps:

docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
4c8c149f4e34        arungupta/couchbase   "/entrypoint.sh /opt/"   44 seconds ago      Up 44 seconds                           swarm-node-02/couchbasedockerswarm_db_1
d3b6a1dbddb5        arungupta/couchbase   "/entrypoint.sh /opt/"   44 seconds ago      Up 44 seconds                           swarm-node-01/couchbasedockerswarm_db_2

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

4c8c149f4e34 arungupta/couchbase "/entrypoint.sh /opt/" 44 seconds ago Up 44 seconds swarm-node-02/couchbasedockerswarm_db_1

d3b6a1dbddb5 arungupta/couchbase "/entrypoint.sh /opt/" 44 seconds ago Up 44 seconds swarm-node-01/couchbasedockerswarm_db_2

Note, one Couchbase server is running on swarm-node-01 and another on swarm-node-02. Each server is configured with an administrator username Administrator and password password.

Find out IP address of the Docker Machine:

docker-machine ls
NAME             ACTIVE      DRIVER       STATE     URL                         SWARM                   DOCKER    ERRORS
consul-machine   -           virtualbox   Running   tcp://192.168.99.106:2376                           v1.11.1   
default          -           virtualbox   Running   tcp://192.168.99.100:2376                           v1.11.0   
swarm-master     * (swarm)   virtualbox   Running   tcp://192.168.99.107:2376   swarm-master (master)   v1.11.1   
swarm-node-01    -           virtualbox   Running   tcp://192.168.99.108:2376   swarm-master            v1.11.1   
swarm-node-02    -           virtualbox   Running   tcp://192.168.99.109:2376   swarm-master            v1.11.1

docker-machine ls

NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS

consul-machine - virtualbox Running tcp://192.168.99.106:2376 v1.11.1

default - virtualbox Running tcp://192.168.99.100:2376 v1.11.0

swarm-master * (swarm) virtualbox Running tcp://192.168.99.107:2376 swarm-master (master) v1.11.1

swarm-node-01 - virtualbox Running tcp://192.168.99.108:2376 swarm-master v1.11.1

swarm-node-02 - virtualbox Running tcp://192.168.99.109:2376 swarm-master v1.11.1

If you have jq installed then IP address can be conveniently found as:

docker-machine inspect swarm-node-01 | jq ".Driver.IPAddress"
"192.168.99.108"

docker-machine inspect swarm-node-01 | jq ".Driver.IPAddress"

"192.168.99.108"

Couchbase Cluster on Docker Swarm

All Couchbase server nodes are created equal. This allows the Couchbase cluster to truly scale horizontally to meet your growing application demands. Independently running Couchbase nodes can be added to a cluster by invoking the server-add CLI command. This is typically a two step process. The first step is adding one or more nodes. The second step then rebalances the cluster where the data on the existing nodes is rebalanced across the updated cluster.

In our case, a single Couchbase container is running on each Docker Machine. Lets pick IP address of any one Couchbase node, and add IP address of the other node:

docker run -it arungupta/couchbase \
couchbase-cli \
server-add \
--cluster=`docker-machine inspect swarm-node-01 | jq -r ".Driver.IPAddress"`:8091 \
--user Administrator \
--password password \
--server-add=`docker-machine inspect swarm-node-02 | jq -r ".Driver.IPAddress"` \
--server-add-username=Administrator \
--server-add-password=password
SUCCESS: server-add 192.168.99.109:8091

docker run -it arungupta/couchbase \

couchbase-cli \

server-add \

--cluster=`docker-machine inspect swarm-node-01 | jq -r ".Driver.IPAddress"`:8091 \

--user Administrator \

--password password \

--server-add=`docker-machine inspect swarm-node-02 | jq -r ".Driver.IPAddress"` \

--server-add-username=Administrator \

--server-add-password=password

SUCCESS: server-add 192.168.99.109:8091

Couchbase Web Console for both the nodes will show a similar output:

This shows that the two nodes now form a cluster, and needs to be rebalanced.

Rebalance Couchbase Cluster

Now, lets rebalance the cluster:

docker run -it arungupta/couchbase couchbase-cli rebalance --cluster=`docker-machine inspect swarm-node-01 | jq -r ".Driver.IPAddress"`:8091 --user Administrator --password password 
INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
SUCCESS: rebalanced cluster

docker run -it arungupta/couchbase couchbase-cli rebalance --cluster=`docker-machine inspect swarm-node-01 | jq -r ".Driver.IPAddress"`:8091 --user Administrator --password password

INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUCCESS: rebalanced cluster

Couchbase Web Console will be updated to show that rebalance is happening:

And finally you’ll see a rebalanced cluster:

Scale and Rebalance Couchbase Cluster

Scale the Couchbase cluster:

docker-compose scale db=3
WARNING: The "db" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating and starting couchbasedockerswarm_db_3 ... done

docker-compose scale db=3

WARNING: The "db" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.

Creating and starting couchbasedockerswarm_db_3 ... done

Check that the container is running on a different Docker Machine:

docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
02f94e2bbd3e        arungupta/couchbase   "/entrypoint.sh /opt/"   9 seconds ago       Up 8 seconds                            swarm-master/couchbasedockerswarm_db_3
4c8c149f4e34        arungupta/couchbase   "/entrypoint.sh /opt/"   About an hour ago   Up About an hour                        swarm-node-02/couchbasedockerswarm_db_1
d3b6a1dbddb5        arungupta/couchbase   "/entrypoint.sh /opt/"   About an hour ago   Up About an hour                        swarm-node-01/couchbasedockerswarm_db_2

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

02f94e2bbd3e arungupta/couchbase "/entrypoint.sh /opt/" 9 seconds ago Up 8 seconds swarm-master/couchbasedockerswarm_db_3

4c8c149f4e34 arungupta/couchbase "/entrypoint.sh /opt/" About an hour ago Up About an hour swarm-node-02/couchbasedockerswarm_db_1

d3b6a1dbddb5 arungupta/couchbase "/entrypoint.sh /opt/" About an hour ago Up About an hour swarm-node-01/couchbasedockerswarm_db_2

As mentioned earlier, scaling a Couchbase cluster is a two-step process. This is so because typically you’ll add multiple servers and then rebalance the cluster. However, in cases where you only need to add a single Couchbase node and then rebalance, the rebalance command can be used achieve that.

In our case, this is done as shown:

docker run -it arungupta/couchbase couchbase-cli rebalance --cluster=`docker-machine inspect swarm-node-01 | jq -r ".Driver.IPAddress"`:8091 --user Administrator --password password --server-add=`docker-machine inspect swarm-master | jq -r ".Driver.IPAddress"` --server-add-username=Administrator --server-add-password=password
SUCCESS: server-add 192.168.99.107:8091
INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
SUCCESS: rebalanced cluster

docker run -it arungupta/couchbase couchbase-cli rebalance --cluster=`docker-machine inspect swarm-node-01 | jq -r ".Driver.IPAddress"`:8091 --user Administrator --password password --server-add=`docker-machine inspect swarm-master | jq -r ".Driver.IPAddress"` --server-add-username=Administrator --server-add-password=password

SUCCESS: server-add 192.168.99.107:8091

SUCCESS: rebalanced cluster

The rebalanced cluster now looks like:

This blog showed how you can easily create and scale a Couchbase Cluster using Docker Swarm, Machine and Compose.

Enjoy!

Couchbase Weekly, May 11, 2016

May 11, 2016couchbasearungupta

Learn what’s latest in the Couchbase Community.
Couchbase Developer Community

Let us know if we missed anything at @couchbasedev, @couchbase or #Couchbase.

Highlights

Events

May 11: Apps World North America, San Francisco (Speaking)
May 11 – 12: GeeCON, Krakow (Speaking)
May 16 – 19: OSCON, Austin
May 18: Couchbase Day — Cromwell

News

Media Mentions

DZone: Document Database Granularity Level for the Relational Minded in 2 Minutes
Business Wire: Aviatrix Names Steven Mih as Chief Executive Officer
arstechnica: Power tools: Sorting through the crowded specialized database toolbox
Venture Beat: Redis launches modules to add extensibility to the open source database
Business Wire: Black Duck and North Bridge Release Results from 10th Annual Future of Open Source Survey
Business Wire: Centrify Honored as Bronze Stevie® Award Winner in 2016 American Business Awards℠
SDTimes: Why developers are sitting pretty for IoT
Fierce Enterprise Communications: Aviatrix beefs up security and support for OpenVPN in latest Cloud Native Networking release
DZone: Polyglot Persistence in NoSQL Space

Blogs

Monitoring Couchbase Server with SignalFx – Cihan Biyikoglu
Couchbase will be at Apps World North America 2016 — Nic Raboy
Couchbase with Windows and .NET – Part 2 — Matthew Groves
High Performance NOSQL Query Consistency with Couchbase 4.5 — Todd Greenstein
Faster Indexing and Query: Introducing Memory Optimized Global Secondary Indexes – PART II — Cihan Biyikoglu
Faster Indexing & Queries: Introducing Global Secondary Indexes (GSI) – PART I — Cihan Biyikoglu
Making most of your Arrays.. with Covering Array Indexes and more.. — Prasad Varakur
Docker container using Apache Mesos and Marathon — Arun Gupta
Introducing New Advanced Features to the Couchbase Shell – CBQ — Nic Raboy
Couchbase 4.5 Application Contest – Chance to win $500 Amazon gift card! — Arun Gupta
Couchbase with Windows and .NET – Part 1 – Setup — Matthew Groves
Introducing Couchbase Server 4.5 Beta — Don Pinto
Couchbase 4.5 Beta is Now Available – Faster, Simpler, More Secure — Arun Gupta
A CRUD REST API with Couchbase, RxJava and Ratpack — Laurent Doguin
Ratpack, Couchbase and RxJava — Laurent Doguin
Getting Started with Ratpack and Couchbase — Laurent Doguin
Setup Docker on Windows Server 2016 — Arun Gupta
Windows Server 2016 using VirtualBox for Docker Containers — Arun Gupta
Couchbase is at Mobile Web DevCon 2016 — Nic Raboy
Hello, Couchbase, it’s me, Matthew Groves! — Matthew Groves
Annoucing PHP SDK 2.1 beta — Sergey Avseyev

Whitepapers

Couchbase Server: An Architectural Overview

Upcoming Events

May 11: Apps World North America, San Francisco (Speaking)
May 11 – 12: GeeCON, Krakow (Speaking)
May 16 – 19: OSCON, Austin
May 23 – 26: Informatica World 2016, San Francisco
May 24 – 26: XLDB, Menlo Park
May 18: Couchbase Day — Cromwell
May 24: Couchbase Day — Houston
May 24: Couchbase Day — Las Vegas
May 26: Couchbase Day — Austin

Webinars

April 27: Future of Open Source Survey 2016

#Couchbase Server 4.5 Beta introduces new security features: x.509 certificates, role-based access control, and more… Click To Tweet Get professional advice from the #Couchbase services team on everything from #JSON modeling to using #SQLfor #NoSQL… Click To Tweet Can’t decide what db to use for your #mobile apps? Here’s 9 reasons to choose #Couchbase: ow.ly/4mXdgc Click To Tweet Enterprises are adopting modern databases like #Couchbase “citing grow flexibility as the top benefit” Learn more:… Click To Tweet Get lightning fast data access: discover #Couchbase XDCR for high availability & better disaster recovery:… Click To Tweet

Source: http://blog.couchbase.com/2016/may/couchbase-weekly-may-11-2016

Couchbase on Mesos using DC/OS and Amazon

May 10, 2016containers, couchbaseapache, containers, couchbase, dcos, docker, mesosarungupta

Couchbase Docker container can easily run on a variety of orchestration platforms:

Couchbase Docker Container
Couchbase on Docker Swarm
Couchbase on Kubernetes
Couchbase on OpenShift 3

Docker container using Apache Mesos and Marathon explained how to setup Mesos, Marathon, and run a simple Docker image. The setup was quite involving and a bit flaky.

It required to download and Install Mesos Master and Slave, ZooKeeper, Docker Engine, and Marathon. In some cases, the correct repo need to be added first. These components need to talk to each and so must be configured accordingly. Even if you get past that setup, how do you monitor the entire infrastructure as one entity?

Meet DC/OS – Datacenter Operating System. Its a distributed operating system using Apache Mesos as its kernel.

dcos-kernel

DC/OS can installed in a variety of ways:

Local using Vagrant
Cloud using Amazon/CloudFormation, Microsoft Azure and Packet
On-prem using CentOS or CoreOS

This blog will show how to setup a DC/OS cluster using CloudFormation templates on Amazon and run a Couchbase Docker container.

Launch DC/OS cluster

Launch DC/OS cluster:
dcos-couchbase-1

Take the defaults:

Give the template a name, select a previously create KeyPair, change the number of slaves:

Take the defaults:

Verify the configuration:

Click on “I acknowledge that …” and on Create to start the template creation.

CloudFormation Stack Status page comes up:

Make sure to choose the appropriate region.

After ~10-15 mins, the status changes:

Wait for the status to change from CREATE_IN_PROGRESS to CREATE_COMPLETE.

Download and Configure DC/OS CLI

DC/OS CLI can be used to manage your cluster nodes, install DC/OS packages, inspect the cluster state, and administer the DC/OS service subcommands.

Install DC/OS CLI on your local machine.

On your CloudFormation Stack Status page, select the created stack, Outputs tab, and copy the address of Mesos master.

Configure the DC/OS CLI to use this cluster:

./bin/dcos config set core.dcos_url https://couchbase-ElasticL-14LTDN783Y89T-1791879285.us-west-1.elb.amazonaws.com
[core.dcos_url]: changed from 'http://couchbase-ElasticL-UJ5K6BOZQX59-570103311.us-west-1.elb.amazonaws.com' to 'https://couchbase-ElasticL-14LTDN783Y89T-1791879285.us-west-1.elb.amazonaws.com'

./bin/dcos config set core.dcos_url https://couchbase-ElasticL-14LTDN783Y89T-1791879285.us-west-1.elb.amazonaws.com

[core.dcos_url]: changed from 'http://couchbase-ElasticL-UJ5K6BOZQX59-570103311.us-west-1.elb.amazonaws.com' to 'https://couchbase-ElasticL-14LTDN783Y89T-1791879285.us-west-1.elb.amazonaws.com'

Authenticate:

./bin/dcos auth login
Removed [core.dcos_acs_token]

Please go to the following link in your browser:

    https://couchbase-ElasticL-14LTDN783Y89T-1791879285.us-west-1.elb.amazonaws.com/login?redirect_uri=urn:ietf:wg:oauth:2.0:oob

Enter authentication token:

./bin/dcos auth login

Removed [core.dcos_acs_token]

Please go to the following link in your browser:

https://couchbase-ElasticL-14LTDN783Y89T-1791879285.us-west-1.elb.amazonaws.com/login?redirect_uri=urn:ietf:wg:oauth:2.0:oob

Enter authentication token:

Enter the URL in the browser, proceed to the unsafe URL:

Copy the token to your clipboard:

Paste the authentication token in the terminal window:

Enter authentication token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Ik9UQkVOakZFTWtWQ09VRTRPRVpGTlRNMFJrWXlRa015Tnprd1JrSkVRemRCTWpBM1FqYzVOZyJ9.eyJlbWFpbCI6ImFydW4uZ3VwdGFAZ21haWwuY29tIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImlzcyI6Imh0dHBzOi8vZGNvcy5hdXRoMC5jb20vIiwic3ViIjoiZ29vZ2xlLW9hdXRoMnwxMDExOTUyMTI0MDUxOTA0Njc1MTIiLCJhdWQiOiIzeUY1VE9TemRsSTQ1UTF4c3B4emVvR0JlOWZOeG05bSIsImV4cCI6MTQ2MzM1MjA1MSwiaWF0IjoxNDYyOTIwMDUxfQ.tFOy6GBPhnDbP0hI1ZU6jjlogHi4DX2bAGcdYsLywfCEMmv5OU9fUL88v2Kebjgd6gNAJoDF8QiO8zkQ3c4PGifXjiEoay8USjSWpnMbbezkH3AwIZM_-WNVVm0nfIBnCiY5wIgHkCpG_n4oBGdBbMgggtRPUrwPasNPtC_eYbQfwJeM3ByTUaTsarNe3FERbTpQbhKx2Lfc8Xp8AXuW8Up65SbOZr3Kp_jehiLgR-DL6llB4wZDopf8mjnq_5lpOYMtSaHPTlA33tdnhmHi6dPCPeRQEI1qBM6sWUWKs02kT1u42nOF9IYrKrfkzjb4LxwtZdsQt8Jb8CtJHkU0KA
[core.dcos_acs_token]: set
Login successful!

Enter authentication token: eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6Ik9UQkVOakZFTWtWQ09VRTRPRVpGTlRNMFJrWXlRa015Tnprd1JrSkVRemRCTWpBM1FqYzVOZyJ9.eyJlbWFpbCI6ImFydW4uZ3VwdGFAZ21haWwuY29tIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImlzcyI6Imh0dHBzOi8vZGNvcy5hdXRoMC5jb20vIiwic3ViIjoiZ29vZ2xlLW9hdXRoMnwxMDExOTUyMTI0MDUxOTA0Njc1MTIiLCJhdWQiOiIzeUY1VE9TemRsSTQ1UTF4c3B4emVvR0JlOWZOeG05bSIsImV4cCI6MTQ2MzM1MjA1MSwiaWF0IjoxNDYyOTIwMDUxfQ.tFOy6GBPhnDbP0hI1ZU6jjlogHi4DX2bAGcdYsLywfCEMmv5OU9fUL88v2Kebjgd6gNAJoDF8QiO8zkQ3c4PGifXjiEoay8USjSWpnMbbezkH3AwIZM_-WNVVm0nfIBnCiY5wIgHkCpG_n4oBGdBbMgggtRPUrwPasNPtC_eYbQfwJeM3ByTUaTsarNe3FERbTpQbhKx2Lfc8Xp8AXuW8Up65SbOZr3Kp_jehiLgR-DL6llB4wZDopf8mjnq_5lpOYMtSaHPTlA33tdnhmHi6dPCPeRQEI1qBM6sWUWKs02kT1u42nOF9IYrKrfkzjb4LxwtZdsQt8Jb8CtJHkU0KA

[core.dcos_acs_token]: set

Mesos and Marathon UI

Mesos UI is available using the address of the Mesos master:

Click on Services to see Marathon service already installed:

Click on marathon to see the list of tasks:

As expected for a freshly created cluster, no tasks have been assigned yet.

Click on Nodes to see the nodes:

Install and Configure Marathon Load Balancer

DC/OS slave nodes are not directly exposed on the Internet. An “external” load balancer can be configured to expose the tasks running on the slaves.

Marathon-lb, short for Marathon Load Balancer, is a load balancer available as a Mesos service. It is based on HAProxy that provides proxying and load balancing for TCP and HTTP based applications, with features such as SSL support, HTTP compression, health checking and more. Marathon-lb subscribes to Marathon’s event bus and updates the HAProxy configuration in real time.

dcos package install marathon-lb
We recommend a minimum of 0.5 CPUs and 256 MB of RAM available for the Marathon-LB DCOS Service.
Continue installing? [yes/no] yes
Installing Marathon app for package [marathon-lb] version [1.2.1]
Marathon-lb DCOS Service has been successfully installed!
See https://github.com/mesosphere/marathon-lb for documentation.

dcos package install marathon-lb

We recommend a minimum of 0.5 CPUs and 256 MB of RAM available for the Marathon-LB DCOS Service.

Continue installing? [yes/no] yes

Installing Marathon app for package [marathon-lb] version [1.2.1]

Marathon-lb DCOS Service has been successfully installed!

See https://github.com/mesosphere/marathon-lb for documentation.

Marathon service UI will show the LB task running:

AWS Load Balancer allows port 80 and 443 by default. We’ll run a Couchbase server that will be exposed at port 8091.

In CloudFormation Stack Status page, copy the value from Values column of PublicSlaveDnsAddress:

In AWS Console, select Load Balancers, add a new firewall rule to allow port 8091 on TCP:

Run Couchbase Server Docker container on DC/OS

Run Couchbase Server Docker container on DC/OS using the following configuration file:

{
  "id": "couchbase-mesos",
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "arungupta/couchbase",
      "network": "BRIDGE",
      "portMappings": [
        { "hostPort": 0, "containerPort": 8091, "servicePort": 8091 },
        { "hostPort": 0, "containerPort": 8092, "servicePort": 8092 },
        { "hostPort": 0, "containerPort": 8093, "servicePort": 8093 },
        { "hostPort": 0, "containerPort": 11210, "servicePort": 11210 }
      ],
      "forcePullImage":true
    }
  },
  "instances": 1,
  "cpus": 2.0,
  "mem": 2048,
  "labels":{
    "HAPROXY_GROUP":"external"
  }
}

{

"id": "couchbase-mesos",

"container": {

"type": "DOCKER",

"docker": {

"image": "arungupta/couchbase",

"network": "BRIDGE",

"portMappings": [

{ "hostPort": 0, "containerPort": 8091, "servicePort": 8091 },

{ "hostPort": 0, "containerPort": 8092, "servicePort": 8092 },

{ "hostPort": 0, "containerPort": 8093, "servicePort": 8093 },

{ "hostPort": 0, "containerPort": 11210, "servicePort": 11210 }

"forcePullImage":true

}

"instances": 1,

"cpus": 2.0,

"mem": 2048,

"labels":{

"HAPROXY_GROUP":"external"

}

This configuration file uses arungupta/couchbase image that configures the Couchbase Server using pre-defined Couchbase REST API. This image is Couchbase Docker Image.

cpus and mem attributes define the processing memory needed to run this task.

Give the command to run Couchbase in DC/OS cluster:

~/tools/dcos/dcos/bin/dcos marathon app add couchbase-mesos.json

~/tools/dcos/dcos/bin/dcos marathon app add couchbase-mesos.json

Use the previously copied for PublicSlaveDnsAddress and access the Couchbase Web Console at http://<URI>:8091. In our case, the URL is: http://couchbase-publicsl-vjzmwpa38k6d-429093455.us-west-1.elb.amazonaws.com:8091/index.html.

This shows up the login page as:

Enter the login credentials as Administrator and password:

Click on Sign In to see:

Learn more about Couchbase Web Console.

Marathon UI is updated to show all the running services:

Couchbase Docker image log can be seen in Log Viewer:

And the standard output view:

Mesos dashboard is updated to show the resources that are consumed:

Finally, the complete stack can be deleted from StackFormation template page:

Docker container using Apache Mesos and Marathon

May 5, 2016containerscontainers, docker, marathon, mesosarungupta

Apache Mesos is an open source cluster manager developed at UC Berkeley. It provides resource isolation and sharing across distributed applications.

The figure shows the main components of Mesos. Mesos consists of a master daemon that manages slave daemons running on each cluster node. Mesos frameworks are applications that run on Mesos and run tasks on these slaves. Slaves are either physical or virtual machines, typically from the same provider.

Mesos uses a two-level scheduling mechanism where resource offers are made to frameworks. The Mesos master node decides how many resources to offer each framework, while each framework determines the resources it accepts and what application to execute on those resources.

Marathon is a container orchestration platform running on Mesos. Multiple container formats are supported and Docker is certainly the most common one!

This blog will show how to setup Mesos, Marathon, and run a simple Docker image. This setup is only for the brave of heart. I’m always interested in looking under the hood and that’s what motivated this post. But a future post will show a more seamless install.

Let’s get started!

Configure CentOS VM

Download CentOS and configure the VM as shown:

Install Components

Install the different components required for this setup.

Configure Mesos repo:

[arun@arun-centos ~]$ sudo rpm -Uvh http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for arun:
Retrieving http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm
warning: /var/tmp/rpm-tmp.9ssCDQ: Header V4 RSA/SHA1 Signature, key ID e56151bf: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mesosphere-el-repo-7-1           ################################# [100%]

[arun@arun-centos ~]$ sudo rpm -Uvh http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm

We trust you have received the usual lecture from the local System

Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.

#2) Think before you type.

#3) With great power comes great responsibility.

[sudo] password for arun:

Retrieving http://repos.mesosphere.com/el/7/noarch/RPMS/mesosphere-el-repo-7-1.noarch.rpm

warning: /var/tmp/rpm-tmp.9ssCDQ: Header V4 RSA/SHA1 Signature, key ID e56151bf: NOKEY

Preparing... ################################# [100%]

Updating / installing...

1:mesosphere-el-repo-7-1 ################################# [100%]

Install Mesos and Marathon:

[arun@arun-centos ~]$ sudo yum -y install mesos marathon
Loaded plugins: fastestmirror, langpacks
base                                                     | 3.6 kB     00:00
extras                                                   | 3.4 kB     00:00
mesosphere                                               | 2.9 kB     00:00
mesosphere-noarch                                        | 2.9 kB     00:00
updates                                                  | 3.4 kB     00:00
(1/6): base/7/x86_64/group_gz                              | 155 kB   00:00
(2/6): extras/7/x86_64/primary_db                          | 117 kB   00:00
(3/6): mesosphere-noarch/primary_db                        | 2.5 kB   00:00
(4/6): mesosphere/x86_64/primary_db                        |  27 kB   00:00
(5/6): base/7/x86_64/primary_db                            | 5.3 MB   00:01
(6/6): updates/7/x86_64/primary_db                         | 4.1 MB   00:02
Determining fastest mirrors
 * base: mirror.supremebytes.com
 * extras: centos.den.host-engine.com
 * updates: mirror.hmc.edu
Resolving Dependencies
--> Running transaction check
---> Package marathon.x86_64 0:1.1.1-1.0.472.el7 will be installed
---> Package mesos.x86_64 0:0.28.1-2.0.20.centos701406 will be installed
--> Processing Dependency: subversion for package: mesos-0.28.1-2.0.20.centos701406.x86_64
--> Running transaction check
---> Package subversion.x86_64 0:1.7.14-10.el7 will be installed
--> Processing Dependency: subversion-libs(x86-64) = 1.7.14-10.el7 for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64
--> Running transaction check
---> Package apr.x86_64 0:1.4.8-3.el7 will be installed
---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed
---> Package subversion-libs.x86_64 0:1.7.14-10.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package            Arch      Version                       Repository     Size
================================================================================
Installing:
 marathon           x86_64    1.1.1-1.0.472.el7             mesosphere     64 M
 mesos              x86_64    0.28.1-2.0.20.centos701406    mesosphere     30 M
Installing for dependencies:
 apr                x86_64    1.4.8-3.el7                   base          103 k
 apr-util           x86_64    1.5.2-6.el7                   base           92 k
 subversion         x86_64    1.7.14-10.el7                 base          1.0 M
 subversion-libs    x86_64    1.7.14-10.el7                 base          921 k

Transaction Summary
================================================================================
Install  2 Packages (+4 Dependent packages)

Total download size: 96 M
Installed size: 168 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/apr-1.4.8-3.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for apr-1.4.8-3.el7.x86_64.rpm is not installed
(1/6): apr-1.4.8-3.el7.x86_64.rpm                          | 103 kB   00:00
(2/6): apr-util-1.5.2-6.el7.x86_64.rpm                     |  92 kB   00:00
(3/6): subversion-1.7.14-10.el7.x86_64.rpm                 | 1.0 MB   00:00
(4/6): subversion-libs-1.7.14-10.el7.x86_64.rpm            | 921 kB   00:00
warning: /var/cache/yum/x86_64/7/mesosphere/packages/mesos-0.28.1-2.0.20.centos701406.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e56151bf: NOKEY
Public key for mesos-0.28.1-2.0.20.centos701406.x86_64.rpm is not installed
(5/6): mesos-0.28.1-2.0.20.centos701406.x86_64.rpm         |  30 MB   00:29
(6/6): marathon-1.1.1-1.0.472.el7.x86_64.rpm               |  64 MB   00:36
--------------------------------------------------------------------------------
Total                                              2.6 MB/s |  96 MB  00:36
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-2.1511.el7.centos.2.10.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere
Importing GPG key 0xE56151BF:
 Userid     : "Mesosphere Archive Automatic Signing Key <support@mesosphere.io>"
 Fingerprint: 8102 6d00 04c4 4cf7 ef55 adf8 df7d 54cb e561 51bf
 Package    : mesosphere-el-repo-7-1.noarch (installed)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : apr-1.4.8-3.el7.x86_64                                       1/6
  Installing : apr-util-1.5.2-6.el7.x86_64                                  2/6
  Installing : subversion-libs-1.7.14-10.el7.x86_64                         3/6
  Installing : subversion-1.7.14-10.el7.x86_64                              4/6
  Installing : mesos-0.28.1-2.0.20.centos701406.x86_64                      5/6
Created symlink from /etc/systemd/system/multi-user.target.wants/mesos-master.service to /usr/lib/systemd/system/mesos-master.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/mesos-slave.service to /usr/lib/systemd/system/mesos-slave.service.
  Installing : marathon-1.1.1-1.0.472.el7.x86_64                            6/6
Created symlink from /etc/systemd/system/multi-user.target.wants/marathon.service to /usr/lib/systemd/system/marathon.service.
  Verifying  : apr-1.4.8-3.el7.x86_64                                       1/6
  Verifying  : subversion-1.7.14-10.el7.x86_64                              2/6
  Verifying  : apr-util-1.5.2-6.el7.x86_64                                  3/6
  Verifying  : mesos-0.28.1-2.0.20.centos701406.x86_64                      4/6
  Verifying  : marathon-1.1.1-1.0.472.el7.x86_64                            5/6
  Verifying  : subversion-libs-1.7.14-10.el7.x86_64                         6/6

Installed:
  marathon.x86_64 0:1.1.1-1.0.472.el7 mesos.x86_64 0:0.28.1-2.0.20.centos701406

Dependency Installed:
  apr.x86_64 0:1.4.8-3.el7             apr-util.x86_64 0:1.5.2-6.el7
  subversion.x86_64 0:1.7.14-10.el7    subversion-libs.x86_64 0:1.7.14-10.el7

Complete!

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

[arun@arun-centos ~]$ sudo yum -y install mesos marathon

Loaded plugins: fastestmirror, langpacks

base | 3.6 kB 00:00

extras | 3.4 kB 00:00

mesosphere | 2.9 kB 00:00

mesosphere-noarch | 2.9 kB 00:00

updates | 3.4 kB 00:00

(1/6): base/7/x86_64/group_gz | 155 kB 00:00

(2/6): extras/7/x86_64/primary_db | 117 kB 00:00

(3/6): mesosphere-noarch/primary_db | 2.5 kB 00:00

(4/6): mesosphere/x86_64/primary_db | 27 kB 00:00

(5/6): base/7/x86_64/primary_db | 5.3 MB 00:01

(6/6): updates/7/x86_64/primary_db | 4.1 MB 00:02

Determining fastest mirrors

* base: mirror.supremebytes.com

* extras: centos.den.host-engine.com

* updates: mirror.hmc.edu

Resolving Dependencies

--> Running transaction check

---> Package marathon.x86_64 0:1.1.1-1.0.472.el7 will be installed

---> Package mesos.x86_64 0:0.28.1-2.0.20.centos701406 will be installed

--> Processing Dependency: subversion for package: mesos-0.28.1-2.0.20.centos701406.x86_64

--> Running transaction check

---> Package subversion.x86_64 0:1.7.14-10.el7 will be installed

--> Processing Dependency: subversion-libs(x86-64) = 1.7.14-10.el7 for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_wc-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_subr-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_repos-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_ra_svn-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_ra_neon-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_ra_local-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_ra-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_fs_util-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_fs_fs-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_fs_base-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_fs-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_diff-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_delta-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libsvn_client-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libaprutil-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Processing Dependency: libapr-1.so.0()(64bit) for package: subversion-1.7.14-10.el7.x86_64

--> Running transaction check

---> Package apr.x86_64 0:1.4.8-3.el7 will be installed

---> Package apr-util.x86_64 0:1.5.2-6.el7 will be installed

---> Package subversion-libs.x86_64 0:1.7.14-10.el7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

Package Arch Version Repository Size

================================================================================

Installing:

marathon x86_64 1.1.1-1.0.472.el7 mesosphere 64 M

mesos x86_64 0.28.1-2.0.20.centos701406 mesosphere 30 M

Installing for dependencies:

apr x86_64 1.4.8-3.el7 base 103 k

apr-util x86_64 1.5.2-6.el7 base 92 k

subversion x86_64 1.7.14-10.el7 base 1.0 M

subversion-libs x86_64 1.7.14-10.el7 base 921 k

Transaction Summary

================================================================================

Install 2 Packages (+4 Dependent packages)

Total download size: 96 M

Installed size: 168 M

Downloading packages:

warning: /var/cache/yum/x86_64/7/base/packages/apr-1.4.8-3.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY

Public key for apr-1.4.8-3.el7.x86_64.rpm is not installed

(1/6): apr-1.4.8-3.el7.x86_64.rpm | 103 kB 00:00

(2/6): apr-util-1.5.2-6.el7.x86_64.rpm | 92 kB 00:00

(3/6): subversion-1.7.14-10.el7.x86_64.rpm | 1.0 MB 00:00

(4/6): subversion-libs-1.7.14-10.el7.x86_64.rpm | 921 kB 00:00

warning: /var/cache/yum/x86_64/7/mesosphere/packages/mesos-0.28.1-2.0.20.centos701406.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e56151bf: NOKEY

Public key for mesos-0.28.1-2.0.20.centos701406.x86_64.rpm is not installed

(5/6): mesos-0.28.1-2.0.20.centos701406.x86_64.rpm | 30 MB 00:29

(6/6): marathon-1.1.1-1.0.472.el7.x86_64.rpm | 64 MB 00:36

--------------------------------------------------------------------------------

Total 2.6 MB/s | 96 MB 00:36

Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

Importing GPG key 0xF4A80EB5:

Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"

Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5

Package : centos-release-7-2.1511.el7.centos.2.10.x86_64 (@anaconda)

From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere

Importing GPG key 0xE56151BF:

Userid : "Mesosphere Archive Automatic Signing Key <support@mesosphere.io>"

Fingerprint: 8102 6d00 04c4 4cf7 ef55 adf8 df7d 54cb e561 51bf

Package : mesosphere-el-repo-7-1.noarch (installed)

From : /etc/pki/rpm-gpg/RPM-GPG-KEY-mesosphere

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : apr-1.4.8-3.el7.x86_64 1/6

Installing : apr-util-1.5.2-6.el7.x86_64 2/6

Installing : subversion-libs-1.7.14-10.el7.x86_64 3/6

Installing : subversion-1.7.14-10.el7.x86_64 4/6

Installing : mesos-0.28.1-2.0.20.centos701406.x86_64 5/6

Created symlink from /etc/systemd/system/multi-user.target.wants/mesos-master.service to /usr/lib/systemd/system/mesos-master.service.

Created symlink from /etc/systemd/system/multi-user.target.wants/mesos-slave.service to /usr/lib/systemd/system/mesos-slave.service.

Installing : marathon-1.1.1-1.0.472.el7.x86_64 6/6

Created symlink from /etc/systemd/system/multi-user.target.wants/marathon.service to /usr/lib/systemd/system/marathon.service.

Verifying : apr-1.4.8-3.el7.x86_64 1/6

Verifying : subversion-1.7.14-10.el7.x86_64 2/6

Verifying : apr-util-1.5.2-6.el7.x86_64 3/6

Verifying : mesos-0.28.1-2.0.20.centos701406.x86_64 4/6

Verifying : marathon-1.1.1-1.0.472.el7.x86_64 5/6

Verifying : subversion-libs-1.7.14-10.el7.x86_64 6/6

Installed:

marathon.x86_64 0:1.1.1-1.0.472.el7 mesos.x86_64 0:0.28.1-2.0.20.centos701406

Dependency Installed:

apr.x86_64 0:1.4.8-3.el7 apr-util.x86_64 0:1.5.2-6.el7

subversion.x86_64 0:1.7.14-10.el7 subversion-libs.x86_64 0:1.7.14-10.el7

Complete!

Install ZooKeeper:

[arun@arun-centos ~]$ sudo yum -y install mesosphere-zookeeper
[sudo] password for arun:
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.supremebytes.com
 * extras: centos.den.host-engine.com
 * updates: mirror.hmc.edu
Resolving Dependencies
--> Running transaction check
---> Package mesosphere-zookeeper.x86_64 0:3.4.6-0.1.20141204175332.centos7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================
 Package                  Arch       Version                                 Repository      Size
==================================================================================================
Installing:
 mesosphere-zookeeper     x86_64     3.4.6-0.1.20141204175332.centos7        mesosphere     2.8 M

Transaction Summary
==================================================================================================
Install  1 Package

Total download size: 2.8 M
Installed size: 3.5 M
Downloading packages:
mesosphere-zookeeper-3.4.6-0.1.20141204175332.centos7.x86_64.rpm           | 2.8 MB  00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : mesosphere-zookeeper-3.4.6-0.1.20141204175332.centos7.x86_64                   1/1
Created symlink from /etc/systemd/system/multi-user.target.wants/zookeeper.service to /usr/lib/systemd/system/zookeeper.service.
  Verifying  : mesosphere-zookeeper-3.4.6-0.1.20141204175332.centos7.x86_64                   1/1

Installed:
  mesosphere-zookeeper.x86_64 0:3.4.6-0.1.20141204175332.centos7

Complete!

[arun@arun-centos ~]$ sudo yum -y install mesosphere-zookeeper

[sudo] password for arun:

Loaded plugins: fastestmirror, langpacks

Loading mirror speeds from cached hostfile

* base: mirror.supremebytes.com

* extras: centos.den.host-engine.com

* updates: mirror.hmc.edu

Resolving Dependencies

--> Running transaction check

---> Package mesosphere-zookeeper.x86_64 0:3.4.6-0.1.20141204175332.centos7 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

==================================================================================================

Package Arch Version Repository Size

==================================================================================================

Installing:

mesosphere-zookeeper x86_64 3.4.6-0.1.20141204175332.centos7 mesosphere 2.8 M

Transaction Summary

==================================================================================================

Install 1 Package

Total download size: 2.8 M

Installed size: 3.5 M

Downloading packages:

mesosphere-zookeeper-3.4.6-0.1.20141204175332.centos7.x86_64.rpm | 2.8 MB 00:00:01

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : mesosphere-zookeeper-3.4.6-0.1.20141204175332.centos7.x86_64 1/1

Created symlink from /etc/systemd/system/multi-user.target.wants/zookeeper.service to /usr/lib/systemd/system/zookeeper.service.

Verifying : mesosphere-zookeeper-3.4.6-0.1.20141204175332.centos7.x86_64 1/1

Installed:

mesosphere-zookeeper.x86_64 0:3.4.6-0.1.20141204175332.centos7

Complete!

Add Docker repo:

[arun@arun-centos ~]$ sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
EOF
[sudo] password for arun:
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

[arun@arun-centos ~]$ sudo tee /etc/yum.repos.d/docker.repo <<-'EOF'

[dockerrepo]

name=Docker Repository

baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/

enabled=1

gpgcheck=1

gpgkey=https://yum.dockerproject.org/gpg

EOF

[sudo] password for arun:

[dockerrepo]

name=Docker Repository

baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/

enabled=1

gpgcheck=1

gpgkey=https://yum.dockerproject.org/gpg

Install Docker:

[arun@localhost ~]$ sudo yum -y install docker-engine
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.scalabledns.com
 * extras: linux.mirrors.es.net
 * updates: mirrors.adams.net
Resolving Dependencies
--> Running transaction check
---> Package docker-engine.x86_64 0:1.11.1-1.el7.centos will be installed
--> Processing Dependency: docker-engine-selinux >= 1.11.1-1.el7.centos for package: docker-engine-1.11.1-1.el7.centos.x86_64
--> Running transaction check
---> Package docker-engine-selinux.noarch 0:1.11.1-1.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package                       Arch           Version                      Repository          Size
====================================================================================================
Installing:
 docker-engine                 x86_64         1.11.1-1.el7.centos          dockerrepo          13 M
Installing for dependencies:
 docker-engine-selinux         noarch         1.11.1-1.el7.centos          dockerrepo          28 k

Transaction Summary
====================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 13 M
Installed size: 54 M
Downloading packages:
warning: /var/cache/yum/x86_64/7/dockerrepo/packages/docker-engine-selinux-1.11.1-1.el7.centos.noarch.rpm: Header V4 RSA/SHA512 Signature, key ID 2c52609d: NOKEY
Public key for docker-engine-selinux-1.11.1-1.el7.centos.noarch.rpm is not installed
(1/2): docker-engine-selinux-1.11.1-1.el7.centos.noarch.rpm                  |  28 kB  00:00:00
(2/2): docker-engine-1.11.1-1.el7.centos.x86_64.rpm                          |  13 MB  00:00:02
----------------------------------------------------------------------------------------------------
Total                                                               5.3 MB/s |  13 MB  00:00:02
Retrieving key from https://yum.dockerproject.org/gpg
Importing GPG key 0x2C52609D:
 Userid     : "Docker Release Tool (releasedocker) <docker@docker.com>"
 Fingerprint: 5811 8e89 f3a9 1289 7c07 0adb f762 2157 2c52 609d
 From       : https://yum.dockerproject.org/gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : docker-engine-selinux-1.11.1-1.el7.centos.noarch                                 1/2
restorecon:  lstat(/var/lib/docker) failed:  No such file or directory
warning: %post(docker-engine-selinux-1.11.1-1.el7.centos.noarch) scriptlet failed, exit status 255
Non-fatal POSTIN scriptlet failure in rpm package docker-engine-selinux-1.11.1-1.el7.centos.noarch
  Installing : docker-engine-1.11.1-1.el7.centos.x86_64                                         2/2
  Verifying  : docker-engine-1.11.1-1.el7.centos.x86_64                                         1/2
  Verifying  : docker-engine-selinux-1.11.1-1.el7.centos.noarch                                 2/2

Installed:
  docker-engine.x86_64 0:1.11.1-1.el7.centos

Dependency Installed:
  docker-engine-selinux.noarch 0:1.11.1-1.el7.centos

Complete!

[arun@localhost ~]$ sudo yum -y install docker-engine

Loaded plugins: fastestmirror, langpacks

Loading mirror speeds from cached hostfile

* base: mirror.scalabledns.com

* extras: linux.mirrors.es.net

* updates: mirrors.adams.net

Resolving Dependencies

--> Running transaction check

---> Package docker-engine.x86_64 0:1.11.1-1.el7.centos will be installed

--> Processing Dependency: docker-engine-selinux >= 1.11.1-1.el7.centos for package: docker-engine-1.11.1-1.el7.centos.x86_64

--> Running transaction check

---> Package docker-engine-selinux.noarch 0:1.11.1-1.el7.centos will be installed

--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================

Package Arch Version Repository Size

====================================================================================================

Installing:

docker-engine x86_64 1.11.1-1.el7.centos dockerrepo 13 M

Installing for dependencies:

docker-engine-selinux noarch 1.11.1-1.el7.centos dockerrepo 28 k

Transaction Summary

====================================================================================================

Install 1 Package (+1 Dependent package)

Total download size: 13 M

Installed size: 54 M

Downloading packages:

warning: /var/cache/yum/x86_64/7/dockerrepo/packages/docker-engine-selinux-1.11.1-1.el7.centos.noarch.rpm: Header V4 RSA/SHA512 Signature, key ID 2c52609d: NOKEY

Public key for docker-engine-selinux-1.11.1-1.el7.centos.noarch.rpm is not installed

(1/2): docker-engine-selinux-1.11.1-1.el7.centos.noarch.rpm | 28 kB 00:00:00

(2/2): docker-engine-1.11.1-1.el7.centos.x86_64.rpm | 13 MB 00:00:02

----------------------------------------------------------------------------------------------------

Total 5.3 MB/s | 13 MB 00:00:02

Retrieving key from https://yum.dockerproject.org/gpg

Importing GPG key 0x2C52609D:

Userid : "Docker Release Tool (releasedocker) <docker@docker.com>"

Fingerprint: 5811 8e89 f3a9 1289 7c07 0adb f762 2157 2c52 609d

From : https://yum.dockerproject.org/gpg

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : docker-engine-selinux-1.11.1-1.el7.centos.noarch 1/2

restorecon: lstat(/var/lib/docker) failed: No such file or directory

warning: %post(docker-engine-selinux-1.11.1-1.el7.centos.noarch) scriptlet failed, exit status 255

Non-fatal POSTIN scriptlet failure in rpm package docker-engine-selinux-1.11.1-1.el7.centos.noarch

Installing : docker-engine-1.11.1-1.el7.centos.x86_64 2/2

Verifying : docker-engine-1.11.1-1.el7.centos.x86_64 1/2

Verifying : docker-engine-selinux-1.11.1-1.el7.centos.noarch 2/2

Installed:

docker-engine.x86_64 0:1.11.1-1.el7.centos

Dependency Installed:

docker-engine-selinux.noarch 0:1.11.1-1.el7.centos

Complete!

Configure Hostname/IP address Mapping

Edit /etc/hosts and create hostname and IP address mapping. Find IP address using ifconfig and choose the network interface enabled during CentOS installation.

Start Services

Start all the services

Start Docker:

[arun@arun-centos ~]$ sudo service docker start
Redirecting to /bin/systemctl start  docker.service

[arun@arun-centos ~]$ sudo service docker start

Redirecting to /bin/systemctl start docker.service

Start ZooKeeper:

sudo systemctl start zookeeper

sudo systemctl start zookeeper

Start Mesos master:

[arun@arun-centos ~]$ sudo service mesos-master start
Redirecting to /bin/systemctl start  mesos-master.service

[arun@arun-centos ~]$ sudo service mesos-master start

Redirecting to /bin/systemctl start mesos-master.service

Configure mesos and docker containerizers:

sudo sh -c "echo 'docker,mesos' > /etc/mesos-slave/containerizers"

sudo sh -c "echo 'docker,mesos' > /etc/mesos-slave/containerizers"

Start Mesos slave:

[arun@arun-centos ~]$ sudo service mesos-slave start
Redirecting to /bin/systemctl start  mesos-slave.service

[arun@arun-centos ~]$ sudo service mesos-slave start

Redirecting to /bin/systemctl start mesos-slave.service

Start Marathon:

[arun@arun-centos ~]$ sudo service marathon start
Redirecting to /bin/systemctl start  marathon.service

[arun@arun-centos ~]$ sudo service marathon start

Redirecting to /bin/systemctl start marathon.service

Check for services:Mesos UI: http://127.0.0.1:5050
Marathon UI: http://127.0.0.1:8080
Logs: tail -f /var/log/messages

[arun@arun-centos log]$ ps aux | grep meso
root      4511  0.1  0.7 3974136 58968 ?       Ssl  19:54   0:04 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /opt/mesosphere/zookeeper/bin/../build/classes:/opt/mesosphere/zookeeper/bin/../build/lib/*.jar:/opt/mesosphere/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/mesosphere/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/mesosphere/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/opt/mesosphere/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/mesosphere/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/mesosphere/zookeeper/bin/../zookeeper-3.4.6.jar:/opt/mesosphere/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg
root     15786  0.0  0.2 754788 19596 ?        Ssl  20:43   0:00 /usr/sbin/mesos-slave --master=zk://localhost:2181/mesos --log_dir=/var/log/mesos
root     15792  0.0  0.0 107892   624 ?        S    20:43   0:00 logger -p user.info -t mesos-slave[15786]
root     15793  0.0  0.0 107892   700 ?        S    20:43   0:00 logger -p user.err -t mesos-slave[15786]
root     15837  1.9  3.0 2818056 239208 ?      Ssl  20:43   0:10 java -Djava.library.path=/usr/local/lib:/usr/lib:/usr/lib64 -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -Xmx512m -cp /usr/bin/marathon mesosphere.marathon.Main --zk zk://localhost:2181/marathon --master zk://localhost:2181/mesos
root     15838  0.0  0.2 829028 22908 ?        Ssl  20:43   0:00 /usr/sbin/mesos-master --zk=zk://localhost:2181/mesos --port=5050 --log_dir=/var/log/mesos --quorum=1 --work_dir=/var/lib/mesos
root     15863  0.0  0.0 107892   620 ?        S    20:43   0:00 logger -p user.info -t mesos-master[15838]
root     15864  0.0  0.0 107892   696 ?        S    20:43   0:00 logger -p user.err -t mesos-master[15838]
arun     16931  0.0  0.0 112644   952 pts/0    R+   20:52   0:00 grep --color=auto meso

[arun@arun-centos log]$ ps aux | grep meso

root 4511 0.1 0.7 3974136 58968 ? Ssl 19:54 0:04 java -Dzookeeper.log.dir=. -Dzookeeper.root.logger=INFO,CONSOLE -cp /opt/mesosphere/zookeeper/bin/../build/classes:/opt/mesosphere/zookeeper/bin/../build/lib/*.jar:/opt/mesosphere/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/mesosphere/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/mesosphere/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/opt/mesosphere/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/mesosphere/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/mesosphere/zookeeper/bin/../zookeeper-3.4.6.jar:/opt/mesosphere/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg

root 15786 0.0 0.2 754788 19596 ? Ssl 20:43 0:00 /usr/sbin/mesos-slave --master=zk://localhost:2181/mesos --log_dir=/var/log/mesos

root 15792 0.0 0.0 107892 624 ? S 20:43 0:00 logger -p user.info -t mesos-slave[15786]

root 15793 0.0 0.0 107892 700 ? S 20:43 0:00 logger -p user.err -t mesos-slave[15786]

root 15837 1.9 3.0 2818056 239208 ? Ssl 20:43 0:10 java -Djava.library.path=/usr/local/lib:/usr/lib:/usr/lib64 -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -Xmx512m -cp /usr/bin/marathon mesosphere.marathon.Main --zk zk://localhost:2181/marathon --master zk://localhost:2181/mesos

root 15838 0.0 0.2 829028 22908 ? Ssl 20:43 0:00 /usr/sbin/mesos-master --zk=zk://localhost:2181/mesos --port=5050 --log_dir=/var/log/mesos --quorum=1 --work_dir=/var/lib/mesos

root 15863 0.0 0.0 107892 620 ? S 20:43 0:00 logger -p user.info -t mesos-master[15838]

root 15864 0.0 0.0 107892 696 ? S 20:43 0:00 logger -p user.err -t mesos-master[15838]

arun 16931 0.0 0.0 112644 952 pts/0 R+ 20:52 0:00 grep --color=auto meso

Check Mesos master:

[arun@arun-centos ~]$ systemctl status mesos-master
● mesos-master.service - Mesos Master
   Loaded: loaded (/usr/lib/systemd/system/mesos-master.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-05-04 19:42:24 EDT; 32min ago
 Main PID: 8499 (mesos-master)
   Memory: 14.6M
   CGroup: /system.slice/mesos-master.service
           ├─8499 /usr/sbin/mesos-master --zk=zk://localhost:2181/mesos --port=5050 --log_dir=/va...
           ├─8513 logger -p user.info -t mesos-master[8499]
           └─8514 logger -p user.err -t mesos-master[8499]

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.564679  8527 master.cpp:3104] Proc...
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.565058  8527 master.hpp:177] Ad...os)
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.565093  8527 master.cpp:3589] Laun...
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.577255  8527 master.cpp:4763] Stat...
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.577287  8527 master.cpp:4811] F...000
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.577325  8527 master.cpp:6421] U...ED)
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.587143  8527 master.cpp:3918] Proc...
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.587198  8527 master.cpp:6487] Remo...
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.726095  8527 master.cpp:5324] S...006
May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.728080  8527 master.cpp:3641] P...006
Hint: Some lines were ellipsized, use -l to show in full.

[arun@arun-centos ~]$ systemctl status mesos-master

● mesos-master.service - Mesos Master

Loaded: loaded (/usr/lib/systemd/system/mesos-master.service; enabled; vendor preset: disabled)

Active: active (running) since Wed 2016-05-04 19:42:24 EDT; 32min ago

Main PID: 8499 (mesos-master)

Memory: 14.6M

CGroup: /system.slice/mesos-master.service

├─8499 /usr/sbin/mesos-master --zk=zk://localhost:2181/mesos --port=5050 --log_dir=/va...

├─8513 logger -p user.info -t mesos-master[8499]

└─8514 logger -p user.err -t mesos-master[8499]

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.564679 8527 master.cpp:3104] Proc...

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.565058 8527 master.hpp:177] Ad...os)

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.565093 8527 master.cpp:3589] Laun...

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.577255 8527 master.cpp:4763] Stat...

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.577287 8527 master.cpp:4811] F...000

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.577325 8527 master.cpp:6421] U...ED)

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.587143 8527 master.cpp:3918] Proc...

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.587198 8527 master.cpp:6487] Remo...

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.726095 8527 master.cpp:5324] S...006

May 04 20:14:41 arun-centos mesos-master[8514]: I0504 20:14:41.728080 8527 master.cpp:3641] P...006

Hint: Some lines were ellipsized, use -l to show in full.

Check Mesos slave:

[arun@arun-centos ~]$ systemctl status mesos-slave
● mesos-slave.service - Mesos Slave
   Loaded: loaded (/usr/lib/systemd/system/mesos-slave.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-05-04 20:00:43 EDT; 12min ago
 Main PID: 9864 (mesos-slave)
   Memory: 57.5M
   CGroup: /system.slice/mesos-slave.service
           ├─9864 /usr/sbin/mesos-slave --master=zk://localhost:2181/mesos --log_dir=/var/log/mes...
           ├─9876 logger -p user.info -t mesos-slave[9864]
           └─9877 logger -p user.err -t mesos-slave[9864]

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.283629  9887 status_update_manag...000
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289118  9887 slave.cpp:3990] Cle...000
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289422  9887 slave.cpp:4078] Cle...000
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289459  9887 gc.cpp:55] Scheduling ...
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289489  9887 gc.cpp:55] Scheduli...ure
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289499  9887 gc.cpp:55] Scheduling ...
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289511  9887 gc.cpp:55] Scheduli...ure
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289520  9887 gc.cpp:55] Scheduli...ure
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289525  9887 gc.cpp:55] Scheduli...ure
May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289533  9887 status_update_manag...000
Hint: Some lines were ellipsized, use -l to show in full.

[arun@arun-centos ~]$ systemctl status mesos-slave

● mesos-slave.service - Mesos Slave

Loaded: loaded (/usr/lib/systemd/system/mesos-slave.service; enabled; vendor preset: disabled)

Active: active (running) since Wed 2016-05-04 20:00:43 EDT; 12min ago

Main PID: 9864 (mesos-slave)

Memory: 57.5M

CGroup: /system.slice/mesos-slave.service

├─9864 /usr/sbin/mesos-slave --master=zk://localhost:2181/mesos --log_dir=/var/log/mes...

├─9876 logger -p user.info -t mesos-slave[9864]

└─9877 logger -p user.err -t mesos-slave[9864]

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.283629 9887 status_update_manag...000

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289118 9887 slave.cpp:3990] Cle...000

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289422 9887 slave.cpp:4078] Cle...000

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289459 9887 gc.cpp:55] Scheduling ...

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289489 9887 gc.cpp:55] Scheduli...ure

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289499 9887 gc.cpp:55] Scheduling ...

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289511 9887 gc.cpp:55] Scheduli...ure

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289520 9887 gc.cpp:55] Scheduli...ure

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289525 9887 gc.cpp:55] Scheduli...ure

May 04 20:13:26 arun-centos mesos-slave[9877]: I0504 20:13:26.289533 9887 status_update_manag...000

Hint: Some lines were ellipsized, use -l to show in full.

Check ZooKeeper:

[arun@arun-centos log]$ /opt/mesosphere/zookeeper/bin/zkCli.sh
Connecting to localhost:2181
log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /mesos
[json.info_0000000001, log_replicas]

[arun@arun-centos log]$ /opt/mesosphere/zookeeper/bin/zkCli.sh

Connecting to localhost:2181

log4j:WARN No appenders could be found for logger (org.apache.zookeeper.ZooKeeper).

log4j:WARN Please initialize the log4j system properly.

log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Welcome to ZooKeeper!

JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0] ls /mesos

[json.info_0000000001, log_replicas]

Deploy Docker application to Mesos

A simple Docker-based application is defined using the configuration file. Marathon runs on port 8080 and so the updated configuration file looks like:

{
  "id": "basic-3",
  "cmd": "python3 -m http.server $PORT0",
  "cpus": 0.5,
  "mem": 32.0,
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "python:3",
      "forcePullImage": true,
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 0, "hostPort": 0 }
      ]
    }
  }
}

{

"id": "basic-3",

"cmd": "python3 -m http.server $PORT0",

"cpus": 0.5,

"mem": 32.0,

"container": {

"type": "DOCKER",

"docker": {

"image": "python:3",

"forcePullImage": true,

"network": "BRIDGE",

"portMappings": [

{ "containerPort": 0, "hostPort": 0 }

]

}

Deploy the application as:

curl -X POST http://127.0.0.1:8080/v2/apps -d @app.json -H "Content-type: application/json"

curl -X POST http://127.0.0.1:8080/v2/apps -d @app.json -H "Content-type: application/json"

The application will take some time in order to download the image and then run the container. This setup is slightly sensitive and multiple runs of the application showed that the Docker image was not successfully downloaded all the times. In that case, the Docker image was manually downloaded using docker pull python:3 and then the application could be successfully deployed.

In our case, master and slave are running on the same machine, and so the list of Docker images and running containers can be easily seen:

[arun@localhost ~]$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
python              3                   67b5207e5d3b        8 days ago          671.1 MB
[arun@localhost ~]$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES
2ee42b120537        python:3            "/bin/sh -c 'python3 "   About an hour ago   Up About an hour    0.0.0.0:31669->31669/tcp   mesos-d03f6e61-063f-4cae-9c1b-3fc6d8384b3a-S0.b99d82f6-9bc1-4a6f-ae3b-53b7f5d06b95

[arun@localhost ~]$ sudo docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

python 3 67b5207e5d3b 8 days ago 671.1 MB

[arun@localhost ~]$ sudo docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

2ee42b120537 python:3 "/bin/sh -c 'python3 " About an hour ago Up About an hour 0.0.0.0:31669->31669/tcp mesos-d03f6e61-063f-4cae-9c1b-3fc6d8384b3a-S0.b99d82f6-9bc1-4a6f-ae3b-53b7f5d06b95

The application is available at port 31669 and can be seen at http://127.0.0.1:31669 as:

Mesos UI (http://127.0.0.1:5050) shows:

Marathon UI (http://127.0.0.1:8080) shows:
mesos-ui-app-output

As you can see, this is quite an involving setup. A future blog post will show how to use DC/OS and set this up more seamlessly.

Couchbase 4.5 Application Contest – Chance to win $500 Amazon gift card!

May 4, 2016couchbasecouchbasearungupta

Couchbase 4.5 Beta was announced yesterday and it can make you $500 richer!

This blog announces a contest asking developers to build an application using Couchbase 4.5 Beta. Each qualified entry wins a #Couchbase swag worth $25. One lucky winner will be awarded a $500 amazon gift card.

This release has several features such as integrated full text search, sub-document API, integrated query workbench, query monitoring and much more. Do you care about performance in a NoSQL database, want to make it more secure, or like simplified/advanced data access? There is something in it for every body.

Here are some ideas for submission:

Pick a language SDK and create a CRUD application
Create a full-stack application using Couchbase as one of the components
Update Couchbase Maven Plugin to work with the latest release
Migrate your favorite NoSQL application to work with Couchbase
Eclipse or IntelliJ plugin for Couchbase

And of course, we love to see your creative ideas!

What are we looking for?

Creativity and originality of the application
How many Couchbase features are used in the application?
Are you running Couchbase as a Docker container? In Kubernetes? In Mesos + Marathon?
Have you used Couchbase NetBeans plugin for development?

Contest timeline: May 5th, 12:01 EDT – Jun 30, 2016, 11:59 PM EDT

Where to submit? Send your submissions to devadvocates at couchbase dot com.

Official contest rules provide more details on the rules.

Good luck!

Source: http://blog.couchbase.com/2016/may/couchbase-4.5-application-contest-chance-to-win-usd500-amazon-gift-card

Couchbase 4.5 Beta is Now Available – Faster, Simpler, More Secure

May 3, 2016couchbasecouchbasearungupta

Couchbase 4.5 Beta is now available, download!

This release has three main themes:

Here is the list of features that you can now play with:

Faster
- Sub Document API
- Memory-Optimized Global Indexes
- Array Indexing
- Faster Read-Your-Own-Write Consistency with N1QL Queries
- Improved Index Write Performance with Circular Writes
- Flexible Commutative JOINs in N1QL
Simpler
- Integrated Query Workbench
- Query Monitoring
- Command-line shell for N1QL
- Integrated Full Text Search and new Search Service (Developer Preview)
- Enterprise Backup and Restore
Secure
- X.509 Certificate-based Security
- Role Based Access Control for Administrators

More details in What’s New in Couchbase 4.5 Beta!

Read more details in Introducing Couchbase Server 4.5 Beta.

Some references for you …

Couchbase 4.5 Beta Landing Page
Couchbase 4.5 Downloads
Couchbase 4.5 Beta Docs
Couchbase 4.5 Beta Release Notes
Couchbase 4.5 Beta YouTube Playlist

Enjoy!

Source: blog.couchbase.com/2016/may/couchbase-4.5-beta-faster-simpler-more-secure