Tag Archives: mesos

Couchbase on Mesos using DC/OS and Amazon

Couchbase Docker container can easily run on a variety of orchestration platforms:

Docker container using Apache Mesos and Marathon explained how to setup Mesos, Marathon, and run a simple Docker image. The setup was quite involving and a bit flaky.

It required to download and Install Mesos Master and Slave, ZooKeeper, Docker Engine, and Marathon. In some cases, the correct repo need to be added first. These components need to talk to each and so must be configured accordingly. Even if you get past that setup, how do you monitor the entire infrastructure as one entity?

Meet DC/OS – Datacenter Operating System. Its a distributed operating system using Apache Mesos as its kernel.

dcos-kernel

DC/OS can installed in a variety of ways:

  • Local using Vagrant
  • Cloud using Amazon/CloudFormation, Microsoft Azure and Packet
  • On-prem using CentOS or CoreOS

This blog will show how to setup a DC/OS cluster using CloudFormation templates on Amazon and run a Couchbase Docker container.

Launch DC/OS cluster

Launch DC/OS cluster:
dcos-couchbase-1

Take the defaults:
dcos-couchbase-2

Give the template a name, select a previously create KeyPair, change the number of slaves:
dcos-couchbase-3

Take the defaults:
dcos-couchbase-4

Verify the configuration:
dcos-couchbase-5

Click on “I acknowledge that …” and on Create to start the template creation.

CloudFormation Stack Status  page comes up:
dcos-couchbase-6

Make sure to choose the appropriate region.

After ~10-15 mins, the status changes:
dcos-couchbase-7

Wait for the status to change from CREATE_IN_PROGRESS to CREATE_COMPLETE.

Download and Configure DC/OS CLI

DC/OS CLI can be used to manage your cluster nodes, install DC/OS packages, inspect the cluster state, and administer the DC/OS service subcommands.

Install DC/OS CLI on your local machine.

On your CloudFormation Stack Status  page, select the created stack, Outputs tab, and copy the address of Mesos master.
dcos-couchbase-8

Configure the DC/OS CLI to use this cluster:

Authenticate:

Enter the URL in the browser, proceed to the unsafe URL:
dcos-couchbase-9

Copy the token to your clipboard:
dcos-couchbase-10

Paste the authentication token in the terminal window:

Mesos and Marathon UI

Mesos UI is available using the address of the Mesos master:
dcos-couchbase-11

Click on Services to see Marathon service already installed:
dcos-couchbase-12

Click on marathon to see the list of tasks:
dcos-couchbase-13

As expected for a freshly created cluster, no tasks have been assigned yet.

Click on Nodes to see the nodes:
dcos-couchbase-14

Install and Configure Marathon Load Balancer

DC/OS slave nodes are not directly exposed on the Internet. An “external” load balancer can be configured to expose the tasks running on the slaves.

Marathon-lb, short for Marathon Load Balancer, is a load balancer available as a Mesos service. It is based on HAProxy that provides proxying and load balancing for TCP and HTTP based applications, with features such as SSL support, HTTP compression, health checking and more. Marathon-lb subscribes to Marathon’s event bus and updates the HAProxy configuration in real time.

Marathon service UI will show the LB task running:
dcos-couchbase-15

AWS Load Balancer allows port 80 and 443 by default. We’ll run a Couchbase server that will be exposed at port 8091.

In CloudFormation Stack Status  page, copy the value from Values column of PublicSlaveDnsAddress:
dcos-couchbase-16

In AWS Console, select Load Balancers, add a new firewall rule to allow port 8091 on TCP:
dcos-couchbase-17

Run Couchbase Server Docker container on DC/OS

Run Couchbase Server Docker container on DC/OS using the following configuration file:

This configuration file uses arungupta/couchbase image that configures the Couchbase Server using pre-defined Couchbase REST API. This image is Couchbase Docker Image.

cpus and mem attributes define the processing memory needed to run this task.

Give the command to run Couchbase in DC/OS cluster:

Use the previously copied for PublicSlaveDnsAddress and access the Couchbase Web Console at http://<URI>:8091. In our case, the URL is: http://couchbase-publicsl-vjzmwpa38k6d-429093455.us-west-1.elb.amazonaws.com:8091/index.html.

This shows up the login page as:
dcos-couchbase-18

Enter the login credentials as Administrator and password:dcos-couchbase-19

Click on Sign In to see:
dcos-couchbase-20

Learn more about Couchbase Web Console.

Marathon UI is updated to show all the running services:
dcos-couchbase-21

Couchbase Docker image log can be seen in Log Viewer:
dcos-couchbase-22

And the standard output view:
dcos-couchbase-23

Mesos dashboard is updated to show the resources that are consumed:
dcos-couchbase-24

Finally, the complete stack can be deleted from StackFormation template page:
dcos-couchbase-25

Further reading:

  • Latest DC/OS docs
  • DC/OS Installation Guide
  • Get Started with DC/OS
  • Manage your DC/OS Cluster
  • Service discovery and load balancing with DC/OS
  • DC/OS Slack Channel
  • Get Started with Couchbase

Enjoy!

Now, you’ve seen Couchbase on Docker Swarm, Couchbase on Kubernetes, Couchbase on OpenShift 3. This blog showed how to run a Couchbase Docker image on Mesos and DC/OS.

Where else would you like Couchbase container to run?

Source: http://blog.couchbase.com/2016/may/couchbase-mesos-dcos-amazon

Docker container using Apache Mesos and Marathon

apache-mesos-logo apache-mesos-marathon-logo Docker Logo

Apache Mesos is an open source cluster manager developed at UC Berkeley. It provides resource isolation and sharing across distributed applications.

The figure shows the main components of Mesos. Mesos consists of a master daemon that manages slave daemons running on each cluster node. Mesos frameworks are applications that run on Mesos and run tasks on these slaves. Slaves are either physical or virtual machines, typically from the same provider.

mesos-architecture

Mesos uses a two-level scheduling mechanism where resource offers are made to frameworks. The Mesos master node decides how many resources to offer each framework, while each framework determines the resources it accepts and what application to execute on those resources.

Marathon is a container orchestration platform running on Mesos. Multiple container formats are supported and Docker is certainly the most common one!

This blog will show how to setup Mesos, Marathon, and run a simple Docker image. This setup is only for the brave of heart. I’m always interested in looking under the hood and that’s what motivated this post. But a future post will show a more seamless install.

Let’s get started!

Configure CentOS VM

Download CentOS and configure the VM as shown:

centos-7.1-install

Install Components

Install the different components required for this setup.

  1. Configure Mesos repo:
  2. Install Mesos and Marathon:
  3. Install ZooKeeper:
  4. Add Docker repo:
  5. Install Docker:

Configure Hostname/IP address Mapping

Edit /etc/hosts and create hostname and IP address mapping. Find IP address using ifconfig and choose the network interface enabled during CentOS installation.

Start Services

Start all the services

  1. Start Docker:
  2. Start ZooKeeper:
  3. Start Mesos master:
  4. Configure mesos and docker containerizers:
  5. Start Mesos slave:
  6. Start Marathon:
  7. Check for services:Mesos UI: http://127.0.0.1:5050
    Marathon UI: http://127.0.0.1:8080
    Logs: tail -f /var/log/messages
  8. Check Mesos master:
  9. Check Mesos slave:
  10. Check ZooKeeper:

Deploy Docker application to Mesos

A simple Docker-based application is defined using the configuration file. Marathon runs on port 8080 and so the updated configuration file looks like:

Deploy the application as:

The application will take some time in order to download the image and then run the container. This setup is slightly sensitive and multiple runs of the application showed that the Docker image was not successfully downloaded all the times. In that case, the Docker image was manually downloaded using docker pull python:3 and then the application could be successfully deployed.

In our case, master and slave are running on the same machine, and so the list of Docker images and running containers can be easily seen:

The application is available at port 31669 and can be seen at http://127.0.0.1:31669 as:

mesos-marathon-app-output

Mesos UI (http://127.0.0.1:5050) shows:
marathon-ui-app-output

Marathon UI (http://127.0.0.1:8080) shows:
mesos-ui-app-output

As you can see, this is quite an involving setup. A future blog post will show how to use DC/OS and set this up more seamlessly.

Further reading …

  • Mesos Marathon
  • Ports in Mesos
  • Mesos CLI
  • Mesos – Under the Hood

Mesos slack channel is awesome! Particularly I learned a lot about Mesos from @jgarcia.mesosphere, @akaplan.mesosphere, @harpreet.mesosphere, @graham.mesosphere. Thanks guys, keep up on engaging with the community!

Enjoy!

Source: http://blog.couchbase.com/2016/may/docker-apache-mesos-marathon