Monthly Archives: November 2015

Docker Swarm Cluster using Consul

containers,

Docker Swarm is native clustering for Docker. It allows you create and access to a pool of Docker hosts using the full suite of Docker tools. Because Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts.

Docker Swarm Logo Consul Logo

Docker Swarm has a Manager, a pre-defined Docker Host, and is a single point for all administration. Swarm manager orchestrates and schedules containers on the entire cluster, and can be configured in High Availability. The containers are deployed on Nodes that are additional Docker Hosts.

Swarm talks to a hosted Discovery Service that maintains a list of IPs in your cluster. For development, its easy to use the default discovery service hosted on Docker Hub. Complete instructions for that are available in Install and Create Docker Swarm. This blog will show how to setup Docker Swarm Cluster using Consul.

Lets get started!

Create Consul Discovery Service

  1. Create a Machine that will host discovery service:
  2. Connect to this Machine:
  3. Run Consul service using the following Compose file:
    This file is also available at github.com/arun-gupta/docker-images/tree/master/consul.

    The service is started as:

    Started container can be verified as:

Create Docker Swarm Cluster using Consul

Swarm is fully integrated with Machine, and so is the easiest way to get started.

  1. Create a Swarm Master using the Consul discovery service:
    Three options to look here:

    • --swarm-discovery defines address of the discovery service
    • --cluster-advertise advertise the machine on the network
    • --cluster-store designate a distributed k/v storage backend for the cluster

    In addition, --swarm configures the Machine with Swarm, --swarm-master configures the created Machine to be Swarm master.

  2. Connect to this newly created master and find some information about it:

    This will show the output as:

  3. Create a new Machine to be part of this Swarm cluster:

    Machine talks to the Discovery Service using --swarm-discovery.

  4. Create a second node in this cluster:

  5. List all the created Machines:

    The machines that are part of the cluster have cluster’s name in the SWARM column, blank otherwise. For example, “default” and “consul-machine” are standalone machines where as all other machines are part of the “swarm-master” cluster. The Swarm master is also identified by (master) in the SWARM column.

  6. Connect to the Swarm cluster and find some information about it:

    The main difference here is --swarm when finding information about Swarm cluster as opposed to a single Machine.

    This shows the output as:

    There are 3 nodes – one Swarm master and 2 Swarm _worker_ nodes. There is a total of 4 containers running in this cluster – one Swarm agent on master and each node, and there is an additional swarm-agent-master running on the master. This can be verified by connecting to the master and listing all the containers.

  7. List nodes in the cluster with the following command:

Subsequent blog will explain how to deploy applications to this Docker Swarm Cluster.

Enjoy!

 

Docker Networking with Couchbase and WildFly

containers, couchbase, javaee, wildfly, , , , ,

Docker Multi-Host networking allows you to create virtual networks and attach containers to them so you can create the network topology that is right for your application. This blog will show how to use it with Docker Compose.

CRUD Java Application with Couchbase, Java EE, and WildFly explained how to use a Java EE application to provide a CRUD/REST interface on a data bucket in Couchbase. It required to manually download and run WildFly. The blog also used Couchbase server using Docker and required manual configuration to load travel-sample bucket.

Configure Couchbase Docker Container using REST API explained how to use Couchbase REST API to configure Couchbase Server.

Docker Multi-Host Networking

This blog will remove the explicit download of WildFly and manual configuration of Couchbase server:

  • Use Docker Compose to start WildFly and Couchbase (no download required)
  • Use a Maven profile to configure Couchbase server (no manual configuration required)
  • Uses Docker multi-host networking so that WildFly and Couchbase server can talk to each other

Lets get started!

Start Couchbase and WildFly using Docker Multi-Host Networking and Compose

  1. Start WildFly and Couchbase server using docker-compose.yml file from github.com/arun-gupta/docker-images/blob/master/wildfly-couchbase-javaee7/docker-compose.yml:
    arungupta/wildfly-admin image is used as it binds WildFly’s management to all network interfaces, and in addition also exposes port 9990. This enables WildFly Maven Plugin to be used to deploy the application.

    container_name is specified for Couchbase service and referred in WildFly service using COUCHBASE_URI. This is then used to connect to Couchbase from the Java EE application.

    The application environment is started as:

    --x-networking is an experimental switch added to Docker Compose 1.9 that allows to create a bridge or an overlay network. By default, it creates a bridge network that works on a single host. The network created can be seen as:

    Issue 2221 provide more explanation about the default networks created. wildflycouchbasejavaee7 is the new bridge network created for our application.  Issue #2345 provide some details about incorrect driver name in the output message.

Configure Couchbase Server

  1. Clone couchbase-javaee repo:

  2. Configure Couchbase server:

    exec-maven-plugin is used to invoke REST API and configure Couchbase server and is configured in a Maven profile. Make sure to setup docker.host property in pom.xml.

  3. Deploy the application to WildFly:

    Make sure to specify the correct host on CLI. In this case, this is the IP address obtained using docker-machine ip default.

Invoke the Application

  1. Invoke the REST endpoint using cURL:

    Complete set of REST endpoints are documented at CRUD Java Application with Couchbase, Java EE and WildFly. They are listed here for convenience:

    1. GET a single airline:
    2. Create a new airline using POST:

    3. Update an existing airline using PUT:
    4. Delete an existing airline using DELETE:

Enjoy!

Configure Couchbase Docker Container using REST API

containers, couchbase, ,

Couchbase Docker image is published at hub.docker.com/_/couchbase. The easiest way to start this image is:

8091 is the network port used by Couchbase Web Console for REST traffic. Complete set of ports are documented at Couchbase Network Configuration. This image can be configured using Single Host Single Container configuration as explained at hub.docker.com/_/couchbase.

This blog will show you can create a single node Couchbase cluster using Docker, configure it with Data, Index, and Query service, load a sample bucket, and query it.

Couchbase Docker Container

Start Couchbase Docker Container

Start Couchbase Docker Container using the following docker-compose.yml:

This Docker Compose file can be downloaded from github.com/arun-gupta/docker-images/tree/master/couchbase-server.

The container can be started as:

Status of the running container can be seen as:

Logs can be seen as:

Configure Couchbase Docker Container

  1. Get IP address of the Docker Host:

    Use this IP address in all the subsequent commands.

  2. Configure memory for Data and Index services:

  3. Configure Data, Query, and Index services:

  4. Setup credentials for the cluster:

Install Couchbase Travel Sample Bucket

Query Couchbase Docker Container using CBQ

  1. List container id of Couchbase server:

    This output shows the complete information about the container. Alternatively, just the container id can be obtained as:

  2. Run Couchbase Query tool:

  3. Run a query:

    Did you realize, this was a SQL query for JSON document? How cool.  Learn more about in this interactive N1QL tutorial.

Cluster overview can be seen at 192.168.99.100:8091:

Couchbase Docker Container Cluster Overview

Data buckets can seen as:

Couchbase Docker Container Databucket

Ask your questions at forums.couchbase.com, learn more about Couchbase REST API or read more in Couchbase 4 Docs.

A subsequent blog will show how all of these steps can be fully automated.

Enjoy!

Attach Shell to Docker container

containers, couchbase,

Docker Logo

Often there is a need to attach a shell to an already running Docker container. This is very useful, especially for debugging. This blog will explain how to attach a shell to an already running Docker container.

So let’s say you run a Couchbase Docker container in detached mode:

Running the container gives you the complete container id, 02061ddf0a3d1b2806a1ee6e354f4064d9d2ff4d84d8c96c0273c8883917a92f in this case.

This can be verified as:

Or complete container id can be verified as:

Attach Shell to Docker Container

Bash shell can be attached to an already running container using docker exec -it {CID} bash. The {CID} can be the complete container id, truncated container id, or even the first few digits of the container ids amongst all the currently running containers.

So the following three commands are equivalent in our case:

Attach Using Complete Container Id

Attach Using Truncated Container Id

Attach Using Unique Numbers from Container Id

In this case, just specifying 0 would work because this is the only running container. If multiple containers are running that have 0 as the starting number then the first few digits that make the container id unique are required.

Enjoy!

CRUD Java Application with Couchbase, Java EE and WildFly

containers, couchbase, javaee, wildfly, , , ,

Couchbase is an open-source, NoSQL, document database. It allows to access, index, and query JSON documents while taking advantage of integrated distributed caching for high performance data access.

Developers can write applications to Couchbase using different languages (Java, Go, .NET, Node, PHP, Python, C) multiple SDKs. This blog will show how you can easily create a CRUD application using Java SDK for Couchbase.

REST with Couchbase

The application will use curl to issue REST commands to a JAX-RS endpoint deployed on WildFly. These commands will then perform CRUD operations on travel-sample bucket in Couchbase. N1QL (SQL query language for JSON) will be used to communicate with Couchbase to retrieve results. Both the “builder pattern” and raw N1QL commands will be used.

Couchbase CRUD using WildFly and Curl

TL;DR

Complete source code and instructions for the sample are available at github.com/arun-gupta/couchbase-javaee.

Lets get started!

Run Couchbase Server

Couchbase server can be easily downloaded from Couchbase Server Downloads page. In a containerized world, its a lot easier to fire up a Couchbase server using Docker.

If Docker is configured on your machine then the easiest way is to use Docker Compose for Couchbase:

Starting up the application server shows:

And then the logs can be seen as:

The database needs to be configured and is explained at Configure Couchbase Server. Make sure to install travel-sample bucket.

Deploy the Java EE Application on WildFly

  • Download WildFly 9.0.2 , unzip, and start WildFly application server as ./wildfly-9.0.0.Final/bin/standalone.sh.
  • Git clone the repo: git clone https://github.com/arun-gupta/couchbase-javaee.git
  • Change directory cd couchbase-javaee
  • Deploy the application to WildFly: mvn install -Pwildfly.

The application uses Java SDK for Couchbase by importing the following Maven coordinates:

Invoke the REST Endpoints Using cURL

GET Airline resources (limit to 10)

Lets query the database to list 10 Airline resources.

Request

Response

The N1QL query for this is written as:
And can also be alternatively written as:
You may optionally update the code to include ORDER BY clause as shown in N1QL Tutorial.

GET one Airline resource

Use id attribute to query a single Airline resource

Request

Response

POST a new Airline resource

Learn how to run N1QL queries from the CLI using CBQ tool and verify the existing sample data:

This query retrieve documents where airline’s name is Airlinair. The count is shown in metrics.resultCount.

Create a new document using POST.

Request

Response

Query again using CBQ and now the results are shown as:
Note that two JSON documents are returned instead of one as before the POST command was issued.

PUT an existing Airline resource

Update an existing resource using HTTP POST.

The data model for travel-sample bucket requires to include “id” attribute in the payload and in the URI as well.

Request

Name of the airline is updated from “Airlinair” to “Airlin Air”, all other attributes stay the same.

Response

The updated record is shown in the response.

Querying for Airlinair gives:

So the previously added record is now updated and thus does not appear in query results. Querying for Airlin Airgives:

This shows the newly updated document.

DELETE an existing Airline resource

Query for a unique id:

Notice that one document is returned.

Lets delete this document.

Request

Response

The deleted document is shown in the response.

Query again for the deleted id:

And no results are returned!

As mentioned earlier, the complete code base is at github.com/arun-gupta/couchbase-javaee.

Enjoy!

Docker, Kubernetes, and Microservices Replay from Devoxx 2015

conferences, containers, microservices, , ,

Devoxx 2015 BE Arun

Java gives us Write Once Run Anywhere (WORA) because of the common abstraction provided by Java Virtual Machine. The binary byte code produced by Java is understood by the JVM running on multiple operating systems. This allows Java code to run on any operating system. But deploying such applications typically requires to tune the JVM, setup application server, install appropriate database drivers, other similar configuration. Docker nicely complements WORA by defining a way to package Java applications, and include all the configuration in a an easy to describe format. This can be called as Package Once Deploy Anywhere, or PODA.

Docker PODA

Do you want to learn more about how Docker helps with PODA?
How do you create multi-container applications using Docker Compose?
How do you deploy this multi-container application on multiple hosts running in a Docker Swarm cluster?
What new features Docker introduced in 1.9 release – particularly multi-host networking and persistent storage?
How do you deploy Docker containers using Kubernetes?
How does Kubernetes help with scaling applications?
How does Docker Swarm compare with Kubernetes?

I had the privilege of delivering a 3-hour university on Docker and Kubernetes followed by a 3-hour hands-on lab. The replay of the university session is now available:

Slides for the university are available at: github.com/javaee-samples/docker-java/tree/master/slides

The contents from the hands-on lab are available at bit.ly/dockerlab.

In addition, I also gave a 3-hour university on Refactor your Java EE Applications using Microservices and Containers. This session explained:

  • Basic characteristics of Microservices
  • Showed a simple shopping cart monolithic application, and how it was refactored to multiple microservices (github.com/arun-gupta/microservices)
  • Talked about transactions, event sourcing and CQRS
  • How KumuluzEE was used to create standalone JARs
  • Explained how such microservices can be deployed using Docker

Slides from this session are available at: github.com/arun-gupta/microservices/tree/master/slides

Replay is available at:

The response on twitter was quite positive:

And in case you are interested, watch a quick interview with Voxxed team:

Enjoy!

 

Docker 1.9 – Virtual networks, persistent storage, production-ready Swarm

containers,

Docker 1.9 Logo

Docker 1.9 is released, read Announcing Docker 1.9 for complete details. This release contains:

  • Docker 1.9.0 Client Binary
  • Docker Machine 0.5.0
  • Docker Compose 0.5.0
  • Docker Toolbox 1.9.0
  • Docker Swarm 1.0.0

Some of the key features are:

  • Create virtual networks that span multiple hosts and backed by multiple plugins such as Calico, Weave, Cisco, and others (full details)
  • Persistent storage support using new volume management system and backed Flocker or Ceph (full details)
  • Docker Swarm is 1.0.0 and ready for production, tested with 1000 nodes and 30,000 containers (complete list, Announcing Swarm 1.0)
  • Docker Machine code is split into core Docker Machine client and multiple binaries, one each for the driver (complete list)
  • Docker Compose now runs on Windows, environment variables are supported in docker-compose.yml file (complete list)

Getting Started with Docker

If you’ve never installed Docker on your machine, then Docker Toolbox 1.9.0 installs everything you need to get started with Docker on Mac OS X and Windows. It even includes Virtual Box 5.0.8.

Docker 1.9 Toolbox

Download Docker Toolbox 1.9.0 for Mac or Windows.

Of course, you can apt-get install docker-engine on Ubuntu or yum install docker-engineon CentOS.

If you like to update binaries independently or learn how to use them to get started, read ahead!

Install Docker 1.9.0 CLI Binary

Download the latest Docker CLI binary:

Install Docker Machine 0.5.0

Docker Machine allows you to create Docker Engines on your laptop, on cloud, or inside your data center.

Docker Machine Logo

Download the latest Docker Machine binary:

Docker Machine has been split into multiple binaries – one for the core Docker Machine client and a binary for each driver. So the installation is different from the previous releases.

And create a new machine as:

A new machine created using this could not be queried by Docker CLI (#2147). Upgrading the machine explicitly using docker-machine upgrade lab helped resolve the issue.

Hopefully this is just a timing issue and will not occur going forward.

Docker Compose 0.5.0

Docker Compose allows to easily run multi-container applications.

A simple way to verify is to run three node Couchbase Cluster.

And the list of running containers:

A later blog will show how to use persistent storage volumes to store the data.

Currently, Docker Compose requires docker-compose.yml from the local filesystem. It would be useful for docker-compose to use these files from a URI (#2313).

Docker Swarm 1.0.0

Docker Swarm provides native clustering for Docker. It converts a pool of Docker Engines into a single virtual engine. The tools, such as Docker CLI, that work with a single engine continues to work Swarm. So your applications that would run on a single Docker Engine can now easily run on multiple Docker Engines.

Docker Swarm Logo

Read more in Clustering using Docker Swarm for more details on how to setup cluster.

Download or upgrade your binaries to Docker 1.9, and use the latest features!

Docker Hands-on Workshop and Slides

Docker hands-on workshop will be updated to use Docker 1.9 in the coming days, stay tuned!

Latest slides to introduce Docker are available at github.com/javaee-samples/docker-java/tree/master/slides.

Docker Machine “client is newer than server” error

containers, techtip

Docker 1.9.0 is getting ready to be released. Docker Release Candidate builds can be downloaded from github.com/docker/docker/releases. Matching Docker Machine Release Candidate builds can be downloaded from github.com/docker/machine/releases.

So, you downloaded Docker Machine and Docker CLI Release Candidate build. The latest one at this time is 1.9.0 RC4 for Docker and 0.5.0 RC4 for Docker Machine.

Download is pretty straight forward:

And now Docker Machine:

A big change in Docker Machine is where implementation of drivers such as virtualbox, digitalocean, amazonec2, etc are no longer packaged in the main binary. Instead the distribution is a zip bundle with multiple drivers packaged and referenced from the main binary. These are now packaged separately and has the following benefits:

  1. Each driver can evolve rapidly without waiting for merging into upstream
  2. Additional drivers can be written and used without merging into the upstream
  3. New version of the drivers can be released more frequently. Hopefully more clarity will be available on how these drivers will be distributed.

That’s why installation is slightly different and looks like:

After installation, the Docker Machine can be created as:

And Docker CLI is configured to talk to this machine as:

But now when you try to see the list of images on this machine as docker images, it gives the following error:

This was filed as #2147.

Fortunately, the fix is rather simple even though non-intuitive. Docker Machine needs to be created as:

This is so because use a RC Docker binary require to specify to use a release candidate ISO. This can be done by using -virtualbox-boot2docker-url option as shown.

Now when the Docker Machine is created this way, the empty list of images is shown correctly:

Voila, back in business!