Monthly Archives: July 2015

Getting Started with ELK Stack on WildFly

Your typical business application would consist of a variety of servers such as WildFly, MySQL, Apache, ActiveMQ, and others. They each have a log format, with minimal to no consistency across them. The log statement typically consist of some sort of timestamp (could be widely varied) and some text information. Logs could be multi-line. If you are running a cluster of servers then these logs are decentralized, in different directories.

How do you aggregate these logs? Provide a consistent visualization over them? Make this data available to business users?

This blog will:

  • Introduce ELK stack
  • Explain how to start it
  • Start a WildFly instance to send log messages to the ELK stack (Logstash)
  • View the messages using ELK stack (Kibana)

What is ELK Stack?

ELK stack provides a powerful platform to index, search and analyze your data. It uses  Logstash for log aggregation, Elasticsearch for searching, and Kibana for visualizing and analyzing data. In short, ELK stack:

  • Collect logs and events data (Logstash)
  • Make it searchable in fast and meaningful ways (Elasticsearch)
  • Use powerful analytics to summarize data across many dimensions (Kibana)


Logstash is a flexible, open source data collection, enrichment, and transportation pipeline.



Elasticsearch is a distributed, open source search and analytics engine, designed for horizontal scalability, reliability, and easy management.



Kibana is an open source data visualization platform that allows you to interact with your data through stunning, powerful graphics.


How does ELK Stack work?

Logstash can collect logs from a variety of sources (using input plugins), process the data into a common format using filters, and stream data to a variety of sources (using output plugins). Multiple filters can be chained to parse the data into a common format. Together, they build a Logstash Processing Pipeline.

Logstash Processing Pipeline

Inputs and outputs support codecs that enable you to encode or decode the data as it enters or exits the pipeline without having to use a separate filter.

Logstash can then store the data in Elasticsearch and Kibana provides a visualization of that data. Here is a sample pipeline that can collect logs from different servers and run it through the ELK stack.

ELK Stack

Start ELK Stack

You can download individual components of ELK stack and start that way. There is plenty of advise on how to configure these components. But I like to start with a KISS, and Docker makes it easy to KISS!

All the source code on this blog is at

  1. Clone the repo:
  2. Run the ELK stack:
    This will use the pre-built Elasticsearch, Logstack, and Kibana images. It is built upon the work done in

    docker ps will show the output as:

    It shows all the containers running.

WildFly and ELK

James (@the_jamezp) blogged about Centralized Logging for WildFly with ELK Stack. The blog explains how to configure WildFly to send log messages to Logstash. It uses the highly modular nature of WildFly to install jboss-logmanager-ext library and install it as a module. The configured logmanager includes @timestamp field to the log messages sent to logstash. These log messages are then sent to Elasticsearch.

Instead of following the steps, lets Docker KISS and use a pre-configured image to get you started.

Start the image as:

Make sure to substitute <DOCKER_HOST_IP> with the IP address of the host where your Docker host is running. This can be easily found using docker-machine ip <MACHINE_NAME>.

View Logs using ELK Stack

Kibana runs on an embedded nginx and is configured to run on port 80 in docker-compose.yml. Lets view the logs using that.

  1. Access http://<DOCKER_HOST_IP> in your machine and it should show the default page as:ELK Stack WildFly PatternThe @timestamp field was created by logmanager configured in WildFly.
  2. Click on Create to create an index pattern and select Discover tab to view the logs as:ELK Stack WildFly Output

Try connecting other sources and enjoy the power of distributed consolidated by ELK!

Some more references …

  • Logstash docs
  • Kibana docs
  • Elasticsearch The Definitive Guide

Distributed logging and visualization is a critical component in a microservices world where multiple services would come and go at a given time. A future blog will show how to use ELK stack with a microservices architecture based application.


Kubernetes Design Patterns

14,000 commits and 400 contributors (including one tiny commit from me!) is what build Kubernetes 1.0. It is now available!

  • Download here
  • API Docs
  • Kubectl command tool
  • Getting Started Guide
  • Kubernetes Introduction Slides

This blog discusses some of the Kubernetes design patterns. All source code for the design patterns discussed below are available at kubernetes-java-sample.

Key Concepts of Kubernetes

At a very high level, there are three key concepts:

  • Pods are the smallest deployable units that can be created, scheduled, and managed. Its a logical collection of containers that belong to an application.
  • Master is the central control point that provides a unified view of the cluster. There is a single master node that control multiple minions.
  • Node is a worker node that run tasks as delegated by the master. Minions can run one or more pods. It provides an application-specific “virtual host” in a containerized environment.

Kubernetes Key Concepts


Some other concepts to be aware of:

  • Replication Controller is a resource at Master that ensures that requested number of pods are running on nodes at all times.
  • Service is an object on master that provides load balancing across a replicated group of pods.
  • Label is an arbitrary key/value pair in a distributed watchable storage that the Replication Controller uses for service discovery.

Start Kubernetes Cluster

  1. Easiest way to start a Kubernetes cluster on a Mac OS is using Vagrant:
  2. Alternatively, Kubernetes can be downloaded from, and cluster can be started as:

Kubernetes Cluster Vagrant

A Pod with One Container

This section will explain how to start a Pod with one Container. WildFly base Docker image will be used as the Container.

Kubernetes One Pod

Pod, Replication Controller, Service, etc are all resources in Kubernetes. They can be created using the kubectl by using a configuration file.

The configuration file in this case:

Complete details on how to create a Pod are explained at

Java EE Application Deployed in a Pod with One Container

This section will show how to deploy a Java EE application in a Pod with one Container. WildFly, with an in-memory H2 database, will be used as the container.

Kubernetes Java EE 7 Application

Configuration file is:

Complete details at–h2-in-memory-database.

A Replication Controller with Two Replicas of a Pod

This section will explain how to start a Replication Controller with two replicas of a Pod. Each Pod will have one WildFly container.

Kubernetes Replication Controller

Configuration file is:

Complete details at

Rescheduling Pods

Replication Controller ensures that specified number of pod “replicas” are running at any one time. If there are too many, the replication controller kills some pods. If there are too few, it starts more.

Kubernetes Pod Rescheduling

Complete details at

Scaling Pods

Replication Controller allows dynamic scaling up and down of Pods.

Kubernetes Scaling Pods

Complete details at

Kubernetes Service

Pods are ephemeral. IP address assigned to a Pod cannot be relied upon. Kubernetes, Replication Controller in particular, create and destroy Pods dynamically. A consumer Pod cannot rely upon the IP address of a producer Pod.

Kubernetes Service is an abstraction which defines a set of logical Pods. The set of Pods targeted by a Service are determined by labels associated with the Pods.

This section will show how to run a WildFly and MySQL containers in separate Pods. WildFly Pod will talk to the MySQL Pod using a Service.

Kubernetes Service

Complete details at

Here are couple of blogs that will help you get started:

The complete set of Kubernetes blog entries provide more details.


Scaling Kubernetes Cluster


Automatic Restarting of Pods inside Replication Controller of Kubernetes Cluster shows how Kubernetes reschedule pods in the cluster if one or more of existing Pods disappear for some reason. This is a common usage pattern and one of the key features of Kubernetes.

Another common usage pattern of Replication Controller is scaling:

The replication controller makes it easy to scale the number of replicas up or down, either manually or by an auto-scaling control agent, by simply updating the replicas field.

Replication Controller#Scaling

This blog will show how a Kubernetes cluster can be easily scaled up and down.

All the code used in this blog is available at kubernetes-java-sample.

Start Replication Controller and Verify

  1. Start a Replication Controller as:
  2. Get status of the Pods:
    Make sure to wait for the status to change to Running.

    Note down name of the Pods as wildfly-rc-bgtkg” and wildfly-rc-bgtkg”.

  3. Get status of the Replication Controller:

    If multiple Replication Controllers are running then you can query for this specific one using the label:

Scaling Kubernetes Cluster Up

Replication Controller allows dynamic scaling up and down of Pods.

  1. Scale up the number of Pods:
  2. Status of the Pods can be seen in another shell:
    Notice a new Pod with the name wildfly-rc-aqaqn is created.

Scale Kubernetes Cluster Down

  1. Scale down the number of Pods:
  2. Status of the Pods using -w is not correctly updated (#11338). But status of the Pods can be seen correctly as:
    Notice only one Pod is now running.

Kubernetes dynamically scales the Pods up and down using the scale --replicas command.

All code used in this blog is available at kubernetes-java-sample.


Automatic Restarting of Pods inside Replication Controller of Kubernetes Cluster


A key feature of Kubernetes is its ability to maintain the “desired state” using declared primitives. Replication Controllers is a key concept that helps achieve this state.

A replication controller ensures that a specified number of pod “replicas” are running at any one time. If there are too many, it will kill some. If there are too few, it will start more.

Replication Controller

Lets take a look on how to spin up a Replication Controller with two replicas of a Pod. Then we’ll kill one pod and see how Kubernetes will start another Pod automatically.

Start Kubernetes Cluster

  1. Easiest way to start a Kubernetes cluster on a Mac OS is using Vagrant:
  2. Alternatively, Kubernetes can be downloaded from, and cluster can be started as:

Start and Verify Replication Controller and Pods

  1. All configuration files required by Kubernetes to start Replication Controller are in kubernetes-java-sample project.  Clone the workspace:
  2. Start a Replication Controller that has two replicas of a pod, each with a WildFly container:
    The configuration file used is shown:
    Default WildFly Docker image is used here.
  3. Get status of the Pods:
    Notice -w refreshes the status whenever there is a change. The status changes from Pending to Running and then Ready to receive requests.
  4. Get status of the Replication Controller:
    If multiple Replication Controllers are running then you can query for this specific one using the label:
  5. Get name of the running Pods:
  6. Find IP address of each Pod (using the name):
    And of the other Pod as well:
  7. Pod’s IP address is accessible only inside the cluster. Login to the minion to access WildFly’s main page hosted by the containers:

Automatic Restart of Pods

Lets delete a Pod and see how a new Pod is automatically created.

Notice how the Pod with name wildfly-rc-15xg5 was deleted and a new Pod with the name wildfly-rc-0xoms was created.

Finally, delete the Replication Controller:

The latest configuration files and detailed instructions are at kubernetes-java-sample.

In real world, you’ll typically wrap this Replication Controller in a Service and front-end with a Load Balancer. But that’s a topic for another blog!


Minecon 2015 Wrapup


From a gathering of ~50 people in 2010, Minecon 2015 with 10,000 attendees created a new world record for the biggest convention for a single game.

Minecon 2015 Experience

Do you want to know what what it feels like to be at Minecon?

Minecraft Modding Workshop

Devoxx4Kids was fortunate to give Minecraft Modding workshops to ~200 kids at Minecon 2015. Feedback from all the parents and kids was quite outstanding. Glad we were able to ignite spark in some kids and get them excited in programming, and open source tools like Java, Eclipse, and Minecraft Forge.

Here are a couple of tweets:

All the instructions for minecraft modding are at

Many thanks to Mark Little and his son Adam, and my son for helping with a successful workshop. Its very important that kids feel comfortable to play with open source tools, and be willing to hack!

Using Mods for Teaching Panel

I also got the opportunity to lead a panel on Using Mods for Teaching with @DorineFlies, @YouthDigital, and @_moonlapse.

Here are some of the questions we addressed:

  1. How are you involved with modding?
  2. How many students/kids have you reached out so far?
  3. What languages/platforms do you use?
  4. Can modding be the right medium for first introduction to programming?
  5. What is an appropriate age to start modding?
  6. What can be done to fundamentally change STEM education in schools?
  7. What would you like from Mojang to improve the modding experience?

The panel was recorded and should be made available at I’ll update this blog when the exact link is available.

Minecraft Youtubers

One of the big craze, and genuine one, in the Minecraft world is about youtubers who produce video of game plays and most of them have 1m+ subscribers. Several of them were attending Minecon and we were fortunate to meet a few of them:


As Lydia walked around the main hall, most of the kids were super excited to meet their favorite youtubers!

Minecon 2015 Cape

Every Minecon attendee also get a cape that their in-game character can wear it and show-off the fact you attended a big celebration! Theme for this year was Iron Golem and it looks like as shown:

Minecon 2015 Cape

Minecraft Characters with Snaak

We also met the team behind Snaak and played around with creating some of the Minecraft characters using it.


Minecraft and HoloLens

A re-run of HoloLens and Minecraft video was also shown during one of the keynotes, and a preview is available here:

Here is the complete opening ceremony animation:

Minecon 2015 Photo Album

Check out some pictures from our trip:


And the complete photo album:

To me the highlight of the conference was meeting @SeargeDP. If there is one name that is responsible for starting modding in Minecraft, that would be him! Many thanks to him for giving us a chance to deliver minecraft modding workshops at Minecon.

And then, of course, meeting @lexmanos who is the lead developer for Minecraft Forge. We’ve authored an O’Reilly book (targeted at 8+ years old kid) and video on this topic. Several Devoxx4Kids chapters around the world have delivered workshops using the instructions based on Minecraft Forge and explained at

Check out a nice credential about book from one of the parents we met:

And last, but not the least, many thanks to the Mojang team for keeping the release cadence and supporting different modding communities.

Minecraft is truly a revolutionary game and allows to introduce Java programming to kids at a very early age!

Hopefully we get invited to Minecon 2016 again :)


Multi-container Applications using Docker Compose and Swarm

Docker Compose to Orchestrate Containers shows how to run two linked Docker containers using Docker Compose. Clustering Using Docker Swarm shows how to configure a Docker Swarm cluster.

This blog will show how to run a multi-container application created using Docker Compose in a Docker Swarm cluster.

Updated version of Docker Compose and Docker Swarm are released with Docker 1.7.0.

Docker 1.7.0 CLI

Get the latest Docker CLI:

and check the version as:

Docker Machine 0.3.0

Get the latest Docker Machine as:

and check the version as:

Docker Compose 1.3.0

Get the latest Docker Compose as:

and verify the version as:

Docker Swarm 0.3.0

Swarm is run as a Docker container and can be downloaded as:

You can learn about Docker Swarm at or Clustering using Docker Swarm.

Create Docker Swarm Cluster

The key components of Docker Swarm are shown below:

and explained in Clustering Using Docker Swarm.

  1. The easiest way of getting started with Swarm is by using the official Docker image:
    This command returns a discovery token, referred as <TOKEN> in this document, and is the unique cluster id. It will be used when creating master and nodes later. This cluster id is returned by the hosted discovery service on Docker Hub.

    It shows the output as:

    The last line is the <TOKEN>.

    Make sure to note this cluster id now as there is no means to list it later. This should be fixed with#661.

  2. Swarm is fully integrated with Docker Machine, and so is the easiest way to get started. Let’s create a Swarm Master next:

    Replace <TOKEN> with the cluster id obtained in the previous step.

    --swarm configures the machine with Swarm, --swarm-master configures the created machine to be Swarm master. Swarm master creation talks to the hosted service on Docker Hub and informs that a master is created in the cluster.

  3. Connect to this newly created master and find some more information about it:

    This will show the output as:

  4. Create a Swarm node

    Replace <TOKEN> with the cluster id obtained in an earlier step.

    Node creation talks to the hosted service at Docker Hub and joins the previously created cluster. This is specified by --swarm-discovery token://... and specifying the cluster id obtained earlier.

  5. To make it a real cluster, let’s create a second node:

    Replace <TOKEN> with the cluster id obtained in the previous step.

  6. List all the nodes created so far:

    This shows the output similar to the one below:

    The machines that are part of the cluster have the cluster’s name in the SWARM column, blank otherwise. For example, “lab” and “summit2015” are standalone machines where as all other machines are part of the “swarm-master” cluster. The Swarm master is also identified by (master) in the SWARM column.

  7. Connect to the Swarm cluster and find some information about it:

    This shows the output as:

    There are 3 nodes – one Swarm master and 2 Swarm nodes. There is a total of 4 containers running in this cluster – one Swarm agent on master and each node, and there is an additional swarm-agent-master running on the master.

  8. List nodes in the cluster with the following command:

    This shows the output as:

Deploy Java EE Application to Docker Swarm Cluster using Docker Compose

Docker Compose to Orchestrate Containers explains how multi container applications can be easily started using Docker Compose.

  1. Use the docker-compose.yml file explained in that blog to start the containers as:
    The docker-compose.yml file looks like:
  2. Check the containers running in the cluster as:
    to see the output as:
  3. “swarm-node-02” is running three containers and so lets look at the list of containers running there:
    and see the list of running containers as:
  4. Application can then be accessed again using:
    and shows the output as:

Latest instructions for this setup are always available at:


Microservices and DevOps Journey at Wix started their journey on DevOps and Microservices about two years ago and recently switched from a monolithic application to a microservices-based application. Yes, it took them full two years to complete the transition from monolith to microservices!

I got connected with Aviran Mordo (@aviranm), head of backend engineering at Wix on twitter.

They migrated to microservices because the “system could not scale” and the requirements for functional components were varied. The journey took their WAR-based deployment on Tomcat to fat JAR with embedded Jetty. On a side note, take a look at WildFly Swarm if you are interested in a similar approach for Java EE applications.

Video Interview

I discussed some points with him about this journey and you can watch the same too.

In this discussion, you’ll learn:

  • Why Continuous Delivery and DevOps are important requirements for microservices?
  • How they migrated from a big monolith to smaller monoliths and then a full-blown microservices architecture
  • How database referential integrity constraints were moved from database to application?
  • “micro” in microservices refers to the area of responsibility, nothing to do with LOC
  • Neither REST nor messaging was used for communication between different services. Which protocol was used? JSON-RPC
  • How do services register and discover each other? Is that required during early phases?
  • Why YAGNI and KISS are important?
  • Chef for configuration management and how to make it accessible for massive deployments
  • TeamCity for CI
  • Is 100% automation a requirement? Can 100% automation be achieved? Learn about Petri, Wix’s open source framework for A/B testing
  • Relevance of hybrid cloud (Google, Amazon, Private data center) and redundancy
  • Hardest part of migrating from monolith to microservice
  • How much code was repurposed during refactoring?
  • Where was the most effort spent during the two years of migration?
  • Distributed transactions
  • What was the biggest challenge in DevOps journey? Look out for a nice story towards the end that could be motivating for your team as well 😉

Additional Material

Watch the slides from DevoxxUK:

You can also learn more about their architecture in Scaling Wix to 60m Users.